src.pm_rank.model.utils¶

Attributes¶

`AGGREGATE_FNS`
`DEFAULT_BOOTSTRAP_CI_CONFIG`

Classes¶

BootstrapCIConfig

Configuration for bootstrap confidence interval computation.

Functions¶

`get_logger`([name])
`forecaster_data_to_rankings`(forecaster_data[, ...])	Convert the forecaster data to rankings.
`spearman_correlation`(→ float)	Compute the Spearman correlation between two rankings.
`kendall_correlation`(→ float)	Compute the Kendall correlation between two rankings.
`log_ranking_table`(logger, ranking_result[, max_rows])	Take in any model's ranking result, which might or might not include scores, and log the table.

Module Contents¶

src.pm_rank.model.utils.get_logger(name: str = 'pm_rank.model')¶

src.pm_rank.model.utils.AGGREGATE_FNS¶

class src.pm_rank.model.utils.BootstrapCIConfig¶

Bases: pydantic.BaseModel

Configuration for bootstrap confidence interval computation.

This configuration class defines parameters for computing bootstrap confidence intervals used in forecaster ranking evaluations.

Parameters:

num_bootstrap_samples – The number of bootstrap samples to draw (default: 1000).
bootstrap_ci_level – The confidence level for the bootstrap confidence interval (default: 0.95).
random_seed – The random seed for reproducible bootstrap sampling. Set to None for random seeding (default: 42).
symmetric – Whether to use symmetric confidence intervals (default: True).

num_bootstrap_samples: int¶

bootstrap_ci_level: float¶

random_seed: int | None¶

symmetric: bool¶

src.pm_rank.model.utils.DEFAULT_BOOTSTRAP_CI_CONFIG¶

src.pm_rank.model.utils.forecaster_data_to_rankings(forecaster_data: Dict[str, List[float]], include_scores: bool = True, include_bootstrap_ci: bool = False, ascending: bool = True, aggregate: Literal['mean', 'median', 'max', 'min'] = 'mean', aggregate_fn: Callable = None, bootstrap_ci_config: BootstrapCIConfig = DEFAULT_BOOTSTRAP_CI_CONFIG)¶

Convert the forecaster data to rankings. A forecaster data is a dictionary that maps forecaster name to a list of scores.

Args:: forecaster_data: a dictionary that maps forecaster name to a list of scores. include_scores: whether to include the scores in the rankings. include_bootstrap_ci: whether to include the bootstrap confidence intervals in the rankings. ascending: if true, the score is smaller, the better; otherwise, the score is larger, the better.
Returns:: A dictionary that maps forecaster name to a list of rankings.

src.pm_rank.model.utils.spearman_correlation(rank_dict_a: Dict[str, int], rank_dict_b: Dict[str, int]) → float¶: Compute the Spearman correlation between two rankings. Reference: https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient

src.pm_rank.model.utils.kendall_correlation(rank_dict_a: Dict[str, int], rank_dict_b: Dict[str, int]) → float¶: Compute the Kendall correlation between two rankings. Reference: https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient

src.pm_rank.model.utils.log_ranking_table(logger: logging.Logger, ranking_result: Tuple | Dict[str, int], max_rows: int = 25)¶: Take in any model’s ranking result, which might or might not include scores, and log the table.