Death Data¶
To run the data generation for the mortality data:
cd LEAP
python3 leap/data_generation/death_data.py --time-delta P1Y
This will update the following data files:
leap/processed_data/{time_delta_tag}/death/life_table_{province}.csv
The --time-delta argument must be in ISO 8601 format:
ISO 8601 |
Meaning |
|---|---|
P1Y1M1DT1H1M1.1S |
1 year, 1 month, 1 day, 1 hour, 1 minute, 1 second, and 100 milliseconds |
P40D |
40 days |
P1Y1D |
1 year and 1 day |
P3DT4H59M |
3 days, 4 hours, and 59 minutes |
PT2H30M |
2 hours and 30 minutes |
P1M |
1 month |
PT1M |
1 minute |
Datasets¶
To obtain the mortality data for each year, we used one table from StatCan:
1996 - 2021:
For past years, we used Table 13-10-00837-01 from StatCan.
The
*.csvfile can be downloaded from here: 13100837-eng.zip.and is saved as:
LEAP/leap/original_data/13100837.csv2021 - 2068:
StatCandoesn’t provide annual projections for death probabilities, but does provide a projection for specific years (which we call calibration years) for theM3projection scenario only. For Canada, this is 2068, and for BC, 2043.
Data Processing¶
1. Load Past Life Table¶
First, we load the past life table from the StatCan data file, 13100837.csv. This file
contains the probability of death for each age, with a time delta of 1 year, for the years 1996
to 2021. The output is a dataframe with the following columns:
Name |
Type |
Description |
|---|---|---|
|
|
The two-letter abbreviation of the province, e.g. |
|
|
The age of the person, in integer years. |
|
|
One of |
|
|
The starting timepoint of the interval during which the data was collected. |
|
|
The probability that a person of the given age and sex, living in the given province, will
die in the time interval |
Example: Past Life Table
|
|
|
|
|
|---|---|---|---|---|
BC |
0 |
M |
1996-01-01 |
0.005 |
BC |
… |
M |
1996-01-01 |
… |
BC |
110 |
M |
1996-01-01 |
0.0023 |
BC |
0 |
M |
1997-01-01 |
0.0043 |
BC |
… |
M |
… |
… |
BC |
110 |
M |
2022-01-01 |
0.0044 |
BC |
0 |
F |
1996-01-01 |
0.0054 |
2. Calibration Data¶
Next, we load the calibration data from the StatCan files:
mortality_projections_table_3-2.csvmortality_projections_table_5-2.csv
These files contain the projected life expectancies for a handful of specific years, which we will use to calibrate our model. The output is a dataframe with the following columns:
Name |
Type |
Description |
|---|---|---|
|
|
The two-letter abbreviation of the province, e.g. |
|
|
One of |
|
|
The starting timepoint of the interval during which the data was collected. |
|
|
The two-letter projection scenario, e.g. |
|
|
The two-letter mortality scenario, e.g. |
|
|
The projected life expectancy for a person of the given sex, living in the given province, under the specified projection and moortality scenarios. |
Example: Calibration Data
|
|
|
|
|
|
|---|---|---|---|---|---|
CA |
M |
2028-01-01 |
LG |
HM |
80.5 |
CA |
F |
2028-01-01 |
LG |
HM |
84.2 |
CA |
M |
2048-01-01 |
LG |
HM |
84.5 |
CA |
F |
2048-01-01 |
LG |
HM |
86.7 |
3. Determine the Beta Parameters¶
The following equation can be used to obtain the probability of death in future years:
where:
\(\sigma^{-1}\) is the inverse sigmoid function, also known as the logit function:
and:
\(x\) is the age in years
\(\Delta x\) is the time between data points of the original data (in our case, 1 year)
\(s\) is the sex
\(t_0\) is the year the collected data ends (in our case, 2022)
\(t\) is the future year
\(\theta = \{\text{sex}, \text{province}, \text{projection_scenario}\}\)
\(q(x, \Delta x, t_0; \theta)\) is the probability that a person of age \(x\) and sex \(s\) in the year \(t_0\) will die between the ages \(x\) and \(x + \Delta x\).
\(q(x, \Delta x, t; \theta)\) is the probability that a person of age \(x\) and sex \(s\) in the year \(t\) will die between the ages \(x\) and \(x + \Delta x\).
The parameter \(\beta(\theta)\) is unknown, and so we first need to calculate it.
To do so, we set \(t = \text{calibration_year}\), and use the Brent root-finding
algorithm to optimize \(\beta(\theta)\) such that the life expectancy in the calibration year
(which is known) matches the predicted life expectancy.
Example: Beta Parameters
{
("CA", "M", "LG"): 0.0123,
("CA", "F", "LG"): 0.0112,
("BC", "M", "LG"): 0.0134,
("BC", "F", "LG"): 0.0109
}
Once we have found \(\beta(\theta)\), we can use this formula to find the projected death probabilities.
Example: Projected Death Probabilities (Original Time Delta)
|
|
|
|
|
|
|---|---|---|---|---|---|
BC |
FA |
0 |
F |
2023-01-01 |
0.004919 |
BC |
FA |
0 |
F |
2024-01-01 |
0.003945 |
BC |
FA |
0 |
F |
… |
… |
BC |
FA |
0 |
F |
2068-01-01 |
0.003409 |
4. Converting the Time Delta for Death Probabilities¶
We have annual data for the past years, but we may want to convert it to a different time delta,
e.g. 1 month. Let \(\Delta x_a\) be the original time interval, and \(\Delta x_b\) be the
new (desired) time interval. We will make the assumption that the hazard rate \(\mu(x, t)\)
is constant over the time interval \([x, x + \Delta x_a)\). The hazard rate is given by:
where \(S_X(x, t)\) is the survival function for age at death, \(X\), at time \(t\). Solving this first order separable linear differential equation, we have:
Math: Solving for \(S_X(x, t)\)
Assuming that \(\mu(x, t) = \lambda\), for some constant \(\lambda\), we have:
Now, we can substitute this into the equation for \(q(x, \Delta x, t)\):
Math: \(q(x, \Delta x, t)\)
We can write this in terms of the original time interval \(\Delta x_a\):
Math: \(q(x, \Delta x_b, t)\)
We have the original probability of death for the time interval \(\Delta x_a\):
and we want to find the probability of death for the time interval \(\Delta x_b\):
This will give us the probability of death for the new time interval \(\Delta x_b\), but only at the timepoints for which we have data. To get the probability of death for all timepoints, we note that since \(\mu(x, t)\) is constant over the time interval \([x, x + \Delta x_a)\):
Math: \(q(x, \Delta x_b, t + n \Delta x_b)\)
Example: Projected Death Probabilities (Monthly Time Delta)
|
|
|
|
|
|
|---|---|---|---|---|---|
BC |
FA |
0 |
F |
2023-01-01 |
0.000919 |
BC |
FA |
0 |
F |
2023-02-01 |
0.000945 |
BC |
FA |
0 |
F |
2023-03-01 |
0.000294 |
BC |
FA |
0 |
F |
… |
… |
BC |
FA |
0 |
F |
2023-12-01 |
0.000545 |
BC |
FA |
0 |
F |
… |
… |
BC |
FA |
0 |
F |
2068-01-01 |
0.003409 |
leap.data_generation.death_data module¶
- leap.data_generation.death_data.convert_prob_death(prob_death: float, time_delta_od: leap.utils.TimeDelta, time_delta: leap.utils.TimeDelta) float[source]¶
Convert the probability of death from the original time delta to the new time delta.
The probability of death, \(q(x, \Delta x)\), depends on the choice of \(\Delta x\), i.e.
time_delta. The original data is collected with \(\Delta x = 1\) year, but we may want to convert it to a different \(\Delta x\), e.g. 1 month.Assuming the hazard ratio is constant over the time interval:
\[q(x, \Delta x_b, t) = 1 - (1 - q(x, \Delta x_a, t))^{\frac{\Delta x_b}{\Delta x_a}}\]- Parameters:¶
- prob_death: float¶
The probability of death for the original time delta, \(q(x, \Delta x_a)\). This is the probability that a person of age \(x\) will die between the ages of \([x, x + \Delta x_a)\).
- time_delta_od: leap.utils.TimeDelta¶
The original time delta, \(\Delta x_a\).
- time_delta: leap.utils.TimeDelta¶
The new time delta, \(\Delta x_b\).
- Returns:¶
The probability of death for the new time delta, \(q(x, \Delta x_b)\).
- leap.data_generation.death_data.calculate_life_expectancy(life_table: pandas.DataFrame, time_delta: leap.utils.TimeDelta) float[source]¶
Determine the life expectancy for a person born in a given time interval.
The life expectancy can be calculated from the death probability using the formulae delineated here: Life Table Definitions
- Parameters:¶
- life_table: pandas.DataFrame¶
A dataframe containing the probability of death for a single timepoint, province and sex, for each age. Columns:
age: the integer age.sex: One ofM= male,F= female.timepoint: the timepoint of the data in the row.province: A string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".projection_scenario: A string indicating the projection scenario, e.g."FA".prob_death: the probability of death for a given age, province, sex, and timepoint.
- time_delta: leap.utils.TimeDelta¶
The duration of time between data points.
- Returns:¶
The life expectancy for a person born in the given time interval, in a given province, for a given sex.
- leap.data_generation.death_data.get_prob_death_projected(prob_death: float, timepoint_initial: datetime.datetime, timepoint: datetime.datetime, beta_time: float) float[source]¶
Given the (known) prob death for a past timepoint, calculate the prob death at a future timepoint.
\[\sigma^{-1}(p(\text{sex}, \text{age}, \text{timepoint})) = \sigma^{-1}(p(\text{sex}, \text{age}, \text{timepoint}_0)) - \beta(\text{sex})(\text{timepoint} - \text{timepoint}_0)\]- Parameters:¶
- prob_death: float¶
The probability of death for
timepoint_initial, the last timepoint that past data was collected, for a given age, sex, province, and projection scenario.- timepoint_initial: datetime.datetime¶
The initial timepoint with a known probability of death. This is the last timepoint that the past data was collected.
- timepoint: datetime.datetime¶
The current timepoint.
- beta_time: float¶
The beta parameter for the given sex, province, and projection scenario.
- Returns:¶
The projected probability of death for the current timepoint.
Examples
>>> timepoint_initial = dt.datetime(1996, 1, 1) >>> timepoint = dt.datetime(2026, 1, 1) >>> get_prob_death_projected(0.01, timepoint_initial, timepoint, -0.02) np.float64(0.005512990331820702)
- leap.data_generation.death_data.get_projected_life_table_single_timepoint(beta_time: float, life_table: pandas.DataFrame, timepoint_initial: datetime.datetime, timepoint: datetime.datetime) pandas.DataFrame[source]¶
Get the life table for a single timepoint.
- Parameters:¶
- beta_time: float¶
The beta parameter for the given sex, province, and projection scenario.
- life_table: pandas.DataFrame¶
A dataframe containing the projected probability of death for the starting timepoint, for a single sex, province, and projection scenario. Columns:
age: the integer age.sex: One ofM= male,F= female.timepoint: the starting calendar year.province: a string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".prob_death: the probability of death for a given age, province, projection scenario, sex, and timepoint.
- timepoint_initial: datetime.datetime¶
The initial year with a known probability of death. This is the last year that the past data was collected.
- timepoint: datetime.datetime¶
The current timepoint.
- Returns:¶
A dataframe containing the projected probability of death for the given timepoint, sex, province, and projection scenario. Columns:
age: the integer age.sex: One ofM= male,F= female. This should match the sex in the originallife_tabledataframe.timepoint: The timepoint for which the projected probability of death was calculated. This should match thetimepointargument.province: a string indicating the province abbreviation, e.g."BC". This should match the province in the originallife_tabledataframe.projection_scenario: a string indicating the projection scenario, e.g."FA". This should match the projection scenario in the originallife_tabledataframe.prob_death: the projected probability of death for a given age, province, projection scenario, sex, and timepoint.
- leap.data_generation.death_data.compute_life_expectancy_diff(beta_time: numpy.ndarray, life_table: pandas.DataFrame, df_calibration: pandas.DataFrame, timepoint_initial: datetime.datetime, time_delta: leap.utils.TimeDelta) numpy.ndarray[source]¶
Calculate the difference between the projected life expectancy and desired life expectancy.
This function is passed to the
scipy.optimize.leastsqfunction. We want to findbeta_timesuch that the projected life expectancy is as close as possible to the desired life expectancy.- Parameters:¶
- beta_time: numpy.ndarray¶
The beta parameter for the given timepoint. The
scipy.optimize.leastsqfunction requires that this be a 1D array, but we only have a single parameter.- life_table: pandas.DataFrame¶
A dataframe containing the projected probability of death for the initial year, for a single sex and province. Columns:
age: the integer age.sex: one of"M"= male,"F"= female.timepoint: the year used as \(t_0\); this is the last year that the past data was collected.province: a 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".prob_death: the probability of death for a given age, province, sex, and year.
- df_calibration: pandas.DataFrame¶
A dataframe containing the life expectancy projections for the calibration years, for a single sex, province, and projection scenario. Columns:
timepoint (dt.datetime): The year the projection is for. Range[1988, 2073].province: A 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".sex: One ofF= female,M= male.projection_scenario (str): Population growth type, one of:LG: low-growth projectionHG: high-growth projectionM1: medium-growth 1 projectionM2: medium-growth 2 projectionM3: medium-growth 3 projectionM4: medium-growth 4 projectionM5: medium-growth 5 projectionM6: medium-growth 6 projectionFA: fast-aging projectionSA: slow-aging projection
mortality_scenario: The mortality scenario. One of:LM: Low mortalityMM: Medium mortalityHM: High mortality
life_expectancy: The life expectancy in years for the given year, province, sex, projection scenario, and mortality scenario.
- timepoint_initial: datetime.datetime¶
The initial timepoint with a known probability of death. This is the last timepoint that the past data was collected.
- time_delta: leap.utils.TimeDelta¶
The duration of time between data points.
- Returns:¶
The difference between the projected life expectancy of the calibration year and the desired life expectancy, for each of the calibration years.
Examples
>>> past_life_table = load_past_death_data() >>> df_calibration = load_projected_death_data(min_timepoint=dt.datetime(1996, 1, 1)) >>> df_calibration = df_calibration.loc[ ... (df_calibration["projection_scenario"] == "LG") & ... (df_calibration["province"] == "CA") & ... (df_calibration["sex"] == "F") ... ] >>> life_table = past_life_table.loc[ ... (past_life_table["province"] == "CA") & ... (past_life_table["sex"] == "F") & ... (past_life_table["timepoint"] == dt.datetime(2022, 1, 1)) ... ] >>> beta_time = np.array([-0.02]) >>> compute_life_expectancy_diff( ... beta_time, life_table, df_calibration, dt.datetime(1996, 1, 1), TIME_DELTA_OD ... ) array([4.90291552, 5.98995694, 7.75593427])
- leap.data_generation.death_data.load_past_death_data() pandas.DataFrame[source]¶
Load the past death data from the
StatCanCSV file.- Returns:¶
A dataframe containing the probability of death for each timepoint, province, age, and sex. The time delta of the data is that of the original data, which is 1 year. Columns:
timepoint: The starting timepoint of the interval during which the data was collected.province: A 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".projection_scenario: The projection scenario, i.e."past".sex: One of"M"= male,"F"= female.age: The integer age.prob_death: The probability that a person of the given age, sex, and province will die in the given year.
- leap.data_generation.death_data.load_projected_death_data(min_timepoint: datetime.datetime) pandas.DataFrame[source]¶
Load the projected death data from the
StatCanCSV files.Statistics Canadaprovides two tables with life expectancy projections:This data is only available for selected years.
- Returns:¶
A dataframe containing the life expectancy from selected calibration years from
Statistics Canada:timepoint (dt.datetime): Range[1988, 2073].province (str): A 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".sex (str): One ofF= female,M= male.projection_scenario (str): Population growth type, one of:past: historical dataLG: low-growth projectionHG: high-growth projectionM1: medium-growth 1 projectionM2: medium-growth 2 projectionM3: medium-growth 3 projectionM4: medium-growth 4 projectionM5: medium-growth 5 projectionM6: medium-growth 6 projectionFA: fast-aging projectionSA: slow-aging projection
mortality_scenario (str): The mortality scenario. One of:LM: Low mortalityMM: Medium mortalityHM: High mortality
life_expectancy (float): The life expectancy in years for the given year, province, sex, projection scenario, and mortality scenario.
-
leap.data_generation.death_data.compute_beta_parameters(past_life_table: pandas.DataFrame, df_calibration: pandas.DataFrame, time_delta_od: leap.utils.TimeDelta =
TimeDelta(years=+1), x0: float =-0.02, xtol: float =1e-05) dict[tuple[str, str, str], float][source]¶ Compute the beta parameters for each province, sex, and projection scenario.
The beta parameters are used to project the probability of death for future timepoints, given the probability of death for past timepoints. We use the following formula to project the probability of death:
\[\sigma^{-1}(q(x, \Delta x, t; \theta, \text{age})) = \sigma^{-1}(q(x, \Delta x, t_0; \theta, \text{age})) - \beta_{\theta} (t - t_0)\]where \(\theta\) is given by:
\[\theta := \{\text{province}, \text{sex}, \text{projection scenario}\}\]In this function, we optimize the \(\beta_{\theta}\) parameter for each \(\theta\) such that the projected life expectancy matches the desired life expectancy from the calibration data.
- Parameters:¶
- past_life_table: pandas.DataFrame¶
A dataframe containing the probability of death for each timepoint, province, age, and sex. Columns:
timepoint: the starting timepoint of the interval during which the data was collected.province: A 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".sex: One ofM= male,F= female.age: the integer age.prob_death: the probability of death.
- df_calibration: pandas.DataFrame¶
A dataframe containing the life expectancy projections for the calibration years. Columns:
timepoint: The calendar year. Range[1988, 2073].province: A 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".sex: One ofF= female,M= male.projection_scenario (str): Population growth type, one of:LG: low-growth projectionHG: high-growth projectionM1: medium-growth 1 projectionM2: medium-growth 2 projectionM3: medium-growth 3 projectionM4: medium-growth 4 projectionM5: medium-growth 5 projectionM6: medium-growth 6 projectionFA: fast-aging projectionSA: slow-aging projection
mortality_scenario: The mortality scenario. One of:LM: Low mortalityMM: Medium mortalityHM: High mortality
life_expectancy: The life expectancy in years for the given year, province, sex, projection scenario, and mortality scenario.
- time_delta_od: leap.utils.TimeDelta =
TimeDelta(years=+1)¶ The original duration of time between data points in the past data.
- x0: float =
-0.02¶ The initial guess for the beta parameter.
- xtol: float =
1e-05¶ The tolerance for the beta parameter.
- Returns:¶
A dictionary containing the beta parameters for each province, sex, and projection scenario. Format:
key:
(province, sex, projection_scenario)value: beta parameter for the given province, sex, and projection scenario.
Examples
>>> past_life_table = load_past_death_data() >>> df_calibration = load_projected_death_data(min_timepoint=dt.datetime(1996, 1, 1)) >>> df_calibration = df_calibration.loc[df_calibration["projection_scenario"].isin(["LG", "M3"])] >>> beta_parameters = compute_beta_parameters(past_life_table, df_calibration) >>> list(beta_parameters.keys()) [('CA', 'M', 'LG'), ('CA', 'M', 'M3'), ('CA', 'F', 'LG'), ('CA', 'F', 'M3'), ('BC', 'M', 'LG'), ('BC', 'M', 'M3'), ('BC', 'F', 'LG'), ('BC', 'F', 'M3')] >>> beta_parameters[("CA", "M", "LG")] np.float64(-0.01)
- leap.data_generation.death_data.get_projected_death_data(beta_parameters: dict[tuple[str, str, str], float], past_life_table: pandas.DataFrame) pandas.DataFrame[source]¶
Load the projected death data from
StatCanCSV file.- Parameters:¶
- beta_parameters: dict[tuple[str, str, str], float]¶
A dictionary containing the beta parameters for each province, sex, and projection scenario. Format:
key:
(province, sex, projection_scenario)value: beta parameter for the given province, sex, and projection scenario.
- past_life_table: pandas.DataFrame¶
A dataframe containing the probability of death for each timepoint, province, age, and sex. Columns:
timepoint: the starting timepoint of the interval during which the data was collected.province: A 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".projection_scenario: The population growth type, i.e."past".sex: One ofM= male,F= female.age: the integer age.prob_death: the probability of death.
- Returns:¶
A dataframe containing the predicted probability of death for each annual timepoint, province, age, and sex. Columns:
timepoint (dt.datetime): The starting timepoint of the interval the data applies to.province (str): A 2-letter string indicating the province abbreviation, e.g."BC". For all of Canada, set province to"CA".projection_scenario (str): Population growth type, one of:past: historical dataLG: low-growth projectionHG: high-growth projectionM1: medium-growth 1 projectionM2: medium-growth 2 projectionM3: medium-growth 3 projectionM4: medium-growth 4 projectionM5: medium-growth 5 projectionM6: medium-growth 6 projectionFA: fast-aging projectionSA: slow-aging projection
sex (str): One ofM= male,F= female.age (int): The integer age.prob_death (float): The probability that a person of the given age, sex, and province will die in the given year.
-
leap.data_generation.death_data.generate_death_data(time_delta: leap.utils.TimeDelta, time_delta_od: leap.utils.TimeDelta =
TimeDelta(years=+1), to_csv: bool =True, draw_plot: bool =False, x0: float =-0.02, xtol: float =1e-05) None | pandas.DataFrame[source]¶ Generate the mortality data CSV.
- Parameters:¶
- time_delta: leap.utils.TimeDelta¶
The duration of time between data points.
- time_delta_od: leap.utils.TimeDelta =
TimeDelta(years=+1)¶ The original duration of time between data points.
- to_csv: bool =
True¶ Whether to save the data to a CSV file. If False, the dataframe will be returned instead.
- draw_plot: bool =
False¶ Whether to draw a plot of the mortality data for validation.
- x0: float =
-0.02¶ The initial guess for the beta parameter.
- xtol: float =
1e-05¶ The tolerance for the beta parameter.
- Returns:¶
If
to_csvis False, a dataframe containing the probability of death and the standard error for each timepoint, province, age, and sex.
-
leap.data_generation.death_data.plot(df: pandas.DataFrame, y: str, color: str, title: str =
'', file_path: pathlib.Path | None =None, width: int =2000, height: int =1500)[source]¶ Plot the mortality data for validation.
- Parameters:¶
- df: pandas.DataFrame¶
A dataframe containing the life table data. Must have columns:
timepoint (dt.datetime): The given timepoint.province (str): The 2-letter province ID, e.g."BC".age (int): The integer age.sex (str): One ofM= male,F= female.prob_death (float): The probability of death for the given timepoint, province, age, and sex.
- y: str¶
The name of the column in the dataframe which will be plotted as the
ydata.- color: str¶
The name of the column in the dataframe which will be used to color the data.
- title: str =
''¶ The title of the plot.
- file_path: pathlib.Path | None =
None¶ The path to save the plot to.
- width: int =
2000¶ The width of the plot.
- height: int =
1500¶ The height of the plot.