Migration Data¶

To run the data generation for the migration data:

cd LEAP
python3 leap/data_generation/migration_data.py --time-delta P1Y

The --time-delta argument must be in ISO 8601 format:

ISO 8601	Meaning
P1Y1M1DT1H1M1.1S	1 year, 1 month, 1 day, 1 hour, 1 minute, 1 second, and 100 milliseconds
P40D	40 days
P1Y1D	1 year and 1 day
P3DT4H59M	3 days, 4 hours, and 59 minutes
PT2H30M	2 hours and 30 minutes
P1M	1 month
PT1M	1 minute

Datasets¶

StatCan does not contain immigration/emigration data broken down by the necessary groups (age, sex, etc), so we do not have exact data for this category. Instead, we use the following data files:

leap/processed_data/life_table.csv (generated by death_data.py)
leap/processed_data/birth/initial_population.csv (generated by birth_data.py)

Initial Population Data¶

The initial_population.csv contains the number of people in a given age, sex, province, and projection scenario, along with the number of births for that year. This data is the net number of people, factoring in death, immigration, and emigration:

initial_population.csv

Column	Type	Description
`timepoint`	`dt.datetime`	the date and time of the start of the time interval (e.g. `2020-01-01 00:00:00`)
`province`	`str`	the 2-letter province or territory ID (e.g., `BC` = British Columbia, `AB` = Alberta, `CA` = Canada)
`age`	`int`	age in years
`prop_male`	`float`	the proportion of births that are male
`n_age`	`float`	number of people of the given age living in the given province during the given time interval
`n_birth`	`float`	number of births in the given province during the given time interval
`projection_scenario`	`str`	`past` for historical data, or one of the projection scenario IDs for future data: `LG`: low-growth projection `HG`: high-growth projection `M1`: medium-growth 1 projection `M2`: medium-growth 2 projection `M3`: medium-growth 3 projection `M4`: medium-growth 4 projection `M5`: medium-growth 5 projection `M6`: medium-growth 6 projection `FA`: fast-aging projection `SA`: slow-aging projection

Life Table Data¶

The life_table.csv contains the probability of death at each timepoint for each age, sex, province, and projection scenario.

Column	Type	Description
`province`	`str`	the 2-letter province or territory ID (e.g., `BC` = British Columbia, `AB` = Alberta, `CA` = Canada)
`projection_scenario`	`str`	`past` for historical data, or one of the projection scenario IDs for future data: `LG`: low-growth projection `HG`: high-growth projection `M1`: medium-growth 1 projection `M2`: medium-growth 2 projection `M3`: medium-growth 3 projection `M4`: medium-growth 4 projection `M5`: medium-growth 5 projection `M6`: medium-growth 6 projection `FA`: fast-aging projection `SA`: slow-aging projection
`timepoint`	`dt.datetime`	the date and time of the start of the time interval (e.g. `2020-01-01 00:00:00`)
`sex`	`str`	`M` for male, `F` for female
`age`	`int`	age in years
`prob_death`	`float`	the probability that a person of the given age, sex, province, and projection scenario at the given timepoint will die during the time interval `[timepoint, timepoint + time_delta)`.

Data Processing¶

1. Load Life Table¶

First, we load the life tables from processed_data/{time_delta_tag}/death/life_table_{province}.csv and concatenate them.

If we are using a time_delta less than that of the original data (1 year), we need to split the ages into smaller intervals. Under the constant hazard rate assumption, $\mu(x, t)$ is constant over the time interval $[x, x + \Delta x_a)$, where $\Delta x_a$ is the age interval of the original data (1 year):

\[\mu(x + \tau, t + \tau) = \lambda(x, t) \quad \forall ~ \tau \in [0, \Delta x_a)\]

This means that the probability of death is constant over the time interval $[x, x + \Delta x_a)$:

\[q(x + \tau, \Delta x_b, t + \tau) = q(x, \Delta x_b, t) \quad \forall ~ \tau \in [0, \Delta x_a)\]

We end up with:

Name	Type	Description
`province`	`str`	The two-letter abbreviation of the province, e.g. `BC` for British Columbia, or `CA` for Canada.
`projection_scenario`	`str`	`past` for historical data, or one of the projection scenario IDs for future data: `LG`: low-growth projection `HG`: high-growth projection `M1`: medium-growth 1 projection `M2`: medium-growth 2 projection `M3`: medium-growth 3 projection `M4`: medium-growth 4 projection `M5`: medium-growth 5 projection `M6`: medium-growth 6 projection `FA`: fast-aging projection `SA`: slow-aging projection
`age`	`Age`	The age of the person, in years.
`sex`	`str`	One of `"M"` = male, `"F"` = female.
`timepoint`	`datetime`	The starting timepoint of the interval during which the data was collected.
`prob_death`	`float`	The probability that a person of the given age and sex, living in the given province, will die in the time interval `[timepoint, timepoint + time_delta)`.

Example: Life Table

`province`	`projection_scenario`	`age`	`sex`	`timepoint`	`prob_death`
BC	LG	0.00000 years	M	2000-01-01	0.00565
BC	LG	0.08333 years	M	2000-01-01	0.00565
BC	LG	…	M	2000-01-01	0.00565
BC	LG	0.91667 years	M	2000-01-01	0.00565
BC	LG	1.00000 years	M	2000-01-01	0.00687
BC	LG	…	M	2000-01-01	…
BC	LG	110.91667 years	M	2000-01-01	1.000000
BC	LG	110.91667 years	M	2068-12-01	1.000000
BC	LG	0.00000 years	F	2000-01-01	0.00463

1. Load Population Data¶

First, we load the population data from processed_data/{time_delta_tag}/birth/initial_population.csv.

If we are using a time_delta less than that of the original data (1 year), we need to split the ages into smaller intervals. We assume that the population is uniformly distributed over the age interval $[x, x + \Delta x_a)$:

\[\ell(x + \tau, \Delta x_b, t) = \ell(x, \Delta x_b, t) \dfrac{\Delta x_b}{\Delta x_a} \quad \forall ~ \tau \in [0, \Delta x_a)\]

For example, if we have:

`province`	`projection_scenario`	`age`	`sex`	`timepoint`	`n`
BC	LG	0	M	2000-01-01	1200

then this would become:

`province`	`projection_scenario`	`age`	`sex`	`timepoint`	`n`
BC	LG	0.00000 years	M	2000-01-01	100
BC	LG	0.083333 years	M	2000-01-01	100
BC	LG	0.166667 years	M	2000-01-01	100
BC	LG	0.250000 years	M	2000-01-01	100
BC	LG	0.333333 years	M	2000-01-01	100
BC	LG	0.416667 years	M	2000-01-01	100
BC	LG	0.500000 years	M	2000-01-01	100
BC	LG	0.583333 years	M	2000-01-01	100
BC	LG	0.666667 years	M	2000-01-01	100
BC	LG	0.750000 years	M	2000-01-01	100
BC	LG	0.833333 years	M	2000-01-01	100
BC	LG	0.916667 years	M	2000-01-01	100

We end up with:

Name	Type	Description
`province`	`str`	The two-letter abbreviation of the province, e.g. `BC` for British Columbia, or `CA` for Canada.
`projection_scenario`	`str`	`past` for historical data, or one of the projection scenario IDs for future data: `LG`: low-growth projection `HG`: high-growth projection `M1`: medium-growth 1 projection `M2`: medium-growth 2 projection `M3`: medium-growth 3 projection `M4`: medium-growth 4 projection `M5`: medium-growth 5 projection `M6`: medium-growth 6 projection `FA`: fast-aging projection `SA`: slow-aging projection
`age`	`Age`	The age of the person, in years.
`sex`	`str`	One of `"M"` = male, `"F"` = female.
`timepoint`	`datetime`	The starting timepoint of the interval during which the data was collected.
`n`	`float`	The number of people of the given age and sex, living in the given province, at the given timepoint, for the projection scenario.

Example: Population Data

`province`	`projection_scenario`	`age`	`sex`	`timepoint`	`n`
BC	LG	0.00000 years	M	2000-01-01	0.00565
BC	LG	0.08333 years	M	2000-01-01	0.00565
BC	LG	…	M	2000-01-01	0.00565
BC	LG	0.91667 years	M	2000-01-01	0.00565
BC	LG	1.00000 years	M	2000-01-01	0.00687
BC	LG	…	M	2000-01-01	…
BC	LG	110.91667 years	M	2000-01-01	1.000000
BC	LG	110.91667 years	M	2068-12-01	1.000000
BC	LG	0.00000 years	F	2000-01-01	0.00463

3. Compute Net Migration¶

Let us define the notation:

\[\theta := \{\text{sex}, \text{province}, \text{projection scenario}\}\]

To obtain the net migration, for anyone of age > 0, we compute the number of people in each age group projected to die during that time interval based on the prob_death column in the life_table.csv. Then we calculate the net change in people using the n column in the initial_population.csv.

\[\Delta n_{\text{total}} (x,\ t; \theta) = \Delta n_{\text{migration}}(x,\ t; \theta) + \Delta n_{\text{deaths}}(x,\ t; \theta)\]

Solving for $\Delta n_{\text{migration}}$ gives us the net migration during that time interval:

\[\begin{split}\Delta n_{\text{migration}}(x,\ t; \theta) &= \textcolor{magenta}{\Delta n(x,\ t; \theta)_{\text{total}}} - \textcolor{orange}{\Delta n(x,\ t; \theta)_{\text{deaths}}} \\ &= \textcolor{magenta}{ \underbrace{\ell(x,\ t; \theta)}_{\text{no. people alive age $x$ at time } t} - \underbrace{\ell(x-\Delta t,\ t-\Delta t; \theta)}_{ \text{no. people alive age $x-\Delta t$ at previous time } t - \Delta t }} \\ &+ \textcolor{orange}{ \underbrace{ \left(\ell(x-\Delta t,\ t-\Delta t; \theta) \cdot q(x-\Delta t,\ t-\Delta t; \theta)\right) }_{\text{no. deaths age $x - \Delta t$ between $[t - \Delta t, t]$}} }\end{split}\]

Example: Calculating Net Migration

For example, suppose we start with the following data:

`province`	`projection_scenario`	`age`	`sex`	`timepoint`	`n`	`prob_death`
BC	LG	1.00000 years	M	2000-01-01	100	0.000235
BC	LG	1.083333 years	M	2000-01-01	100	0.000235
BC	LG	1.166667 years	M	2000-01-01	100	0.000235
BC	LG	1.250000 years	M	2000-01-01	100	0.000235
BC	LG	1.333333 years	M	2000-01-01	100	0.000235
BC	LG	1.416667 years	M	2000-01-01	100	0.000235
BC	LG	1.500000 years	M	2000-01-01	100	0.000235
BC	LG	1.583333 years	M	2000-01-01	100	0.000235
BC	LG	1.666667 years	M	2000-01-01	100	0.000235
BC	LG	1.750000 years	M	2000-01-01	100	0.000235
BC	LG	1.833333 years	M	2000-01-01	100	0.000235
BC	LG	1.916667 years	M	2000-01-01	100	0.000235
BC	LG	2.00000 years	M	2001-01-01	115	0.000369
BC	LG	2.083333 years	M	2001-01-01	115	0.000369
BC	LG	2.166667 years	M	2001-01-01	115	0.000369
BC	LG	2.250000 years	M	2001-01-01	115	0.000369
BC	LG	2.333333 years	M	2001-01-01	115	0.000369
BC	LG	2.416667 years	M	2001-01-01	115	0.000369
BC	LG	2.500000 years	M	2001-01-01	115	0.000369
BC	LG	2.583333 years	M	2001-01-01	115	0.000369
BC	LG	2.666667 years	M	2001-01-01	115	0.000369
BC	LG	2.750000 years	M	2001-01-01	115	0.000369
BC	LG	2.833333 years	M	2001-01-01	115	0.000369
BC	LG	2.916667 years	M	2001-01-01	115	0.000369

Then we would add an n_prev column and calculate the net change in people, $\Delta n_{\text{total}}$, using the n and n_prev columns:

delta_n = n - n_prev * (1 - prob_death)

`province`	`projection_scenario`	`age`	`sex`	`timepoint`	`n`	`n_prev`	`prob_death`	`prob_death_prev`	`delta_n`
BC	LG	1.00000 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.083333 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.166667 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.250000 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.333333 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.416667 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.500000 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.583333 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.666667 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.750000 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.833333 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	1.916667 years	M	2000-01-01	100	nan	0.000235	nan	nan
BC	LG	2.00000 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.083333 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.166667 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.250000 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.333333 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.416667 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.500000 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.583333 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.666667 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.750000 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235
BC	LG	2.833333 years	M	2001-01-01	115	100	0.000235	0.000369	15.0235
BC	LG	2.916667 years	M	2001-01-01	115	100	0.000369	0.000235	15.0235

4. Convert the Age to Integer Years¶

Next, we convert the age back to integer years, by summing the number of migrants in each age interval $[x, x + \Delta x_a)$:

\[\Delta n_{\text{migration}}(x, \Delta x_a, t; \theta) = \sum_k \Delta n_{\text{migration}}(x + k \Delta x_b, \Delta x_b, t; \theta)\]

5. Compute Probability of Emigration¶

\[p_{\text{emigration}}(x,\ t; \theta) = \dfrac{\Delta n_{\text{migration}}(x,\ t; \theta)}{ \ell(x,\ t; \theta) }\]

6. Compute Proportions of Migrants¶

We calculate the proportion of migrants of a given age, sex, province, and projection scenario, relative to the total number of births in that province and projection scenario:

\[\text{prop_migrants_birth}(x,\ t; \theta) = \dfrac{\Delta n_{\text{migration}}(x,\ t; \theta)}{ \sum_{s} \ell(x=0,\ t; \theta(\text{sex}=s)) }\]

Next we calculate the proportion of immigrants/emigrants of a given age, sex, province, and projection scenario at a given timepoint, relative to the total number of immigrants/emigrants in that province and projection scenario, at that timepoint:

\[\text{prop_immigrants_timepoint}(x,\ t; \theta) = \dfrac{\Delta n_{\text{migration}}(x,\ t; \theta)}{ \sum_{x} \sum_{s} \ell(x,\ t; \theta(\text{sex}=s)) } \quad \text{for } \Delta n_{\text{migration}}(x,\ t; \theta) > 0\]

\[\text{prop_emigrants_timepoint}(x,\ t; \theta) = \dfrac{\Delta n_{\text{migration}}(x,\ t; \theta)}{ \sum_{x} \sum_{s} \ell(x,\ t; \theta(\text{sex}=s)) } \quad \text{for } \Delta n_{\text{migration}}(x,\ t; \theta) < 0\]

leap.data_generation.migration_data module¶

leap.data_generation.migration_data.get_delta_n(n: float, n_prev: float, prob_death: float) → float[source]¶

Get the population change due to migration for a given age and sex in a single time interval.

Parameters:¶

n: float¶: The number of people living in Canada for a single age, sex, timepoint, province, and projection scenario.
n_prev: float¶: The number of people living in Canada in the previous time interval for the same age, sex, province, and projection scenario as defined for n. So if n is the number of females aged 10 in the year 2020, n_prev is the number of females aged 9 in the year 2019.
prob_death: float¶: The probability that a person with a given age and sex at a given timepoint will die between the previous timepoint and this timepoint. So if the person is a female aged 10 in 2020, prob_death is the probability that a female aged 9 in 2019 will die by the age of 10.

Returns:¶

The change in population for a given timepoint, age, and sex due to migration.

leap.data_generation.migration_data.convert_age_to_int(df: pandas.DataFrame, agg: dict[str, str], groupby_cols: list[str]) → pandas.DataFrame[source]¶

Convert age values to integer years.

Parameters:¶

df: pandas.DataFrame¶: A dataframe containing an age column with age values which may be non-integer years.
agg: dict[str, str]¶: A dictionary specifying the aggregation method to use for other columns when grouping by integer age. The keys should be the column names, and the values should be the aggregation method to use for that column, e.g. "sum" or "mean".
groupby_cols: list[str]¶: The columns to group by when converting age to integer. Should not include the age column, but should include all other columns which are needed to uniquely identify rows after grouping by integer age.

Returns:¶

A dataframe with the same columns as the input dataframe, but with the age column converted to integer years. The values in the other columns are aggregated according to the specified aggregation methods when grouping by integer age.

leap.data_generation.migration_data.load_mortality_data(time_delta: leap.utils.TimeDelta) → pandas.DataFrame[source]¶

Load the mortality data for the given time delta.

Parameters:¶

time_delta: leap.utils.TimeDelta¶: The duration of time between subsequent data points.

Returns:¶

timepoint: The timepoint which the data applies to.
province: A string indicating the 2-letter province abbreviation, e.g. "BC". For all of Canada, set province to "CA".
age: The integer age.
sex: One of M = male, F = female.
prob_death: The probability that a person with a given age and sex at a given timepoint will die between the previous timepoint and this timepoint.

Return type:¶

A dataframe with the following columns

leap.data_generation.migration_data.load_population_data(time_delta: leap.utils.TimeDelta) → pandas.DataFrame[source]¶

Load the population data for the given time delta.

Parameters:¶

time_delta: leap.utils.TimeDelta¶: The duration of time between subsequent data points.

Returns:¶

timepoint: The timepoint which the data applies to.
province: A string indicating the 2-letter province abbreviation, e.g. "BC". For all of Canada, set province to "CA".
age: The integer age.
sex: One of M = male, F = female.
n: The number of people living in Canada for a single age, sex, timepoint, province, and projection scenario.
projection_scenario: The projection scenario.

Return type:¶

A dataframe with the following columns

leap.data_generation.migration_data.load_migration_data(df_population: pandas.DataFrame, life_table: pandas.DataFrame, time_delta: leap.utils.TimeDelta) → pandas.DataFrame[source]¶

Generate migration data for the given provinces and years.

Parameters:¶

df_population: pandas.DataFrame¶

A dataframe containing the population data. Must have columns: * timepoint (dt.datetime): The given timepoint. * province (str): The 2-letter province ID, e.g. BC. * age (int): The integer age. * sex (str): One of M = male, F = female. * projection_scenario (str): The projection scenario abbreviation. * n (float): The number of people living in Canada for a single age, sex, timepoint,

province, and projection scenario.

life_table: pandas.DataFrame¶

A dataframe containing the life table data. Must have columns: * timepoint (dt.datetime): The given timepoint. * province (str): The 2-letter province ID, e.g. BC. * age (int): The integer age. * sex (str): One of M = male, F = female. * prob_death (float): The probability that a person with a given age and sex at a

given timepoint will die between the previous timepoint and this timepoint.

time_delta: leap.utils.TimeDelta¶

The duration of time between subsequent data points.

Returns:¶

timepoint: The timepoint which the data applies to.
province: A string indicating the 2-letter province abbreviation, e.g. "BC". For all of Canada, set province to "CA".
sex: One of M = male, F = female.
age: The integer age.
projection_scenario: The projection scenario.
delta_n: The signed change in population for a given timepoint, age, sex, province, and projection scenario due to net migration. Positive values indicate net immigration; negative values indicate net emigration.
prop_migrants_birth: The signed proportion of delta_n relative to the total number of births in that time interval for the given province and projection scenario. Positive = net immigration, negative = net emigration.
prop_immigrants_timepoint: For cells where delta_n > 0, the proportion of immigrants for this age and sex relative to the total number of immigrants in that time interval. Zero for emigration cells. Denominator includes only immigration cells.
prop_emigrants_timepoint: For cells where delta_n < 0, the proportion of emigrants for this age and sex relative to the total number of emigrants in that time interval. Zero for immigration cells. Denominator includes only emigration cells.
prob_emigration: For cells where delta_n < 0, the per-person probability of emigrating (abs(delta_n) / N). Zero for immigration cells.

Return type:¶

A dataframe with the following columns

leap.data_generation.migration_data.generate_migration_data(time_delta: leap.utils.TimeDelta, to_csv: bool = True) → pandas.DataFrame | None[source]¶

Generate immigration/emigration data.

Parameters:¶

time_delta: leap.utils.TimeDelta¶: The duration of time between subsequent data points.

leap.data_generation.migration_data.plot(df: pandas.DataFrame, y: str, color: str, title: str = '', ylabel: str = '', file_path: pathlib.Path | None = None, width: int = 2000, height: int = 1500)[source]¶

Plot the migration data for validation.

Parameters:¶

df: pandas.DataFrame¶

A dataframe containing the life table data. Must have columns:

timepoint (dt.datetime): The given timepoint.
province (str): The 2-letter province ID, e.g. BC.
age (int): The integer age.
sex (str): One of M = male, F = female.
projection_scenario (str): The projection scenario abbreviation.

y: str¶

The name of the column in the dataframe which will be plotted as the y data.

color: str¶

The name of the column in the dataframe which will be used to color the data.

title: str = ''¶

The title of the plot.

ylabel: str = ''¶

The label for the y-axis.

file_path: pathlib.Path | None = None¶

The path to save the plot to.

width: int = 2000¶

The width of the plot.

height: int = 1500¶

The height of the plot.