Pystagegate Demos#
Executable notebook demonstrating code usage
Notebook kernel setup
import os
from pystagegate.pipelines import prov_fin_main
os.chdir("..")
config_path = "tests/data/testing_config.json"
Provisional-Final demos#
Run the prov_fin_main pipeline
prov_fin_df = prov_fin_main(config_path)
/home/runner/work/pystagegate/pystagegate/src/pystagegate/pipelines.py:85: UserWarning: No directory found at tests/data/output/, writing in new directory /home/runner/work/pystagegate/pystagegate/tests/data/output
utils.write_outputs(config["output_path"], (output, "prov_fin_output.csv"))
Preview the sum of squared differences between provisional and final migration estimates:
prov_fin_df
| Local Authority Code | imm_prov | em_prov | net_prov | sqdiff_imm | sqdiff_em | sqdiff_net | sqdiff_imm_sc | sqdiff_em_sc | sqdiff_net_sc | nation | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | E1 | 261.0 | 277.0 | -16.0 | 13.341052 | 14.066657 | 289.735312 | 0.051115 | 0.050782 | 1.110097 | E |
| 1 | E2 | 252.0 | 268.0 | -16.0 | 25.350321 | 10.515225 | 527.507549 | 0.100597 | 0.039236 | 2.093284 | E |
| 2 | E3 | 280.0 | 278.0 | 2.0 | 10.993675 | 11.317117 | 133.477720 | 0.039263 | 0.040709 | 0.476706 | E |
| 0 | W1 | 276.0 | 249.0 | 27.0 | 22.312052 | 22.793154 | 656.625005 | 0.080841 | 0.091539 | 2.379076 | W |
| 1 | W2 | 279.0 | 236.0 | 43.0 | 18.799314 | 19.939075 | 258.685863 | 0.067381 | 0.084488 | 0.927189 | W |
| 2 | W3 | 249.0 | 267.0 | -18.0 | 8.327029 | 30.382319 | 1292.074216 | 0.033442 | 0.113791 | 5.189053 | W |
| 0 | S1 | 277.0 | 251.0 | 26.0 | 20.132340 | 42.844433 | 160.628766 | 0.072680 | 0.170695 | 0.579887 | S |
| 1 | S2 | 264.0 | 286.0 | -22.0 | 13.690763 | 17.261867 | 174.521394 | 0.051859 | 0.060356 | 0.661066 | S |
| 2 | S3 | 260.0 | 292.0 | -32.0 | 23.394849 | 20.271424 | 111.796278 | 0.089980 | 0.069423 | 0.429986 | S |
Example of further data analysis with the prov_fin_df DataFrame
prov_fin_df.groupby("nation")[["sqdiff_imm_sc", "imm_prov"]].corr()
| sqdiff_imm_sc | imm_prov | ||
|---|---|---|---|
| nation | |||
| E | sqdiff_imm_sc | 1.000000 | -0.855763 |
| imm_prov | -0.855763 | 1.000000 | |
| S | sqdiff_imm_sc | 1.000000 | -0.172818 |
| imm_prov | -0.172818 | 1.000000 | |
| W | sqdiff_imm_sc | 1.000000 | 0.932312 |
| imm_prov | 0.932312 | 1.000000 |
National Profile / Year-on-Year demos#
Run the national_profile pipeline
from pystagegate.pipelines import national_profile
year_agg_df, year_agg_adjusted_df = national_profile(config_path)
Preview the sum of squared differences between two years of final migration estimates:
year_agg_df
| Local Authority Code | Age | count_imm_2024 | count_em_2024 | count_net_2024 | total_count_imm_2024 | total_count_em_2024 | total_count_net_2024 | count_imm_2025 | count_em_2025 | count_net_2025 | total_count_imm_2025 | total_count_em_2025 | total_count_net_2025 | diff_imm | sqdiff_imm | diff_em | sqdiff_em | diff_net | sqdiff_net | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | E1 | 25 | 22 | 18.0 | 4.0 | 192 | 214.0 | -22.0 | 20 | 18.0 | 2.0 | 187 | 197.0 | -10.0 | -1.427083 | 2.036567 | 1.429907 | 2.044633 | 0.181818 | 0.033058 |
| 1 | E1 | 26 | 24 | 35.0 | -11.0 | 196 | 204.0 | -8.0 | 30 | 21.0 | 9.0 | 187 | 195.0 | -8.0 | 7.102041 | 50.438984 | -12.455882 | 155.149005 | 20.000000 | 400.000000 |
| 2 | E1 | 27 | 19 | 18.0 | 1.0 | 184 | 186.0 | -2.0 | 23 | 25.0 | -2.0 | 194 | 194.0 | 0.0 | 2.967391 | 8.805411 | 6.225806 | 38.760666 | -2.000000 | 4.000000 |
| 3 | E1 | 28 | 23 | 20.0 | 3.0 | 222 | 197.0 | 25.0 | 28 | 18.0 | 10.0 | 213 | 208.0 | 5.0 | 5.932432 | 35.193755 | -3.116751 | 9.714138 | 9.400000 | 88.360000 |
| 4 | E1 | 29 | 24 | 18.0 | 6.0 | 200 | 213.0 | -13.0 | 25 | 29.0 | -4.0 | 184 | 223.0 | -39.0 | 2.920000 | 8.526400 | 10.154930 | 103.122595 | -22.000000 | 484.000000 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 85 | W3 | 30 | 19 | 26.0 | -7.0 | 197 | 195.0 | 2.0 | 17 | 24.0 | -7.0 | 206 | 200.0 | 6.0 | -2.868020 | 8.225540 | -2.666667 | 7.111111 | 14.000000 | 196.000000 |
| 86 | W3 | 31 | 25 | 23.0 | 2.0 | 220 | 192.0 | 28.0 | 19 | 24.0 | -5.0 | 186 | 192.0 | -6.0 | -2.136364 | 4.564050 | 1.000000 | 1.000000 | -4.571429 | 20.897959 |
| 87 | W3 | 32 | 29 | 21.0 | 8.0 | 225 | 205.0 | 20.0 | 20 | 29.0 | -9.0 | 195 | 200.0 | -5.0 | -5.133333 | 26.351111 | 8.512195 | 72.457466 | -7.000000 | 49.000000 |
| 88 | W3 | 33 | 33 | 13.0 | 20.0 | 192 | 192.0 | 0.0 | 17 | 23.0 | -6.0 | 174 | 205.0 | -31.0 | -12.906250 | 166.571289 | 9.119792 | 83.170600 | inf | 0.000000 |
| 89 | W3 | 34 | 18 | 22.0 | -4.0 | 189 | 216.0 | -27.0 | 22 | 25.0 | -3.0 | 197 | 226.0 | -29.0 | 3.238095 | 10.485261 | 1.981481 | 3.926269 | 1.296296 | 1.680384 |
90 rows × 20 columns
Preview the adjusted sum of squared differences between two years of final migration estimates:
year_agg_adjusted_df
| Local Authority Code | imm_ssq_total | em_ssq_total | net_ssq_total | imm_la_total | em_la_total | net_la_total | imm_adjusted_size_ssq | em_adjusted_size_ssq | net_adjusted_size_ssq | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | E1 | 213.340923 | 432.980808 | 1077.241524 | 222 | 225.0 | -3.0 | 0.960995 | 1.924359 | -359.080508 |
| 1 | E2 | 267.170004 | 689.416874 | 1309.935307 | 217 | 230.0 | -13.0 | 1.231198 | 2.997465 | -100.764254 |
| 2 | E3 | 500.412400 | 368.713167 | 1041.212930 | 236 | 231.0 | 5.0 | 2.120392 | 1.596161 | 208.242586 |
| 3 | S1 | 382.228731 | 323.920615 | 2066.807917 | 232 | 214.0 | 18.0 | 1.647538 | 1.513648 | 114.822662 |
| 4 | S2 | 344.750336 | 728.784788 | 2237.250829 | 225 | 246.0 | -21.0 | 1.532224 | 2.962540 | -106.535754 |
| 5 | S3 | 260.512934 | 445.041690 | 1382.669919 | 205 | 248.0 | -43.0 | 1.270795 | 1.794523 | -32.155114 |
| 6 | W1 | 80.797431 | 283.287736 | 1213.396101 | 227 | 203.0 | 24.0 | 0.355936 | 1.395506 | 50.558171 |
| 7 | W2 | 377.240025 | 377.685041 | 1147.034931 | 239 | 192.0 | 47.0 | 1.578410 | 1.967110 | 24.404999 |
| 8 | W3 | 303.724806 | 348.991981 | 375.970409 | 214 | 225.0 | -11.0 | 1.419275 | 1.551075 | -34.179128 |
Sex Ratio demos#
Run the sex_ratio_main pipeline
from pystagegate.pipelines import sex_ratio_main
sr_df, sr_national_df, sr_merged_df = sex_ratio_main(config_path)
Preview sex ratios for Male and Female estimates between two years of final migration estimates:
sr_df
| em_fin | imm_fin | sex_ratio_imm_2024 | sex_ratio_em_2024 | sex_ratio_imm_2025 | sex_ratio_em_2025 | imm_weight | em_weight | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sex | Female | Male | Female | Male | |||||||||||
| Year | 2024 | 2025 | 2024 | 2025 | 2024 | 2025 | 2024 | 2025 | |||||||
| Local Authority Code | Age | ||||||||||||||
| E1 | 25 | 7.0 | 9.0 | 11.0 | 9.0 | 12.0 | 12.0 | 10.0 | 8.0 | 0.833333 | 1.571429 | 0.666667 | 1.000000 | 42.0 | 36.0 |
| 26 | 22.0 | 12.0 | 13.0 | 9.0 | 6.0 | 13.0 | 18.0 | 17.0 | 3.000000 | 0.590909 | 1.307692 | 0.750000 | 54.0 | 56.0 | |
| 27 | 8.0 | 13.0 | 10.0 | 12.0 | 6.0 | 14.0 | 13.0 | 9.0 | 2.166667 | 1.250000 | 0.642857 | 0.923077 | 42.0 | 43.0 | |
| 28 | 8.0 | 5.0 | 12.0 | 13.0 | 9.0 | 17.0 | 14.0 | 11.0 | 1.555556 | 1.500000 | 0.647059 | 2.600000 | 51.0 | 38.0 | |
| 29 | 11.0 | 13.0 | 7.0 | 16.0 | 13.0 | 15.0 | 11.0 | 10.0 | 0.846154 | 0.636364 | 0.666667 | 1.230769 | 49.0 | 47.0 | |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| W3 | 30 | 18.0 | 14.0 | 8.0 | 10.0 | 9.0 | 11.0 | 10.0 | 6.0 | 1.111111 | 0.444444 | 0.545455 | 0.714286 | 36.0 | 50.0 |
| 31 | 9.0 | 11.0 | 14.0 | 13.0 | 9.0 | 10.0 | 16.0 | 9.0 | 1.777778 | 1.555556 | 0.900000 | 1.181818 | 44.0 | 47.0 | |
| 32 | 9.0 | 17.0 | 12.0 | 12.0 | 13.0 | 9.0 | 16.0 | 11.0 | 1.230769 | 1.333333 | 1.222222 | 0.705882 | 49.0 | 50.0 | |
| 33 | 6.0 | 13.0 | 7.0 | 10.0 | 11.0 | 8.0 | 22.0 | 9.0 | 2.000000 | 1.166667 | 1.125000 | 0.769231 | 50.0 | 36.0 | |
| 34 | 13.0 | 13.0 | 9.0 | 12.0 | 10.0 | 10.0 | 8.0 | 12.0 | 0.800000 | 0.692308 | 1.200000 | 0.923077 | 40.0 | 47.0 | |
90 rows × 14 columns
Preview sex ratios for Male and Female estimates between two years of final migration estimates, aggregated to national level:
sr_national_df
| em_fin | imm_fin | sex_ratio_imm_2024 | sex_ratio_em_2024 | sex_ratio_imm_2025 | sex_ratio_em_2025 | imm_weight | em_weight | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sex | Female | Male | Female | Male | ||||||||||
| Year | 2024 | 2025 | 2024 | 2025 | 2024 | 2025 | 2024 | 2025 | ||||||
| Age | ||||||||||||||
| 25 | 106.0 | 102.0 | 108.0 | 95.0 | 96.0 | 91.0 | 96.0 | 96.0 | 1.000000 | 1.018868 | 1.054945 | 0.931373 | 379.0 | 411.0 |
| 26 | 108.0 | 111.0 | 96.0 | 84.0 | 94.0 | 94.0 | 102.0 | 93.0 | 1.085106 | 0.888889 | 0.989362 | 0.756757 | 383.0 | 399.0 |
| 27 | 96.0 | 109.0 | 90.0 | 85.0 | 90.0 | 102.0 | 94.0 | 92.0 | 1.044444 | 0.937500 | 0.901961 | 0.779817 | 378.0 | 380.0 |
| 28 | 89.0 | 101.0 | 108.0 | 107.0 | 112.0 | 102.0 | 110.0 | 111.0 | 0.982143 | 1.213483 | 1.088235 | 1.059406 | 435.0 | 405.0 |
| 29 | 105.0 | 112.0 | 108.0 | 111.0 | 103.0 | 107.0 | 97.0 | 77.0 | 0.941748 | 1.028571 | 0.719626 | 0.991071 | 384.0 | 436.0 |
| 30 | 96.0 | 107.0 | 99.0 | 93.0 | 95.0 | 115.0 | 102.0 | 91.0 | 1.073684 | 1.031250 | 0.791304 | 0.869159 | 403.0 | 395.0 |
| 31 | 90.0 | 98.0 | 102.0 | 94.0 | 95.0 | 94.0 | 125.0 | 92.0 | 1.315789 | 1.133333 | 0.978723 | 0.959184 | 406.0 | 384.0 |
| 32 | 99.0 | 105.0 | 106.0 | 95.0 | 120.0 | 86.0 | 105.0 | 109.0 | 0.875000 | 1.070707 | 1.267442 | 0.904762 | 420.0 | 405.0 |
| 33 | 99.0 | 115.0 | 93.0 | 90.0 | 92.0 | 86.0 | 100.0 | 88.0 | 1.086957 | 0.939394 | 1.023256 | 0.782609 | 366.0 | 397.0 |
| 34 | 117.0 | 103.0 | 99.0 | 123.0 | 102.0 | 100.0 | 87.0 | 97.0 | 0.852941 | 0.846154 | 0.970000 | 1.194175 | 386.0 | 442.0 |
Preview sex ratios for Male and Female estimates between two years of final migration estimates, merged with national aggregates:
sr_merged_df
| Age | em_fin_local | imm_fin_local | sex_ratio_imm_2024_local | ... | imm_fin_national | sex_ratio_imm_2024_national | sex_ratio_em_2024_national | sex_ratio_imm_2025_national | sex_ratio_em_2025_national | imm_weight_national | em_weight_national | ssq_imm | ssq_em | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sex | Female | Male | Female | Male | ... | Male | |||||||||||||||
| Year | 2024 | 2025 | 2024 | 2025 | 2024 | 2025 | 2024 | 2025 | ... | 2024 | 2025 | ||||||||||
| Local Authority Code | |||||||||||||||||||||
| E1 | 25 | 7.0 | 9.0 | 11.0 | 9.0 | 12.0 | 12.0 | 10.0 | 8.0 | 0.833333 | ... | 96.0 | 96.0 | 1.000000 | 1.018868 | 1.054945 | 0.931373 | 379.0 | 411.0 | 2.062694 | 8.430888 |
| E1 | 26 | 22.0 | 12.0 | 13.0 | 9.0 | 6.0 | 13.0 | 18.0 | 17.0 | 3.000000 | ... | 102.0 | 93.0 | 1.085106 | 0.888889 | 0.989362 | 0.756757 | 383.0 | 399.0 | 137.646726 | 4.749408 |
| E1 | 27 | 8.0 | 13.0 | 10.0 | 12.0 | 6.0 | 14.0 | 13.0 | 9.0 | 2.166667 | ... | 94.0 | 92.0 | 1.044444 | 0.937500 | 0.901961 | 0.779817 | 378.0 | 380.0 | 80.138568 | 1.231608 |
| E1 | 28 | 8.0 | 5.0 | 12.0 | 13.0 | 9.0 | 17.0 | 14.0 | 11.0 | 1.555556 | ... | 110.0 | 111.0 | 0.982143 | 1.213483 | 1.088235 | 1.059406 | 435.0 | 405.0 | 52.498950 | 59.762966 |
| E1 | 29 | 11.0 | 13.0 | 7.0 | 16.0 | 13.0 | 15.0 | 11.0 | 10.0 | 0.846154 | ... | 97.0 | 77.0 | 0.941748 | 1.028571 | 0.719626 | 0.991071 | 384.0 | 436.0 | 0.089066 | 18.767320 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| W3 | 30 | 18.0 | 14.0 | 8.0 | 10.0 | 9.0 | 11.0 | 10.0 | 6.0 | 1.111111 | ... | 102.0 | 91.0 | 1.073684 | 1.031250 | 0.791304 | 0.869159 | 403.0 | 395.0 | 2.888845 | 9.328280 |
| W3 | 31 | 9.0 | 11.0 | 14.0 | 13.0 | 9.0 | 10.0 | 16.0 | 9.0 | 1.777778 | ... | 125.0 | 92.0 | 1.315789 | 1.133333 | 0.978723 | 0.959184 | 406.0 | 384.0 | 12.864243 | 1.872257 |
| W3 | 32 | 9.0 | 17.0 | 12.0 | 12.0 | 13.0 | 9.0 | 16.0 | 11.0 | 1.230769 | ... | 105.0 | 109.0 | 0.875000 | 1.070707 | 1.267442 | 0.904762 | 420.0 | 405.0 | 7.878812 | 10.649381 |
| W3 | 33 | 6.0 | 13.0 | 7.0 | 10.0 | 11.0 | 8.0 | 22.0 | 9.0 | 2.000000 | ... | 100.0 | 88.0 | 1.086957 | 0.939394 | 1.023256 | 0.782609 | 366.0 | 397.0 | 32.910327 | 2.084859 |
| W3 | 34 | 13.0 | 13.0 | 9.0 | 12.0 | 10.0 | 10.0 | 8.0 | 12.0 | 0.800000 | ... | 87.0 | 97.0 | 0.852941 | 0.846154 | 0.970000 | 1.194175 | 386.0 | 442.0 | 3.202228 | 0.646154 |
90 rows × 31 columns
Example of further data analysis with the merged_sr_df DataFrame:
sr_merged_df.groupby(level=0).agg(sum)[["ssq_imm", "ssq_em"]]
/tmp/ipykernel_2829/1515162787.py:1: FutureWarning: The provided callable <built-in function sum> is currently using DataFrameGroupBy.sum. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "sum" instead.
sr_merged_df.groupby(level=0).agg(sum)[["ssq_imm", "ssq_em"]]
| ssq_imm | ssq_em | |
|---|---|---|
| Sex | ||
| Year | ||
| Local Authority Code | ||
| E1 | 313.912732 | 116.685444 |
| E2 | 123.353946 | 67.838701 |
| E3 | 142.734727 | 38.792242 |
| S1 | 224.866963 | 416.252075 |
| S2 | 459.953344 | 202.842431 |
| S3 | 95.305516 | 751.762137 |
| W1 | 177.247394 | 140.331897 |
| W2 | 202.883520 | 51.858552 |
| W3 | 105.264645 | 89.316616 |