API Reference

Run the GDHI adjustment pipeline.

Parameters:

Calculates the interquartile range (IQR) for each LSOA in the DataFrame.

Parameters:

Returns:

Calculates the mean GDHI for each non outlier LSOA in the DataFrame.

Parameters:

Returns:

Calculate the rate of change going forward and backwards in time in the DataFrame.

Parameters:

Returns:

Calculates the z-scores for percent changes and raw data in DataFrame.

Parameters:

Returns:

Creates a master flag based on z score and IQR flag columns.

Parameters:

Returns:

Extracts the start and end years from the column headings. Args: df (pd.DataFrame): The input DataFrame with years as headers. Returns: Tuple[int, int]: A tuple containing the start and end years.

Flags years where the GDHI has rolled back from future years. Typically 2010-2014 has 2015 data copied to them as it is missing.

Parameters:

Returns:

Concatenates two wide dataframes to create a final wide DataFrame.

Parameters:

Returns:

Calculate contrained and unconstrained values for each outlier case.

Parameters:

Returns:

Pivots the output DataFrame to long format. Args: df (pd.DataFrame): The input DataFrame in wide format. uncon_gdhi (str): The column name for unconstrained GDHI. con_gdhi (str): The column name for constrained GDHI.

Returns:

Pivots the DataFrame from long to wide format.

Parameters:

Returns:

Pivots the DataFrame based on specified index, columns, and values.

Parameters:

Returns: pd.DataFrame: The pivoted DataFrame.

Run the preprocessing steps for the GDHI adjustment project.

This function performs the following steps: 1. Load the configuration settings. 2. Load the input data. 3. Pivot the DataFrame to long format. 4. Calculate percentage rate of change and flag rollback years. 5. Calculate z-scores and IQRs if desired as per config. 6. Create master flags. 7. Save interim data with all calculated values. 8. Calculate LAD mean GDHI. 9. Constrain outliers to regional accounts. 10. Pivot the DataFrame back to wide format. 11. Save the preprocessed data ready for PowerBI analysis.

Parameters:

Returns: None: The function does not return any value. It saves the processed DataFrame to a CSV file.

Apportion the adjustment values to all years for each LSOA.

Parameters:

Returns:

Change negative values to 0 and apportion negative adjustment values to all LSOAs within an LAD/year group.

Parameters:

Returns:

Continue to apportion the adjustments for years that are flagged as rollback years.

Parameters:

Returns:

Calculate the proportion of a non-outlier LSOA to the LAD for each year.

Parameters:

Returns:

Check that adjusted_con_gdhi has no negative values.

Parameters:

Raises:

Calculate the imputed value for a given LSOA code where the year that has been flagged as an outlier to adjust only has one valid safe year either side.

The imputed value is extrapolated from the nearest safe year and the year 4 years after. This is to avoid short term fluctuations.

Parameters:

Returns:

Calculate the imputed value for a given LSOA code where the year that has been flagged as an outlier to adjust has a valid safe year either side.

Parameters:

Returns: pd.DataFrame: DataFrame containing outlier imputed values.

Filter data to keep only LSOAs for adjustment and subset.

Parameters:

Returns:

Filter DataFrame by component codes.

Parameters:

Returns:

Filter DataFrame by a range of years inclusively. Args: df (pd.DataFrame): Input DataFrame containing year data. start_year (int): Start year for filtering (inclusive). end_year (int): End year for filtering (inclusive). Returns: pd.DataFrame: Filtered DataFrame containing only rows within the year range.

Identify safe years for each LSOA where no adjustment is needed.

For sequential years flagged for adjustment, the previous and next safe years are located at the end of the sequence of years.

For end of range years flagged for adjustment, it will return one safe year in the range, and one outside, which will return NaN for con_gdhi

Parameters:

Returns: df (pd.DataFrame): DataFrame with additional columns for safe years. safe_years_df (pd.DataFrame): DataFrame containing only the rows that need adjustment with non-outlier year values either side of outlier years.

Join analyst data to constrained data based on LSOA code and LAD code.

Parameters:

Returns:

Join analyst data to unconstrained data based on LSOA code and LAD code.

Parameters:

Returns:

Un-pivot (melt) the adjustment DataFrame from wide to long format.

Parameters:

Returns:

Pivots the DataFrame from long to wide format.

Parameters:

Returns:

Reformat data within the adjust column.

Parameters:

Returns:

Reformat data within the year column.

Parameters:

Returns:

Run the adjustment steps for the GDHI adjustment project.

This function performs the following steps: 1. Load the configuration settings. 2. Load the input data. 3. Reformat adjust and year columns. 4. Filter of data for adjustment. 5. Join analyst output with constrained and unconstrained data. 6. Pivot the DataFrame to long format for manipulation. 7. Filter data by the specified year range. 8. Calculate the imputed gdhi values for outlier years. 9. Calculate adjustment values based on imputed gdhi. 10. Apportion adjustment values to all years. 11. Save interim data with all calculated values. 12. Pivot data to wide format for PowerBI QA reiteration. 13. Pivot final DataFrame to wide format for exporting. 14. Save the final adjusted data.

Parameters:

Returns: None: The function does not return any value. It saves the processed DataFrame to a CSV file.

Check that for LSOAs marked for adjustment, the year column is not empty.

Parameters:

Returns:

Raises:

Check that not all LSOAs within an LAD are flagged for adjustment.

This is so that there are some non-outlier LSOAs to calculate non-outlier proportions of the total GDHI within an LAD.

Parameters:

Returns:

Raises:

Check that not all years within an LSOA are flagged for adjustment.

This is so that there are some non-outlier LSOAs to interpolate/extrapolate from.

Parameters:

Returns:

Raises:

Aggregate values on LADs and other identifiers.

Parameters:

Returns:

Subset the mapper and get a unique DataFrame of values.

Parameters:

Returns:

Join mapper containing a lookup of LAU and LAD values, to adjusted data.

Parameters:

Returns:

Run the mapping steps for the GDHI adjustment pipeline.

This function performs the follwing steps: 1. Rename column containing S30 values to LAU and verify mapping is required. If mapping is required: 2. Load in and clean mapper containing LAU to LAD lookup. 3. Join LAU-LAD mapper to adjusted data. 4. Aggregate to LAD if specified in config. 5. Reformat output.

Rename LAD columns for end format.

Parameters:

Returns:

Rename column containing S30 area codes to lau_

Parameters:

Returns:

Append all DataFrames that contain separatesub-components together so that each LSOA has all sub-components present in one DataFrame.

Parameters:

Returns:

Set cells in target_cols to "X" where both conditions are met: - The value in transaction_col equals transaction_value. - The value in lsoa_col starts with any values in lsoa_val list.

Parameters:

Returns:

Performs an internal consistency check on the DataFrame to ensure 'lsoa_code' uniqueness matches the total row count.

This function verifies that the number of unique values in the 'lsoa_code' column is exactly equal to the number of rows in the DataFrame.

Parameters:

Returns:

Raises:

Perform a validation check to ensure that the unique lsoa_codes in the constrained DataFrame matches that in the unconstrained DataFrame.

Parameters:

Returns:

Raises:

Checks all numeric columns in the DataFrame to ensure no values are less than 0.

This function isolates numeric columns (integers and floats) and verifies that all values are non-negative. It ignores non-numeric columns (e.g., strings).

Parameters:

Returns:

Raises:

Checks the entire DataFrame to ensure it contains no Null, NaN, or None values.

This function scans all cells in the DataFrame. It detects standard numpy NaNs, Python None objects, and pandas pd.NA values.

If any such value is found, it raises a ValueError.

Parameters:

Returns:

Raises:

This function verifies that each unique value combination in the 'transaction' and 'account_entry' columns from the subcomponent lookup are present in the DataFrame.

Parameters:

Returns:

Raises:

Verifies that the DataFrame contains a complete set of consecutive year columns.

This function automatically identifies numeric column names (integers or strings representing integers), determines the minimum and maximum years, and checks if every year between that minimum and maximum exists as a column.

Parameters:

Returns:

Raises:

Run the CORD preparation steps for the GDHI adjustment project.

This function performs the following steps: 1. Load the configuration settings. 2. Load the input data and append all subcomponents together. 3. Map LAU S30 codes to LAD S12 codes. 4. Perform validation checks on the input data. 5. Apply CORD-specific transformations. 6. Save the prepared CORD data for further processing.

Parameters:

Returns:

Convert DataFrame columns data types as specified in the schema.

Parameters:

Returns:

Raises:

Load a schema from a TOML file.

Parameters:

Returns:

Load a .toml file from a path, with logging and safe error handling.

Parameters:

Returns:

Reads in a csv file and compares it to a data dictionary schema.

Parameters:

Returns:

Rename columns in the DataFrame based on the schema. Schema should be a dict where keys are new column names and values are dicts with 'old_name'.

Parameters:

Returns:

Validate the DataFrame against the schema.

Parameters:

Raises:

Writes a DataFrame to a CSV file, renaming columns and validating against a schema.

Parameters:

Raises:

Returns:

Ensure the input is returned as a list.

Parameters:

Returns: list: The input value wrapped in a list if it was not already a list.

Increase or decrease year until it is not in a list of adjust_years. Args: year (int): The starting year. adjust_years (list): List of years to avoid. limit_year (int): The limit year to stop at. is_increasing (bool): If True, increase year; if False, decrease year. Returns: int: The first year not in adjust_years list.

Check that the sums of adjusted column, matches that of the unadjusted column for the same groupings.

If the difference exceeds a specified tolerance, raise an error.

Parameters:

Returns:

Convert a cell to a list of ints. Accepts: - a comma-separated string like "2010,2011, 2012" - a list/tuple of strings or numbers - NaN/None -> returns [] Raises ValueError if an item cannot be converted to int.