Selective Editing in Python

Selective Editing in Python#

SML User Guide#

Overview#

Descriptive

Details

Support Area

Methodology - Editing & Imputation

Method Theme

Editing

Status

Ready to Use

Inputs

Reference, question list, adjusted return, predicted value, auxillary predicted value, standardising factor, design weight, threshold

Outputs

Reference, Score1, ScoreM, Final_Score, Selective Editing Marker, Predicted Marker

Method Version

1.3.2

Code Repository

ONSdigital/sml-python-small

Summary#

Selective Editing is an internationally recognised editing validation method where suspicious values that are potential errors are prioritised according to their expected impact on key outputs. Selective Editing assigns a score to specified variables (questions) where the score reflects the impact the respondent’s values will have on the estimates. Contributors with a score above a predetermined threshold are flagged to be validated, while contributors with a score below the threshold are flagged as passed.

Terminology#

  • Contributor reference - Unique identifier assigned to each respondent.

  • Adjusted return - The most recent unedited returned data value (for a given variable (question)) in the current period, t.

  • Predicted value - The first predictor value (for a given variable (question)) for the current period adjusted return. Typically the clean adjusted return for the previous period.

  • Auxiliary predicted value - This is the secondary predictor (for a given variable (question)) for the current period adjusted return.

  • Standardising factor - The domain group estimate used to standardise scores within a given domain group.

  • Design weight - An a-weight supplied input values. These are not generated by this function.

  • Selective Editing domain group - Indicates which Selective Editing domain group a given respondent belongs to.

  • Selective Editing threshold - A supplied threshold, to compare the ultimate contributor score with. Each threshold is set individually for each domain group. This threshold is an input and not calculated by this function.

  • Combining function - If the method is applied to more than one variable (question) for a responder, then the scores can be combined in the following ways: Mean, Sum, Max, Weighted mean, Minkowski distance function.


User Notes#

Finding and Installing the method#

This method requires Python >=3.10, <3.15 and uses the Pandas package v2.2.3.

The method package can be installed from Artifactory/PyPI using the following code in the terminal or command prompt:

pip install sml_small

Alternatively, download the latest code from Github, then copy the sml_small folder to the relevant folder (for example “C:\ONSapps\My_Python\Python_3_8\Lib\site-packages\sml_small”).

In your code you can import the method using:

import sml_small.selective_editing as seled

Requirements and Dependencies#

This method requires input data supplied as a Pandas dataframe.

Assumptions and Validity#

  • The method will automatically assign adjusted return (ar), predicted value (pv), auxiliary predicted value (apv) and standardising factor (sf) based on the column names of the inputs.

  • Unless otherwise noted, fields must not contain Null values.

  • If the “weighted mean score” is selected to combine multiple variable (question) scores, then positive weights should be supplied as an input.

How to Use the Method#

Once the selective editing method is available on your computer you will be able to call the method and perform selective editing on a dataset.

The input dataset will need to contain a specific suffix structure for each question used in calculating the score. The adjusted return value will have ‘_ar’, the predicted value will have ‘_pv’, the auxiliary predicted value will have ‘_apv’, the standardising factor will have ‘_sf’ and the weight (if weighted scores are chosen) will have ‘_wt’, where the sum of ‘_wt’ has to add up to 1.

Method Input#

Input records must include the following fields of the correct types:

  • reference (any type): Unique to each respondent. Should not contain null values.

  • design_weight (numeric): An a-weight. Should not contain null values.

  • threshold (numeric): The selective editing threshold. A supplied threshold, to compare the ultimate contributor score with. Each threshold is set individually for each domain group. This threshold is an input and not calculated by this function. Should not contain null values.

  • question_1_ar (numeric): This is the adjusted return for question 1. The adjusted return has usually been through other editing strategies before Selective Editing.

  • question_1_pv (numeric): This is the predicted value for question 1, usually the respondent’s clean value from the previous period. May contain null values.

  • question_1_apv (numeric): This is the auxiliary predicted variable used when there is not a previous period value available. Should not contain null values.

  • question_1_sf (numeric): This is the standardising factor, which is the weighted domain estimate for the previous period. Should not contain null values.

  • question_1_wt (numeric): This column is a weight column, which is used when the weighted option is used as the combination method for multiple question scores and in this case should not contain null values. For a single variable (question) or where another combination for multiple scores is selected the column is not required (so may contain null values).

Example

Reference

design_weight

threshold

question_1_ar

question_1_pv

question_1_apv

question_1_sf

question_1_wt

49900001

20

0.6

800

424

800000

1

49900002

20

0.6

656

390

259

800000

1

49900003

20

0.6

997

773

912

800000

1

49900004

20

0.6

676

334

800000

1

49900005

20

0.6

632

871

684

800000

1

49900006

20

0.6

985

345

312

800000

1

49900007

20

0.6

468

963

773

800000

1

49900008

20

0.6

772

733

833

800000

1

49900009

20

0.6

621

673

898

800000

1

49900010

20

0.6

736

377

646

800000

1

Method Output#

Output records will contain the following new fields:

  • question_1_s: This is the score generated for question 1.

  • question_1_pm: This is a predicted marker, which indicates whether the predicted value (True) or the auxiliary value (False) has been used.

  • Final_score: The score after a combination method is applied to multiple variables /questions (if used). For multiple variables / questions this is the value that is compared to the threshold.

  • Selective_edting_marker: If the method suggests further validation is required (such as responder recontact), then the marker will be False (That is, the score is >= the threshold value). If it is not recommended for any further validation for the variables tested is required then the marker will be True.

Example

Reference

design_weight

threshold

question_1_ar

question_1_pv

question_1_apv

question_1_sf

question_1_s

question_1_pm

final_score

selective_editing_marker

49900001

20

0.6

800

424

800000

0.94

FALSE

0.94

FALSE

49900002

20

0.6

656

390

259

800000

0.665

TRUE

0.665

FALSE

49900003

20

0.6

997

773

912

800000

0.56

TRUE

0.56

TRUE

49900004

20

0.6

676

334

800000

0.855

FALSE

0.855

FALSE

49900005

20

0.6

632

871

684

800000

0.5975

TRUE

0.5975

TRUE

49900006

20

0.6

985

345

312

800000

1.6

TRUE

1.6

FALSE

49900007

20

0.6

468

963

773

800000

1.2375

TRUE

1.2375

FALSE

49900008

20

0.6

772

733

833

800000

0.0975

TRUE

0.0975

TRUE

49900009

20

0.6

621

673

898

800000

0.13

TRUE

0.13

TRUE

49900010

20

0.6

736

377

646

800000

0.8975

TRUE

0.8975

FALSE

Note that in this example, References 49900001 and 49900004 do not have predicted values available for the score calculation as the cells are blank. Thus, for these references question_1_pm is False and so the auxillary value is then used for these respondents. The other respondents have their predicted value available and therefore, question_1_pm is True.

Example (Synthetic) Data#

Files containing the example input and output data given above are supplied in the example_data section of this repository.

Input data:

selective_editing_input_data_example_1.csv

Expected output after running the worked example:

selective_editing_output_data_example_1.csv

Worked Example#

import pandas as pd
import sml_small.selective_editing as seled

# Location of csv file
datafile = "selective_editing_input_data_example_1.csv"

# Read in csv file above
df = pd.read_csv(datafile)

# Call the Selective Editing method
output = seled.selective_editing(input_dataframe = df, # DataFrame of the test data (above)
                                 reference_col = 'reference', # Reference column
                                 design_weight_col = 'design_weight', # Design weight column
                                 threshold_col = 'threshold', # Threshold column
                                 question_list = ['question_1'], # Question(s) we are performing Selective Editing on
                                 combination_method = 'maximum', # Type combination, will accept 'maximum', 'mean', 'weighted', 'minkowski', default = maximum
                                 minkowski_distance = 0, # Set to 0 if minkowski is not selected. Set to value of p if minkowski is selected above. 
                                 show_sums = 0) # Provides additional data on score calculations
                                 
output.to_csv("selective_editing_output_data_example_1.csv")

The output gets exported as a .csv file, which will be saved in your working directory.


Methodology - Process Flow#

A selective editing score is calculated for each reporting unit i in time t. The selective editing score is calculated by multiplying 100 by the design weight by the absolute value of the adjusted return for reporting unit i at time t subtracted from the predicted value for reporting unit i at time t, then all divided by the standardising factor.

Score = 100*{{a-weight*|current value-predicted value|}/Standardising factor}

The predicted value should be a clean response (that is, a free from error adjusted return) for reporting unit i at time t-1. Note that these clean responses may be imputed or constructed previous period data. If this value is not available, then the auxiliary predicted value for reporting unit i at time t is used.

The standardising factor is the weighted domain estimate for a given variable at time t-1, which is determines a respondent’s potential impact on key output estimates.

Combining Scores

If selective editing is applied to more than one variable (question) for each respondent, then a final score is calculated using one of the following combination methods:

  • Maximum score: The maximum score of all the variables.

  • Weighted mean score: Where each score is multiplied by a weight. These weights should be supplied and be positive values.

  • Mean score: The scores are added up and divided by the number of scores.

  • Minkowski distance: Each variable’s (question’s) score is raised to the power of p. For each responder all these values are then summed, and the sum total raised to the power of 1/p, where p >= 1.

Generate selective editing marker

Each final score is compared to the threshold associated with the respondent’s domain group.

If the final score is greater than or equal to the selective editing threshold, then the selective editing marker for the reporting unit is marked False and the respondent is recommended for further validation.

If the final score is less than the selective editing threshold, the selective editing marker for the reporting unit is marked True and the respondent does not require validation.

Assumptions and Vailidity#

  • All data inputs required by the method are available and are on a standardised basis.

  • The predicted value (where it exists) and auxiliary predicted value should both be good predictors of the returned data for time t. The predicted value may originate from a genuine clean returned value, an imputed value or a constructed value.

  • The predictor value in the score calculation is of sufficiently good quality.

  • Each respondent is classified into one mutually exclusive domain group.

  • The thresholds specified are valid and appropriate (>0). The domain group thresholds are an input and are not generated by this method.

Additional Information#

The ONS Statistical Methods Library at https://statisticalmethodslibrary.ons.gov.uk/ contains:

  • Further information about the methods including a link to the GitHub repository which contains detailed API information as part of the method code.

  • Information about other methods available through the library.

License#

Unless stated otherwise, the SML codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

The documentation is available under the terms of the Open Government 3.0 license.