Survey Weight Calibration: A Realistic Example¶
This example demonstrates how to use fairlex for calibrating survey weights in a realistic scenario. We’ll simulate a political opinion survey with typical demographic biases and calibrate against US Census benchmarks.
Scenario¶
A polling organization conducted a survey with 200 respondents to gauge public opinion. Like most surveys, the sample has demographic biases:
Over-representation of older, higher-educated respondents
Under-representation of Hispanic and younger demographics
Geographic skew toward certain regions
We’ll use leximin calibration to adjust the weights to match known population demographics from the 2023 US Census.
import numpy as np
import pandas as pd
from fairlex import calibrate
# Set random seed for reproducibility
np.random.seed(42)
Step 1: Create Realistic Survey Data¶
We’ll simulate survey respondents with demographic characteristics that exhibit typical survey biases.
n_respondents = 200
# Generate biased survey sample
# Age groups: 18-29, 30-44, 45-64, 65+
age_groups = np.random.choice(
["18-29", "30-44", "45-64", "65+"],
size=n_respondents,
p=[0.15, 0.20, 0.35, 0.30], # Skewed toward older respondents
)
# Gender: Male, Female
gender = np.random.choice(
["Male", "Female"],
size=n_respondents,
p=[0.48, 0.52], # Close to population
)
# Race/Ethnicity
race_ethnicity = np.random.choice(
["White_NH", "Black", "Hispanic", "Asian", "Other"],
size=n_respondents,
p=[
0.70,
0.11,
0.10,
0.06,
0.03,
], # Under-representation of Hispanic, over-representation of White
)
# Education: HS_or_less, Some_college, Bachelor_plus
education = np.random.choice(
["HS_or_less", "Some_college", "Bachelor_plus"],
size=n_respondents,
p=[0.25, 0.30, 0.45], # Over-representation of college educated
)
# Region: Northeast, Midwest, South, West
region = np.random.choice(
["Northeast", "Midwest", "South", "West"],
size=n_respondents,
p=[0.20, 0.22, 0.35, 0.23], # Roughly representative
)
# Create DataFrame
survey_data = pd.DataFrame(
{
"age_group": age_groups,
"gender": gender,
"race_ethnicity": race_ethnicity,
"education": education,
"region": region,
}
)
print("Survey Sample Demographics:")
print("===========================")
for col in survey_data.columns:
print(f"\n{col.replace('_', ' ').title()}:")
print(survey_data[col].value_counts(normalize=True).round(3))
Survey Sample Demographics:
===========================
Age Group:
age_group
65+ 0.300
45-64 0.290
30-44 0.245
18-29 0.165
Name: proportion, dtype: float64
Gender:
gender
Female 0.555
Male 0.445
Name: proportion, dtype: float64
Race Ethnicity:
race_ethnicity
White_NH 0.670
Hispanic 0.130
Asian 0.080
Black 0.075
Other 0.045
Name: proportion, dtype: float64
Education:
education
Bachelor_plus 0.410
Some_college 0.315
HS_or_less 0.275
Name: proportion, dtype: float64
Region:
region
South 0.34
Midwest 0.24
Northeast 0.23
West 0.19
Name: proportion, dtype: float64
Step 2: Define Population Benchmarks¶
These targets are based on 2023 US Census data and represent the true population distributions we want to match.
# Population benchmarks from 2023 US Census data
population_targets = {
# Age distribution (approximate from Census data)
"age_18_29": 0.18,
"age_30_44": 0.25,
"age_45_64": 0.32,
"age_65_plus": 0.25,
# Gender distribution
"male": 0.495,
"female": 0.505,
# Race/Ethnicity distribution
"white_nh": 0.582,
"black": 0.120,
"hispanic": 0.190,
"asian": 0.058,
"other_race": 0.050,
# Education distribution (adults 25+, approximate)
"hs_or_less": 0.38,
"some_college": 0.28,
"bachelor_plus": 0.34,
# Regional distribution
"northeast": 0.17,
"midwest": 0.21,
"south": 0.38,
"west": 0.24,
}
print("Population Targets (2023 US Census):")
print("====================================")
for category, target in population_targets.items():
print(f"{category.replace('_', ' ').title()}: {target:.1%}")
Population Targets (2023 US Census):
====================================
Age 18 29: 18.0%
Age 30 44: 25.0%
Age 45 64: 32.0%
Age 65 Plus: 25.0%
Male: 49.5%
Female: 50.5%
White Nh: 58.2%
Black: 12.0%
Hispanic: 19.0%
Asian: 5.8%
Other Race: 5.0%
Hs Or Less: 38.0%
Some College: 28.0%
Bachelor Plus: 34.0%
Northeast: 17.0%
Midwest: 21.0%
South: 38.0%
West: 24.0%
Step 3: Calibrate¶
Targets go in as {variable: {level: target}}. These are shares of the population, so we pass shares=True and the population size. Every weight may move between 0.2 and 5 times its base weight.
variable_keys = {
"age_group": {
"18-29": "age_18_29",
"30-44": "age_30_44",
"45-64": "age_45_64",
"65+": "age_65_plus",
},
"gender": {"Male": "male", "Female": "female"},
"race_ethnicity": {
"White_NH": "white_nh",
"Black": "black",
"Hispanic": "hispanic",
"Asian": "asian",
"Other": "other_race",
},
"education": {
"HS_or_less": "hs_or_less",
"Some_college": "some_college",
"Bachelor_plus": "bachelor_plus",
},
"region": {
"Northeast": "northeast",
"Midwest": "midwest",
"South": "south",
"West": "west",
},
}
targets = {
var: {level: population_targets[key] for level, key in levels.items()}
for var, levels in variable_keys.items()
}
report = calibrate(
survey_data, targets, shares=True, total=n_respondents, bounds=(0.2, 5.0)
)
print(report)
largest miss: 0.00% of target largest weight change: 60.0% ESS: 159 design effect: 1.26
variable level target before after miss miss_pct
age_group 18-29 36.0 33.0 36.0 0.0 +0.00%
age_group 30-44 50.0 49.0 50.0 0.0 +0.00%
age_group 45-64 64.0 58.0 64.0 0.0 +0.00%
age_group 65+ 50.0 60.0 50.0 0.0 +0.00%
gender Male 99.0 89.0 99.0 0.0 +0.00%
gender Female 101.0 111.0 101.0 0.0 +0.00%
race_ethnicity White_NH 116.4 134.0 116.4 0.0 +0.00%
race_ethnicity Black 24.0 15.0 24.0 0.0 +0.00%
race_ethnicity Hispanic 38.0 26.0 38.0 0.0 +0.00%
race_ethnicity Asian 11.6 16.0 11.6 0.0 +0.00%
race_ethnicity Other 10.0 9.0 10.0 0.0 +0.00%
education HS_or_less 76.0 55.0 76.0 0.0 +0.00%
education Some_college 56.0 63.0 56.0 0.0 +0.00%
education Bachelor_plus 68.0 82.0 68.0 0.0 +0.00%
region Northeast 34.0 46.0 34.0 0.0 +0.00%
region Midwest 42.0 48.0 42.0 0.0 +0.00%
region South 76.0 68.0 76.0 0.0 +0.00%
region West 48.0 38.0 48.0 0.0 +0.00%
(total) all 200.0 200.0 200.0 0.0 +0.00%
Every target is hit. The report also shows the price: the largest change to any weight and the design effect, which says how much the weighting inflates variance.
Step 4: When the targets cannot all be met¶
Tighter bounds, 0.8 to 1.25 times the base weight, stop the weights from correcting the sample’s biases fully. fairlex then makes the largest relative miss as small as the bounds allow, then the next largest, and so on, and names the targets with the worst miss and why they miss.
tight = calibrate(
survey_data, targets, shares=True, total=n_respondents, bounds=(0.8, 1.25)
)
print(tight)
largest miss: 21.88% of target largest weight change: 25.0% ESS: 191 design effect: 1.05
worst-missed targets: race_ethnicity=Black (weight bounds)
variable level target before after miss miss_pct
age_group 18-29 36.0 33.0 33.6 -2.4 -6.66%
age_group 30-44 50.0 49.0 49.6 -0.4 -0.77%
age_group 45-64 64.0 58.0 58.5 -5.5 -8.61%
age_group 65+ 50.0 60.0 57.0 7.0 +14.00%
gender Male 99.0 89.0 90.5 -8.5 -8.60%
gender Female 101.0 111.0 108.2 7.2 +7.15%
race_ethnicity White_NH 116.4 134.0 126.4 10.0 +8.61%
race_ethnicity Black 24.0 15.0 18.7 -5.3 -21.88%
race_ethnicity Hispanic 38.0 26.0 30.8 -7.2 -19.03%
race_ethnicity Asian 11.6 16.0 13.3 1.7 +14.78%
race_ethnicity Other 10.0 9.0 9.5 -0.5 -5.49%
education HS_or_less 76.0 55.0 64.8 -11.2 -14.78%
education Some_college 56.0 63.0 58.0 2.0 +3.55%
education Bachelor_plus 68.0 82.0 76.0 8.0 +11.69%
region Northeast 34.0 46.0 40.5 6.5 +19.03%
region Midwest 42.0 48.0 46.6 4.6 +10.87%
region South 76.0 68.0 69.6 -6.4 -8.46%
region West 48.0 38.0 42.1 -5.9 -12.29%
(total) all 200.0 200.0 198.7 -1.3 -0.65%
Black respondents would need weights about 1.6 times their base weight to reach their target, beyond the 1.25 cap, so the report blames the weight bounds rather than another target.
Step 5: Priorities¶
If your outcomes depend strongly on race, say so. importance={"race_ethnicity": 4} makes a miss on a race margin count four times as much, so race is held closer to its targets and the other variables absorb more of the miss.
favoured = calibrate(
survey_data,
targets,
shares=True,
total=n_respondents,
bounds=(0.8, 1.25),
importance={"race_ethnicity": 4},
)
comparison = tight.margins[["variable", "level", "miss_pct"]].assign(
with_race_priority=favoured.margins["miss_pct"]
)
print(comparison.round(1).to_string(index=False))
variable level miss_pct with_race_priority
age_group 18-29 -6.7 -7.8
age_group 30-44 -0.8 -4.5
age_group 45-64 -8.6 -8.5
age_group 65+ 14.0 11.2
gender Male -8.6 -8.5
gender Female 7.1 3.5
race_ethnicity White_NH 8.6 4.5
race_ethnicity Black -21.9 -21.9
race_ethnicity Hispanic -19.0 -14.5
race_ethnicity Asian 14.8 10.3
race_ethnicity Other -5.5 -5.5
education HS_or_less -14.8 -17.9
education Some_college 3.6 3.7
education Bachelor_plus 11.7 9.7
region Northeast 19.0 24.1
region Midwest 10.9 8.6
region South -8.5 -10.7
region West -12.3 -17.9
(total) all -0.6 -2.5
The White, Hispanic and Asian misses shrink, while region and education take up the slack. The Black target does not move: it was already as close as the weight bounds allow, and no priority can change that.
Summary¶
calibratetakes a data frame and targets per variable and returns weights plus a report.When targets can be met, it meets them and moves weights as little and as evenly as possible.
When they cannot, it shares the misses out as evenly as possible across targets, relative to their size, and tells you which targets bind and why.
importancesays which variables matter most.For standard errors, calibrate replicate weights with
calibrate_replicates; see the “Standard errors” page.