SolasAI Disparity Calculations#

import solas_disparity as sd
import pandas as pd

Certain notebook environments have limited rendering functionality. Uncomment this cell as a potential workaround if plots are not displaying.

# import plotly.io as pio
# pio.renderers.default = "png"

It’s preferable to explicitly and specifically handle warnings. For the purposes of this notebook, we will filter out all warnings.

from warnings import simplefilter
simplefilter("ignore")

Some predictions were created using a tree model run on an HMDA dataset.

label = "Interest Rate"
data = pd.read_parquet("hmda_test.parquet")

Store commonly reused function arguments.

protected_groups = ["Black", "Asian", "Native American", "Hispanic", "Female"]
reference_groups = ["White", "White", "White", "Non-Hispanic", "Male"]
groups = sd.pgrg_ordered(
    protected_groups=protected_groups,
    reference_groups=reference_groups,
)
reused_arguments = dict(
    group_data=data[groups],
    protected_groups=protected_groups,
    reference_groups=reference_groups,
    group_categories=["Race", "Race", "Race", "Ethnicity", "Sex"],
    sample_weight=None,
)
binary_outcome = data["Prediction"] <= data["Prediction"].quantile(0.5)
binary_label = data[label] <= data[label].quantile(0.5)

Adverse Impact Ratio (AIR)#

air = sd.adverse_impact_ratio(
    outcome=binary_outcome,
    air_threshold=0.8,
    percent_difference_threshold=0.0,
    **reused_arguments,
)

air

Disparity Calculation: Adverse Impact Ratio

┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ Protected Groups                          │ Black, Asian, Native American, Hispanic, Female                     │
│ Reference Groups                          │ White, White, White, Non-Hispanic, Male                             │
│ Group Categories                          │ Race, Race, Race, Ethnicity, Sex                                    │
│ AIR Threshold                             │ 0.8                                                                 │
│ Percent Difference Threshold              │ 0.0                                                                 │
│ Shortfall Method                          │ to_reference_mean                                                   │
│ Affected Groups                           │ Hispanic                                                            │
│ Affected Reference                        │ Non-Hispanic                                                        │
│ Affected Categories                       │ Ethnicity                                                           │
└───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Adverse Impact Ratio Summary Table

* Percent Missing: Ethnicity: 13.68%, Race: 13.56%, Sex: 46.88%

Group	Reference Group	Group Category	Total	Favorable	Percent Favorable	Percent Difference Favorable	AIR	P-Values	Practically Significant	Shortfall
Black	White	Race	340.0	141.0	41.47%	9.70%	0.810	0.001	No
Asian	White	Race	327.0	243.0	74.31%	-23.14%	1.452	0.000	No
Native American	White	Race	20.0	9.0	45.00%	6.17%	0.879	0.657	No
White		Race	3,623.0	1,854.0	51.17%
Hispanic	Non-Hispanic	Ethnicity	508.0	167.0	32.87%	21.54%	0.604	0.000	Yes	109.4
Non-Hispanic		Ethnicity	3,808.0	2,072.0	54.41%
Female	Male	Sex	1,034.0	414.0	40.04%	9.78%	0.804	0.000	No
Male		Sex	1,622.0	808.0	49.82%

Standardized Mean Difference (SMD)#

smd = sd.standardized_mean_difference(
    outcome=data["Prediction"],
    label=data[label],
    smd_threshold=30,
    lower_score_favorable=True,
    **reused_arguments,
)

smd

Disparity Calculation: Standardized Mean Difference

┌────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────┐
│ Protected Groups                   │ Black, Asian, Native American, Hispanic, Female                            │
│ Reference Groups                   │ White, White, White, Non-Hispanic, Male                                    │
│ Group Categories                   │ Race, Race, Race, Ethnicity, Sex                                           │
│ SMD Threshold                      │ 30.0                                                                       │
│ SMD Denominator                    │ population                                                                 │
│ Lower Score Favorable              │ True                                                                       │
│ Affected Groups                    │ Hispanic                                                                   │
│ Affected Reference                 │ Non-Hispanic                                                               │
│ Affected Categories                │ Ethnicity                                                                  │
└────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────┘

Standardized Mean Difference Summary Table

* Percent Missing: Ethnicity: 13.68%, Race: 13.56%, Sex: 46.88%

Group	Reference Group	Group Category	Total	Average Label	Average Outcome	Difference in Average Outcome	SMD	P-Values	Practically Significant
Black	White	Race	340.0	0.05	0.05	0.000653	27.559	0.000	No
Asian	White	Race	327.0	0.04	0.05	-0.001668	-70.355	0.000	No
Native American	White	Race	20.0	0.05	0.05	0.000655	27.617	0.199	No
White		Race	3,623.0	0.05	0.05
Hispanic	Non-Hispanic	Ethnicity	508.0	0.05	0.05	0.001032	43.545	0.000	Yes
Non-Hispanic		Ethnicity	3,808.0	0.05	0.05
Female	Male	Sex	1,034.0	0.05	0.05	0.000395	16.655	0.000	No
Male		Sex	1,622.0	0.05	0.05

Adverse Impact Ratio by Quantile#

airq = sd.adverse_impact_ratio_by_quantile(
    outcome=data["Prediction"],
    air_threshold=0.8,
    percent_difference_threshold=0.0,
    quantiles=[decile / 10 for decile in range(1, 11)],
    lower_score_favorable=True,
    **reused_arguments,
)

airq

Disparity Calculation: Adverse Impact Ratio By Quantile

┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ Protected Groups                          │ Black, Asian, Native American, Hispanic, Female                     │
│ Reference Groups                          │ White, White, White, Non-Hispanic, Male                             │
│ Group Categories                          │ Race, Race, Race, Ethnicity, Sex                                    │
│ AIR Threshold                             │ 0.8                                                                 │
│ Percent Difference Threshold              │ 0.0                                                                 │
│ Lower Score Favorable                     │ True                                                                │
│ Affected Groups                           │ Black, Hispanic, Female                                             │
│ Affected Reference                        │ White, Non-Hispanic, Male                                           │
│ Affected Categories                       │ Race, Ethnicity, Sex                                                │
└───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Adverse Impact Ratio By Quantile Summary Table

Group	Quantile	Reference Group	Group Category	Quantile Cutoff	Observations	Percent Missing	Total	Favorable	Percent Favorable	Percent Difference Favorable	AIR	P-Values	Practically Significant
Black	10.0%	White	Race	0.044761	4,322	13.56%	340.0	13.0	3.82%	4.93%	0.437	0.001	Yes
Asian	10.0%	White	Race	0.044761	4,322	13.56%	327.0	91.0	27.83%	-19.08%	3.181	0.000	No
Native American	10.0%	White	Race	0.044761	4,322	13.56%	20.0	1.0	5.00%	3.75%	0.571	1.000	No
White	10.0%		Race	0.044761	4,322	13.56%	3,623.0	317.0	8.75%
Hispanic	10.0%	Non-Hispanic	Ethnicity	0.044761	4,316	13.68%	508.0	15.0	2.95%	7.79%	0.275	0.000	Yes
Non-Hispanic	10.0%		Ethnicity	0.044761	4,316	13.68%	3,808.0	409.0	10.74%
Female	10.0%	Male	Sex	0.044761	2,656	46.88%	1,034.0	66.0	6.38%	3.05%	0.677	0.006	Yes
Male	10.0%		Sex	0.044761	2,656	46.88%	1,622.0	153.0	9.43%
Black	20.0%	White	Race	0.045863	4,322	13.56%	340.0	37.0	10.88%	9.85%	0.525	0.000	Yes
Asian	20.0%	White	Race	0.045863	4,322	13.56%	327.0	132.0	40.37%	-19.64%	1.947	0.000	No
Native American	20.0%	White	Race	0.045863	4,322	13.56%	20.0	2.0	10.00%	10.73%	0.482	0.403	No
White	20.0%		Race	0.045863	4,322	13.56%	3,623.0	751.0	20.73%
Hispanic	20.0%	Non-Hispanic	Ethnicity	0.045863	4,316	13.68%	508.0	42.0	8.27%	14.95%	0.356	0.000	Yes
Non-Hispanic	20.0%		Ethnicity	0.045863	4,316	13.68%	3,808.0	884.0	23.21%
Female	20.0%	Male	Sex	0.045863	2,656	46.88%	1,034.0	155.0	14.99%	4.92%	0.753	0.002	Yes
Male	20.0%		Sex	0.045863	2,656	46.88%	1,622.0	323.0	19.91%
Black	30.0%	White	Race	0.046427	4,322	13.56%	340.0	62.0	18.24%	11.30%	0.617	0.000	Yes
Asian	30.0%	White	Race	0.046427	4,322	13.56%	327.0	175.0	53.52%	-23.98%	1.812	0.000	No
Native American	30.0%	White	Race	0.046427	4,322	13.56%	20.0	4.0	20.00%	9.53%	0.677	0.464	No
White	30.0%		Race	0.046427	4,322	13.56%	3,623.0	1,070.0	29.53%
Hispanic	30.0%	Non-Hispanic	Ethnicity	0.046427	4,316	13.68%	508.0	69.0	13.58%	19.14%	0.415	0.000	Yes
Non-Hispanic	30.0%		Ethnicity	0.046427	4,316	13.68%	3,808.0	1,246.0	32.72%
Female	30.0%	Male	Sex	0.046427	2,656	46.88%	1,034.0	225.0	21.76%	5.74%	0.791	0.001	Yes
Male	30.0%		Sex	0.046427	2,656	46.88%	1,622.0	446.0	27.50%
Black	40.0%	White	Race	0.046703	4,322	13.56%	340.0	103.0	30.29%	16.38%	0.649	0.000	Yes
Asian	40.0%	White	Race	0.046703	4,322	13.56%	327.0	238.0	72.78%	-26.11%	1.559	0.000	No
Native American	40.0%	White	Race	0.046703	4,322	13.56%	20.0	8.0	40.00%	6.67%	0.857	0.656	No
White	40.0%		Race	0.046703	4,322	13.56%	3,623.0	1,691.0	46.67%
Hispanic	40.0%	Non-Hispanic	Ethnicity	0.046703	4,316	13.68%	508.0	139.0	27.36%	22.30%	0.551	0.000	Yes
Non-Hispanic	40.0%		Ethnicity	0.046703	4,316	13.68%	3,808.0	1,891.0	49.66%
Female	40.0%	Male	Sex	0.046703	2,656	46.88%	1,034.0	380.0	36.75%	7.27%	0.835	0.000	No
Male	40.0%		Sex	0.046703	2,656	46.88%	1,622.0	714.0	44.02%
Black	50.0%	White	Race	0.047009	4,322	13.56%	340.0	141.0	41.47%	9.70%	0.810	0.001	No
Asian	50.0%	White	Race	0.047009	4,322	13.56%	327.0	243.0	74.31%	-23.14%	1.452	0.000	No
Native American	50.0%	White	Race	0.047009	4,322	13.56%	20.0	9.0	45.00%	6.17%	0.879	0.657	No
White	50.0%		Race	0.047009	4,322	13.56%	3,623.0	1,854.0	51.17%
Hispanic	50.0%	Non-Hispanic	Ethnicity	0.047009	4,316	13.68%	508.0	167.0	32.87%	21.54%	0.604	0.000	Yes
Non-Hispanic	50.0%		Ethnicity	0.047009	4,316	13.68%	3,808.0	2,072.0	54.41%
Female	50.0%	Male	Sex	0.047009	2,656	46.88%	1,034.0	414.0	40.04%	9.78%	0.804	0.000	No
Male	50.0%		Sex	0.047009	2,656	46.88%	1,622.0	808.0	49.82%
Black	60.0%	White	Race	0.047266	4,322	13.56%	340.0	161.0	47.35%	13.62%	0.777	0.000	Yes
Asian	60.0%	White	Race	0.047266	4,322	13.56%	327.0	260.0	79.51%	-18.54%	1.304	0.000	No
Native American	60.0%	White	Race	0.047266	4,322	13.56%	20.0	11.0	55.00%	5.97%	0.902	0.648	No
White	60.0%		Race	0.047266	4,322	13.56%	3,623.0	2,209.0	60.97%
Hispanic	60.0%	Non-Hispanic	Ethnicity	0.047266	4,316	13.68%	508.0	214.0	42.13%	21.53%	0.662	0.000	Yes
Non-Hispanic	60.0%		Ethnicity	0.047266	4,316	13.68%	3,808.0	2,424.0	63.66%
Female	60.0%	Male	Sex	0.047266	2,656	46.88%	1,034.0	520.0	50.29%	7.60%	0.869	0.000	No
Male	60.0%		Sex	0.047266	2,656	46.88%	1,622.0	939.0	57.89%
Black	80.0%	White	Race	0.048018	4,322	13.56%	340.0	248.0	72.94%	7.96%	0.902	0.001	No
Asian	80.0%	White	Race	0.048018	4,322	13.56%	327.0	308.0	94.19%	-13.29%	1.164	0.000	No
Native American	80.0%	White	Race	0.048018	4,322	13.56%	20.0	14.0	70.00%	10.90%	0.865	0.250	No
White	80.0%		Race	0.048018	4,322	13.56%	3,623.0	2,931.0	80.90%
Hispanic	80.0%	Non-Hispanic	Ethnicity	0.048018	4,316	13.68%	508.0	364.0	71.65%	10.83%	0.869	0.000	No
Non-Hispanic	80.0%		Ethnicity	0.048018	4,316	13.68%	3,808.0	3,141.0	82.48%
Female	80.0%	Male	Sex	0.048018	2,656	46.88%	1,034.0	765.0	73.98%	4.44%	0.943	0.010	No
Male	80.0%		Sex	0.048018	2,656	46.88%	1,622.0	1,272.0	78.42%
Black	90.0%	White	Race	0.048694	4,322	13.56%	340.0	288.0	84.71%	5.41%	0.940	0.003	No
Asian	90.0%	White	Race	0.048694	4,322	13.56%	327.0	321.0	98.17%	-8.05%	1.089	0.000	No
Native American	90.0%	White	Race	0.048694	4,322	13.56%	20.0	17.0	85.00%	5.12%	0.943	0.441	No
White	90.0%		Race	0.048694	4,322	13.56%	3,623.0	3,265.0	90.12%
Hispanic	90.0%	Non-Hispanic	Ethnicity	0.048694	4,316	13.68%	508.0	428.0	84.25%	6.85%	0.925	0.000	No
Non-Hispanic	90.0%		Ethnicity	0.048694	4,316	13.68%	3,808.0	3,469.0	91.10%
Female	90.0%	Male	Sex	0.048694	2,656	46.88%	1,034.0	887.0	85.78%	4.04%	0.955	0.002	No
Male	90.0%		Sex	0.048694	2,656	46.88%	1,622.0	1,457.0	89.83%
Black	100.0%	White	Race	0.058530	4,322	13.56%	340.0	340.0	100.00%	0.00%	1.000	1.000	No
Asian	100.0%	White	Race	0.058530	4,322	13.56%	327.0	327.0	100.00%	0.00%	1.000	1.000	No
Native American	100.0%	White	Race	0.058530	4,322	13.56%	20.0	20.0	100.00%	0.00%	1.000	1.000	No
White	100.0%		Race	0.058530	4,322	13.56%	3,623.0	3,623.0	100.00%
Hispanic	100.0%	Non-Hispanic	Ethnicity	0.058530	4,316	13.68%	508.0	508.0	100.00%	0.00%	1.000	1.000	No
Non-Hispanic	100.0%		Ethnicity	0.058530	4,316	13.68%	3,808.0	3,808.0	100.00%
Female	100.0%	Male	Sex	0.058530	2,656	46.88%	1,034.0	1,034.0	100.00%	0.00%	1.000	1.000	No
Male	100.0%		Sex	0.058530	2,656	46.88%	1,622.0	1,622.0	100.00%

Odds Ratio#

odds_ratio = sd.odds_ratio(
    outcome=binary_outcome,
    odds_ratio_threshold=0.68,
    percent_difference_threshold=0.0,
    **reused_arguments,
)

odds_ratio

Disparity Calculation: Odds Ratio

┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ Protected Groups                          │ Black, Asian, Native American, Hispanic, Female                     │
│ Reference Groups                          │ White, White, White, Non-Hispanic, Male                             │
│ Group Categories                          │ Race, Race, Race, Ethnicity, Sex                                    │
│ Odds Ratio Threshold                      │ 0.68                                                                │
│ Percent Difference Threshold              │ 0.0                                                                 │
│ Lower Score Favorable                     │ True                                                                │
│ Affected Groups                           │ Black, Hispanic, Female                                             │
│ Affected Reference                        │ White, Non-Hispanic, Male                                           │
│ Affected Categories                       │ Race, Ethnicity, Sex                                                │
└───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Odds Ratio Summary Table

* Percent Missing: Ethnicity: 13.68%, Race: 13.56%, Sex: 46.88%

Group	Reference Group	Group Category	Total	Favorable	Percent Favorable	Odds	Percent Difference Favorable	Odds Ratio	P-Values	Practically Significant
Black	White	Race	340.0	141.0	41.47%	0.708543	9.70%	0.676058	0.001	Yes
Asian	White	Race	327.0	243.0	74.31%	2.892857	-23.14%	2.760229	0.000	No
Native American	White	Race	20.0	9.0	45.00%	0.818182	6.17%	0.780671	0.657	No
White		Race	3,623.0	1,854.0	51.17%	1.048050
Hispanic	Non-Hispanic	Ethnicity	508.0	167.0	32.87%	0.489736	21.54%	0.410319	0.000	Yes
Non-Hispanic		Ethnicity	3,808.0	2,072.0	54.41%	1.193548
Female	Male	Sex	1,034.0	414.0	40.04%	0.667742	9.78%	0.672700	0.000	Yes
Male		Sex	1,622.0	808.0	49.82%	0.992629

Categorical Adverse Impact Ratio#

# Generate an example categorical outcome.
categorical_outcome = pd.qcut(data["Prediction"], q=[0.0, 0.25, 0.5, 0.75, 1.0])
categories = categorical_outcome.cat.categories.to_series()
categories = pd.Series(["Best", "Great", "Good", "Bad"], index=categories.index)
categorical_outcome.replace(categories.to_dict(), inplace=True)

cair = sd.categorical_adverse_impact_ratio(
    outcome=categorical_outcome,
    category_order=list(reversed(categories.tolist())),
    air_threshold=0.8,
    percent_difference_threshold=0.0,
    **reused_arguments,
)

cair

Disparity Calculation: Categorical Adverse Impact Ratio

┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ Protected Groups                          │ Black, Asian, Native American, Hispanic, Female                     │
│ Reference Groups                          │ White, White, White, Non-Hispanic, Male                             │
│ Group Categories                          │ Race, Race, Race, Ethnicity, Sex                                    │
│ AIR Threshold                             │ 0.8                                                                 │
│ Percent Difference Threshold              │ 0.0                                                                 │
│ Affected Groups                           │ Hispanic, Black, Female                                             │
│ Affected Reference                        │ Non-Hispanic, White, Male                                           │
│ Affected Categories                       │ Ethnicity, Race, Sex                                                │
└───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Categorical Adverse Impact Ratio Summary Table

Group	Category	Reference Group	Group Category	Ordinal	Observations	Percent Missing	Total	Favorable	Percent Favorable	Percent Difference Favorable	AIR	P-Values	Practically Significant
Black	Bad	White	Race	0	4,322	13.56%	340.0	340.0	100.00%	0.00%	1.000	1.000	No
Asian	Bad	White	Race	0	4,322	13.56%	327.0	327.0	100.00%	0.00%	1.000	1.000	No
Native American	Bad	White	Race	0	4,322	13.56%	20.0	20.0	100.00%	0.00%	1.000	1.000	No
White	Bad		Race	0	4,322	13.56%	3,623.0	3,623.0	100.00%
Hispanic	Bad	Non-Hispanic	Ethnicity	0	4,316	13.68%	508.0	508.0	100.00%	0.00%	1.000	1.000	No
Non-Hispanic	Bad		Ethnicity	0	4,316	13.68%	3,808.0	3,808.0	100.00%
Female	Bad	Male	Sex	0	2,656	46.88%	1,034.0	1,034.0	100.00%	0.00%	1.000	1.000	No
Male	Bad		Sex	0	2,656	46.88%	1,622.0	1,622.0	100.00%
Black	Good	White	Race	1	4,322	13.56%	340.0	248.0	72.94%	7.96%	0.902	0.001	No
Asian	Good	White	Race	1	4,322	13.56%	327.0	308.0	94.19%	-13.29%	1.164	0.000	No
Native American	Good	White	Race	1	4,322	13.56%	20.0	14.0	70.00%	10.90%	0.865	0.250	No
White	Good		Race	1	4,322	13.56%	3,623.0	2,931.0	80.90%
Hispanic	Good	Non-Hispanic	Ethnicity	1	4,316	13.68%	508.0	364.0	71.65%	10.83%	0.869	0.000	No
Non-Hispanic	Good		Ethnicity	1	4,316	13.68%	3,808.0	3,141.0	82.48%
Female	Good	Male	Sex	1	2,656	46.88%	1,034.0	765.0	73.98%	4.44%	0.943	0.010	No
Male	Good		Sex	1	2,656	46.88%	1,622.0	1,272.0	78.42%
Black	Great	White	Race	2	4,322	13.56%	340.0	141.0	41.47%	9.70%	0.810	0.001	No
Asian	Great	White	Race	2	4,322	13.56%	327.0	243.0	74.31%	-23.14%	1.452	0.000	No
Native American	Great	White	Race	2	4,322	13.56%	20.0	9.0	45.00%	6.17%	0.879	0.657	No
White	Great		Race	2	4,322	13.56%	3,623.0	1,854.0	51.17%
Hispanic	Great	Non-Hispanic	Ethnicity	2	4,316	13.68%	508.0	167.0	32.87%	21.54%	0.604	0.000	Yes
Non-Hispanic	Great		Ethnicity	2	4,316	13.68%	3,808.0	2,072.0	54.41%
Female	Great	Male	Sex	2	2,656	46.88%	1,034.0	414.0	40.04%	9.78%	0.804	0.000	No
Male	Great		Sex	2	2,656	46.88%	1,622.0	808.0	49.82%
Black	Best	White	Race	3	4,322	13.56%	340.0	44.0	12.94%	10.88%	0.543	0.000	Yes
Asian	Best	White	Race	3	4,322	13.56%	327.0	145.0	44.34%	-20.52%	1.862	0.000	No
Native American	Best	White	Race	3	4,322	13.56%	20.0	2.0	10.00%	13.82%	0.420	0.191	No
White	Best		Race	3	4,322	13.56%	3,623.0	863.0	23.82%
Hispanic	Best	Non-Hispanic	Ethnicity	3	4,316	13.68%	508.0	47.0	9.25%	17.22%	0.350	0.000	Yes
Non-Hispanic	Best		Ethnicity	3	4,316	13.68%	3,808.0	1,008.0	26.47%
Female	Best	Male	Sex	3	2,656	46.88%	1,034.0	175.0	16.92%	5.21%	0.765	0.001	Yes
Male	Best		Sex	3	2,656	46.88%	1,622.0	359.0	22.13%

Residual Standardized Mean Difference#

rsmd = sd.residual_standardized_mean_difference(
    prediction=data["Prediction"],
    label=data[label],
    residual_smd_threshold=30,
    lower_score_favorable=True,
    **reused_arguments,
)

rsmd

Disparity Calculation: Residual Standardized Mean Difference

┌───────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────┐
│ Protected Groups                      │ Black, Asian, Native American, Hispanic, Female                         │
│ Reference Groups                      │ White, White, White, Non-Hispanic, Male                                 │
│ Group Categories                      │ Race, Race, Race, Ethnicity, Sex                                        │
│ Residual SMD Threshold                │ 30.0                                                                    │
│ Residual SMD Denominator              │ population                                                              │
│ Lower Score Favorable                 │ True                                                                    │
│ Affected Groups                       │                                                                         │
│ Affected Reference                    │                                                                         │
│ Affected Categories                   │                                                                         │
└───────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────┘

Residual Standardized Mean Difference Summary Table

* Percent Missing: Ethnicity: 13.68%, Race: 13.56%, Sex: 46.88%

Group	Reference Group	Group Category	Total	Average Prediction	Average Label	Average Residual	Difference in Average Residual	Std. Dev. of Residuals	Residual SMD	P-Values	Practically Significant
Black	White	Race	340.0	0.047486	0.05	0.000472	0.000265	0.004864	5.445415	0.337	No
Asian	White	Race	327.0	0.045165	0.04	-0.000769	-0.000976	0.004864	-20.069038	0.000	No
Native American	White	Race	20.0	0.047487	0.05	0.000628	0.000421	0.004864	8.646488	0.699	No
White		Race	3,623.0	0.046833	0.05	0.000207		0.004864
Hispanic	Non-Hispanic	Ethnicity	508.0	0.047667	0.05	0.001097	0.001046	0.004864	21.513889	0.000	No
Non-Hispanic		Ethnicity	3,808.0	0.046634	0.05	0.000051		0.004864
Female	Male	Sex	1,034.0	0.047268	0.05	0.000435	0.000246	0.004864	5.047722	0.229	No
Male		Sex	1,622.0	0.046873	0.05	0.000189		0.004864

Segmented Adverse Impact Ratio#

# Generate example income segments.
segments = pd.qcut(data["Income"], q=[0.0, 1 / 3, 2 / 3, 1.0])
categories = segments.cat.categories.to_series()
categories = pd.Series(
    ["Low Income", "Mid Income", "High Income"], index=categories.index
)
segments.replace(categories.to_dict(), inplace=True)

sair = sd.segmented_adverse_impact_ratio(
    outcome=binary_outcome,
    air_threshold=0.8,
    percent_difference_threshold=0.0,
    fdr_threshold=0.2,
    segment=segments,
    **reused_arguments,
)

sair

Disparity Calculation: Segmented Adverse Impact Ratio

┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ Protected Groups                          │ Black, Asian, Native American, Hispanic, Female                     │
│ Reference Groups                          │ White, White, White, Non-Hispanic, Male                             │
│ Group Categories                          │ Race, Race, Race, Ethnicity, Sex                                    │
│ AIR Threshold                             │ 0.8                                                                 │
│ Percent Difference Threshold              │ 0.0                                                                 │
│ FDR Threshold                             │ 0.2                                                                 │
│ Affected Groups                           │ Hispanic                                                            │
│ Affected Reference                        │ Non-Hispanic                                                        │
│ Affected Categories                       │ Ethnicity                                                           │
└───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Segmented Adverse Impact Ratio Summary Table

Group	Segment	Reference Group	Group Category	Observations	Percent Missing	Total	Favorable	Percent Favorable	Percent Difference Favorable	AIR	P-Values	Practically Significant
Black	Low Income	White	Race	1,510	11.80%	151.0	33.0	21.85%	4.18%	0.839	0.280	No
Black	Mid Income	White	Race	1,414	13.14%	117.0	56.0	47.86%	5.02%	0.905	0.332	No
Black	High Income	White	Race	1,398	15.78%	72.0	52.0	72.22%	5.15%	0.933	0.313	No
Asian	Low Income	White	Race	1,510	11.80%	69.0	22.0	31.88%	-5.85%	1.225	0.325	No
Asian	Mid Income	White	Race	1,414	13.14%	89.0	67.0	75.28%	-22.39%	1.423	0.000	No
Asian	High Income	White	Race	1,398	15.78%	169.0	154.0	91.12%	-13.75%	1.178	0.000	No
Native American	Low Income	White	Race	1,510	11.80%	9.0	3.0	33.33%	-7.30%	1.280	0.704	No
Native American	Mid Income	White	Race	1,414	13.14%	6.0	4.0	66.67%	-13.78%	1.261	0.690	No
Native American	High Income	White	Race	1,398	15.78%	5.0	2.0	40.00%	37.37%	0.517	0.081	No
White	Low Income		Race	1,510	11.80%	1,279.0	333.0	26.04%
White	Mid Income		Race	1,414	13.14%	1,195.0	632.0	52.89%
White	High Income		Race	1,398	15.78%	1,149.0	889.0	77.37%
Hispanic	Low Income	Non-Hispanic	Ethnicity	1,527	10.81%	242.0	40.0	16.53%	10.86%	0.603	0.000	Yes
Hispanic	Mid Income	Non-Hispanic	Ethnicity	1,393	14.43%	183.0	74.0	40.44%	16.09%	0.715	0.000	Yes
Hispanic	High Income	Non-Hispanic	Ethnicity	1,396	15.90%	83.0	53.0	63.86%	15.05%	0.809	0.002	Yes
Non-Hispanic	Low Income		Ethnicity	1,527	10.81%	1,285.0	352.0	27.39%
Non-Hispanic	Mid Income		Ethnicity	1,393	14.43%	1,210.0	684.0	56.53%
Non-Hispanic	High Income		Ethnicity	1,396	15.90%	1,313.0	1,036.0	78.90%
Female	Low Income	Male	Sex	1,263	26.23%	528.0	107.0	20.27%	6.27%	0.764	0.012	No
Female	Mid Income	Male	Sex	815	49.94%	316.0	160.0	50.63%	8.89%	0.851	0.016	No
Female	High Income	Male	Sex	578	65.18%	190.0	147.0	77.37%	4.07%	0.950	0.268	No
Male	Low Income		Sex	1,263	26.23%	735.0	195.0	26.53%
Male	Mid Income		Sex	815	49.94%	499.0	297.0	59.52%
Male	High Income		Sex	578	65.18%	388.0	316.0	81.44%
Black	CMH Test	White	Race	4,322	13.56%	340.0				0.899	0.071	No
Black	Breslow-Day Test	White	Race	4,322	13.56%	340.0					0.976
Asian	CMH Test	White	Race	4,322	13.56%	327.0				1.241	0.000	No
Asian	Breslow-Day Test	White	Race	4,322	13.56%	327.0					0.058
Native American	CMH Test	White	Race	4,322	13.56%	20.0				0.959	0.849	No
Native American	Breslow-Day Test	White	Race	4,322	13.56%	20.0					0.107
Hispanic	CMH Test	Non-Hispanic	Ethnicity	4,316	13.68%	508.0				0.710	0.000	Yes
Hispanic	Breslow-Day Test	Non-Hispanic	Ethnicity	4,316	13.68%	508.0					0.929
Female	CMH Test	Male	Sex	2,656	46.88%	1,034.0				0.857	0.000	No
Female	Breslow-Day Test	Male	Sex	2,656	46.88%	1,034.0					0.906

Custom Disparity Metric#

We can recreate the AIR as an example of how custom disparity metrics can be used. Many more advanced disparity metrics can benefit from the framework and additional validation provided by the custom disparity metric interface.

# Define a function for calculating perecent favorable.
def percent_favorable(outcome, sample_weight):
    return (outcome.mul(sample_weight, axis=0)).sum(
        axis=0, min_count=1
    ) / sample_weight.sum(axis=0, min_count=1)

custom_air = sd.custom_disparity_metric(
    outcome=binary_outcome,
    metric=percent_favorable,
    difference_calculation=sd.types.DifferenceCalculation.REFERENCE_MINUS_PROTECTED,
    difference_threshold=lambda value: value > 0.0,
    ratio_calculation=sd.types.RatioCalculation.PROTECTED_OVER_REFERENCE,
    ratio_threshold=lambda value: value < 1.0,
    statistical_significance_test=sd.types.StatSigTest.FISHERS_OR_CHI_SQUARED,
    **reused_arguments,
)

custom_air

Disparity Calculation: Custom Disparity Metric

┌─────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────┐
│ Protected Groups                    │ Black, Asian, Native American, Hispanic, Female                           │
│ Reference Groups                    │ White, White, White, Non-Hispanic, Male                                   │
│ Group Categories                    │ Race, Race, Race, Ethnicity, Sex                                          │
│ Metric                              │ percent_favorable                                                         │
│ Difference Calculation              │ reference_minus_protected                                                 │
│ Ratio Calculation                   │ protected_over_reference                                                  │
│ Affected Groups                     │ Black, Hispanic, Female                                                   │
│ Affected Reference                  │ White, Non-Hispanic, Male                                                 │
│ Affected Categories                 │ Race, Ethnicity, Sex                                                      │
└─────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────┘

Custom Disparity Metric Summary Table

* Percent Missing: Ethnicity: 13.68%, Race: 13.56%, Sex: 46.88%

Group	Reference Group	Group Category	Total	PERCENT FAVORABLE	Difference	Ratio	P-Values	Practically Significant
Black	White	Race	340.0	0.414706	0.097	0.810	0.001	Yes
Asian	White	Race	327.0	0.743119	-0.231	1.452	0.000	No
Native American	White	Race	20.0	0.450000	0.062	0.879	0.657	No
White		Race	3,623.0	0.511731
Hispanic	Non-Hispanic	Ethnicity	508.0	0.328740	0.215	0.604	0.000	Yes
Non-Hispanic		Ethnicity	3,808.0	0.544118
Female	Male	Sex	1,034.0	0.400387	0.098	0.804	0.000	Yes
Male		Sex	1,622.0	0.498150

Note that the values from the custom disparity metric summary correspond to those of the original AIR calculation.

pd.concat(
    (
        custom_air.summary_table["PERCENT FAVORABLE"],
        air.summary_table[sd.const.PERCENT_FAVORABLE],
    ),
    axis=1,
)

	PERCENT FAVORABLE	Percent Favorable
Group
Black	0.414706	0.414706
Asian	0.743119	0.743119
Native American	0.450000	0.450000
White	0.511731	0.511731
Hispanic	0.328740	0.328740
Non-Hispanic	0.544118	0.544118
Female	0.400387	0.400387
Male	0.498150	0.498150

pd.concat(
    (
        custom_air.summary_table[sd.const.RATIO],
        air.summary_table[sd.const.AIR_VALUES],
    ),
    axis=1,
)

	Ratio	AIR
Group
Black	0.810399	0.810399
Asian	1.452169	1.452169
Native American	0.879369	0.879369
White	NaN	NaN
Hispanic	0.604171	0.604171
Non-Hispanic	NaN	NaN
Female	0.803747	0.803747
Male	NaN	NaN

Confusion Matrix Metrics#

SolasAI-provides ready-made implementations of confusion matrix metrics by wrapping around the custom disparity metric.

(
    sd.false_discovery_rate,
    sd.false_negative_rate,
    sd.false_positive_rate,
    sd.precision,
    sd.true_negative_rate,
    sd.true_positive_rate,
)

(<cyfunction false_discovery_rate at 0x7f102d9941e0>,
 <cyfunction false_negative_rate at 0x7f102d9a16c0>,
 <cyfunction false_positive_rate at 0x7f102d9adba0>,
 <cyfunction precision at 0x7f102d949520>,
 <cyfunction true_negative_rate at 0x7f102d8d0040>,
 <cyfunction true_positive_rate at 0x7f102d8d8520>)

precision_arguments = reused_arguments.copy()
precision_arguments["group_data"] = precision_arguments["group_data"].fillna(0.0)
precision = sd.precision(
    outcome=binary_outcome,
    label=binary_label,
    ratio_threshold=1.0,
    difference_threshold=0.0,
    **precision_arguments,
)
precision

Disparity Calculation: Precision

┌─────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────┐
│ Protected Groups                    │ Black, Asian, Native American, Hispanic, Female                           │
│ Reference Groups                    │ White, White, White, Non-Hispanic, Male                                   │
│ Group Categories                    │ Race, Race, Race, Ethnicity, Sex                                          │
│ Metric                              │ precision_score                                                           │
│ Difference Calculation              │ reference_minus_protected                                                 │
│ Difference Threshold                │ 0.0                                                                       │
│ Ratio Calculation                   │ protected_over_reference                                                  │
│ Ratio Threshold                     │ 1.0                                                                       │
│ Affected Groups                     │ Black, Native American, Hispanic, Female                                  │
│ Affected Reference                  │ White, White, Non-Hispanic, Male                                          │
│ Affected Categories                 │ Race, Race, Ethnicity, Sex                                                │
└─────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────┘

Precision Summary Table

* Percent Missing: Ethnicity: 0.00%, Race: 0.00%, Sex: 0.00%

Group	Reference Group	Group Category	Total	Total Label	Average Label	PRECISION SCORE	Difference	Ratio	Practically Significant
Black	White	Race	340.0	139.0	0.41	0.511	0.105	0.829	Yes
Asian	White	Race	327.0	210.0	0.64	0.724	-0.108	1.176	No
Native American	White	Race	20.0	10.0	0.50	0.556	0.060	0.902	Yes
White		Race	3,623.0	1,809.0	0.50	0.616
Hispanic	Non-Hispanic	Ethnicity	508.0	200.0	0.39	0.563	0.069	0.891	Yes
Non-Hispanic		Ethnicity	3,808.0	1,976.0	0.52	0.632
Female	Male	Sex	1,034.0	451.0	0.44	0.560	0.063	0.898	Yes
Male		Sex	1,622.0	803.0	0.50	0.624

sd.precision is essentially a convience wrapper for the following call to the custom disparity metric.

def precision(y_true, y_pred, sample_weight):
    from sklearn.metrics import confusion_matrix

    tn, fp, fn, tp = confusion_matrix(
        y_true=y_true,
        y_pred=y_pred,
        sample_weight=sample_weight,
    ).ravel()

    return tp / (tp + fp)


sd.custom_disparity_metric(
    outcome=binary_outcome,
    metric=precision,
    label=binary_label,
    difference_calculation=sd.types.DifferenceCalculation.REFERENCE_MINUS_PROTECTED,
    difference_threshold=lambda difference: difference > 0.0,
    ratio_calculation=sd.types.RatioCalculation.PROTECTED_OVER_REFERENCE,
    ratio_threshold=lambda ratio: ratio < 1.0,
    statistical_significance_test=sd.types.StatSigTest.BOOTSTRAPPING,
    p_value_threshold=0.05,
    **precision_arguments,
)

Disparity Calculation: Custom Disparity Metric

┌─────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────┐
│ Protected Groups                    │ Black, Asian, Native American, Hispanic, Female                           │
│ Reference Groups                    │ White, White, White, Non-Hispanic, Male                                   │
│ Group Categories                    │ Race, Race, Race, Ethnicity, Sex                                          │
│ Metric                              │ precision                                                                 │
│ Difference Calculation              │ reference_minus_protected                                                 │
│ Ratio Calculation                   │ protected_over_reference                                                  │
│ Affected Groups                     │                                                                           │
│ Affected Reference                  │                                                                           │
│ Affected Categories                 │                                                                           │
└─────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────┘

Custom Disparity Metric Summary Table

* Percent Missing: Ethnicity: 0.00%, Race: 0.00%, Sex: 0.00%

Group	Reference Group	Group Category	Total	Total Label	Average Label	PRECISION	Difference	Ratio	P-Values	Practically Significant
Black	White	Race	340.0	139.0	0.41	0.510638	0.105	0.829	0.092	No
Asian	White	Race	327.0	210.0	0.64	0.724280	-0.108	1.176	0.020	No
Native American	White	Race	20.0	10.0	0.50	0.555556	0.060	0.902		No
White		Race	3,623.0	1,809.0	0.50	0.615965
Hispanic	Non-Hispanic	Ethnicity	508.0	200.0	0.39	0.562874	0.069	0.891	0.224	No
Non-Hispanic		Ethnicity	3,808.0	1,976.0	0.52	0.631757
Female	Male	Sex	1,034.0	451.0	0.44	0.560386	0.063	0.898	0.141	No
Male		Sex	1,622.0	803.0	0.50	0.623762

Additionally, statistical significance can be set to None for custom disparity metrics, causing statistical significance calculations to be skipped.

sd.custom_disparity_metric(
    outcome=binary_outcome,
    metric=precision,
    label=binary_label,
    difference_calculation=sd.types.DifferenceCalculation.REFERENCE_MINUS_PROTECTED,
    difference_threshold=lambda difference: difference > 0.0,
    ratio_calculation=sd.types.RatioCalculation.PROTECTED_OVER_REFERENCE,
    ratio_threshold=lambda ratio: ratio < 1.0,
    statistical_significance_test=None,
    p_value_threshold=0.05,
    **precision_arguments,
)

Disparity Calculation: Custom Disparity Metric

┌─────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────┐
│ Protected Groups                    │ Black, Asian, Native American, Hispanic, Female                           │
│ Reference Groups                    │ White, White, White, Non-Hispanic, Male                                   │
│ Group Categories                    │ Race, Race, Race, Ethnicity, Sex                                          │
│ Metric                              │ precision                                                                 │
│ Difference Calculation              │ reference_minus_protected                                                 │
│ Ratio Calculation                   │ protected_over_reference                                                  │
│ Affected Groups                     │ Black, Native American, Hispanic, Female                                  │
│ Affected Reference                  │ White, White, Non-Hispanic, Male                                          │
│ Affected Categories                 │ Race, Race, Ethnicity, Sex                                                │
└─────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────┘

Custom Disparity Metric Summary Table

* Percent Missing: Ethnicity: 0.00%, Race: 0.00%, Sex: 0.00%

Group	Reference Group	Group Category	Total	Total Label	Average Label	PRECISION	Difference	Ratio	Practically Significant
Black	White	Race	340.0	139.0	0.41	0.510638	0.105	0.829	Yes
Asian	White	Race	327.0	210.0	0.64	0.724280	-0.108	1.176	No
Native American	White	Race	20.0	10.0	0.50	0.555556	0.060	0.902	Yes
White		Race	3,623.0	1,809.0	0.50	0.615965
Hispanic	Non-Hispanic	Ethnicity	508.0	200.0	0.39	0.562874	0.069	0.891	Yes
Non-Hispanic		Ethnicity	3,808.0	1,976.0	0.52	0.631757
Female	Male	Sex	1,034.0	451.0	0.44	0.560386	0.063	0.898	Yes
Male		Sex	1,622.0	803.0	0.50	0.623762

SolasAI documentation

SolasAI Disparity Calculations

Contents

SolasAI Disparity Calculations#

Adverse Impact Ratio (AIR)#

Standardized Mean Difference (SMD)#

Adverse Impact Ratio by Quantile#

Odds Ratio#

Categorical Adverse Impact Ratio#

Residual Standardized Mean Difference#

Segmented Adverse Impact Ratio#

Custom Disparity Metric#

Confusion Matrix Metrics#