Bias and Fairness in AI: Why Models Discriminate
Bias usually comes from the data and the problem definition rather than the model. Historical bias, representation gaps, fairness metrics, and why those metrics contradict each other.
A model has no malice; it simply learns the world you show it. If that world is skewed, the model learns the skew — and then scales it. Bias is therefore as much a definitional problem as a technical one: it depends on what you chose to predict and which data you taught it with.
Where bias comes from
- Historical bias: the data reflects past decisions. If those decisions were skewed, so is the data.
- Representation bias: some groups appear rarely in the dataset, and the model performs worse for them.
- Measurement bias: the target is not what you actually want to measure. If you record “arrests” rather than “crime”, the model learns patrol density.
- Aggregation bias: distinct groups are compressed into one model that fits none of them well.
The proxy-variable trap
Deleting a gender or ethnicity column is not enough, because other variables carry that information indirectly. These are proxies. The real answer is to measure and monitor the sensitive attribute rather than delete it. You cannot correct an inequity you refuse to measure.
Fairness metrics
| Metric | What it demands |
|---|---|
| Demographic parity | Equal rate of positive predictions across groups |
| Equal opportunity | Equal true-positive rate across groups |
| Equalised odds | Equal true- and false-positive rates |
| Calibration | The same score means the same real probability in every group |
Why the metrics conflict
These criteria cannot all hold at once. When base rates differ between groups, calibration and equalised odds are mathematically incompatible. This is not an engineering shortcoming but a proven impossibility.
The consequence is that there is no single definition of a “fair model”. Which definition you adopt is a value judgement tied to your domain and to who is harmed by which error — and it should be written down explicitly.
What to do in practice
- Identify the sensitive groups and measure performance separately for each.
- Judge by the worst-performing group, not the aggregate metric.
- Close representation gaps with targeted data collection.
- Document the fairness definition you chose and why.
- Keep monitoring by group in production; when the distribution shifts, fairness shifts with it.
Frequently asked questions
Is the model biased, or the data?
Almost always the data and the problem definition. The model merely amplifies the relationships it is shown.
Is collecting sensitive data a privacy problem?
Fairness auditing is a legitimate purpose, but it requires separate legal assessment, strict purpose limitation and access control. Data gathered for auditing should not be used as a model input.
How do I measure this for small groups?
With confidence intervals. In small samples, group differences are indistinguishable from noise; do not draw conclusions without reporting uncertainty.
Fairness is not a filter bolted on afterwards but a design decision that starts with the problem definition. For related reading, see model interpretability and data quality.
