When AI goes wrong: understanding and preventing algorithmic bias
Myriam Blanchet
Analyst
Artificial intelligence is a powerful ally, capable of analyzing billions of pieces of data to support our decisions. But like any technology, it is not neutral. AI models learn from historical data, often unbalanced or incomplete, and can reproduce (or even amplify) existing biases.
These deviations are not theoretical: they have very real effects on individuals, organizations… and even financial markets. Here are just a few of the striking examples presented at the “Is My Robot Racist? And other Ethical Data Dilemmas” by Jon Lane from Ad Astra.
Stereotypes embedded in training data
Computer vision models, used to analyze images or recognize faces, often perpetuate stereotypes about professions or demographic characteristics.
Why? Because the datasets they are trained on reflect geographical or social imbalances.
For example :
- An algorithm learning to recognize an “engineer” might associate this role predominantly with white men, if it has only seen this type of profile in its data.
- An image analysis could misidentify a black person in a low-diversity neighborhood, simply because the model has never seen a dark face in that neighborhood. So it hasn’t been trained with complete data.
Serious health consequences
The impact of algorithmic biases can become crucial in healthcare:
- Pulse oximeters (devices that measure oxygen in the blood) use optical sensors. However, several studies have shown that their signal processing algorithms are less accurate on darker skins, causing diagnostic errors.
- More generally, when clinicians rely solely on automated diagnostic tools, without cross-referencing results with their clinical judgment, the risk of error increases.
The invisible bias in HR and credit systems
AI models used in human resources or finance are particularly vulnerable to indirect bias, often via so-called “proxy” variables:
- In HR, hiring algorithms have been singled out for discriminating against women who have taken maternity leave, simply because career breaks were under-represented in the training data.
- In credit rating, the zip code is sometimes used as an indicator of creditworthiness. Problem: it can correlate with ethnic origin, creating an indirect bias based on race, without this being explicitly taken into account.
And when financial markets pay the price...
One of the most striking examples is that of Knight Capital, in 2012.
Within 45 minutes, a bug in an automated trading system generated $440 million worth of erroneous trades.
Why was this? Because the human operators accepted the results without checking them.
This case reminds us of a key lesson: even the most advanced AI should never be used without human supervision.
How can we prevent these abuses?
Algorithmic biases are not inevitable. Here are a few practices to limit them right from the project design stage:
- Diversify training data, geographically, demographically and behaviorally.
- Validate models with business experts AND stakeholders (e.g. patients, employees, citizens).
- Avoid sensitive proxy variables or those correlated with protected characteristics (gender, origin, age…).
- Integrate human verification mechanisms into automated decision-making processes.
- Document the model’s limitations: what it knows, what it doesn’t know, and in what context it may be biased.
ADNIA, a human partner in your AI projects
At ADNIA, we believe that the power of data must serve justice, transparency and collective intelligence.
That’s why we support our customers not only in the technical development of their AI projects, but also in their ethics, rigor and human adoption.
AI is a formidable lever… provided it is designed and used with vigilance, sensitivity and responsibility.
AI tools may have supported the creation of this content
