May, 2026
Gender Bias in AI and the Limits of Technical Solutions
In 2014, Amazon's engineers set out to automate résumé screening using a decade of the company's own hiring data. By 2017, they shelved the project. The model had taught itself to penalise applications containing the word "women's" and to downgrade graduates of ‘all-women's’ colleges. Attempts to fix the behaviour failed. The system was not malfunctioning but was performing exactly as designed, faithfully mirroring the decisions that had trained it.
More than a decade later, the lesson has not been learned. It has been scaled.
AI systems are now embedded in hiring, credit scoring, medical diagnostics, content moderation and citizen-facing public services. Where those systems were trained on data from an unequal world inherit, reproduce, and in many cases amplify that inequality. For India, where AI-powered public infrastructure is being deployed faster than the governance architecture meant to supervise it, the question is not whether this is a problem. It is whether policymakers will treat it as one.
The data was skewed before the algorithm arrived
Gender bias does not enter AI systems at a single identifiable point that can be patched. It enters under three distinct stages, framing it as a 'technical compliance' problem addresses none of them adequately.
The first is the training corpus itself. Large language models and image generators are trained on vast aggregations of human-produced content: web text, images, speech transcripts, health records, hiring decisions, credit histories. Where that material under-represents or misrepresents women, the model inherits the distortion.
An infamous 2018 audit by Buolamwini and Gebru of three commercial facial-analysis systems found error rates below one percent for lighter-skinned men and as high as 34.7 percent for darker-skinned women, a category that includes most Indian women. Speech recognition shows comparable disparities (as seen in table 1), creating biases at enormous scaled. For instance, medical imaging datasets remain heavily skewed toward male subjects, reflecting more than a century of clinical research that over-represented men- and AI diagnostic tools trained in that literature does inherits the old biases, and then scales it.
Table 1. Female representation in data sources commonly used to train AI systems.
|
Source |
Share of women (%) |
|
English Wikipedia, biographical articles |
19 |
|
Medical imaging datasets |
38 |
|
Global technology workforce |
28 |
|
New CS PhD graduates (US/Canada) |
22 |
Sources: Wikimedia Foundation; WEF Global Gender Gap Report 2024; Stanford AI Index 2024; Larrazabal et al. 2020.
Design decisions amplify what the data reflects
Even a perfectly balanced dataset would not eliminate bias, because it also enters the stage of model design, in the decisions about which features to weight, what constitutes a successful prediction and how performance is evaluated.
A hiring or credit-scoring algorithm asked to identify applicants likely to “succeed” or repay will, absent deliberate intervention, search for features that correlate with past success. Where past success was itself a gendered outcome, shaped by household financial arrangements, restricted mobility or informal gatekeeping in professional networks, the model reproduces the pattern. It is, in a precise sense, the correct answer to the wrong question.
A 2023 Bloomberg analysis of Stable Diffusion across thousands of occupational prompts found that images of judges and CEOs as women appeared in low single-digit percentages, and images of doctors as women fell below the actual workforce representation. For India, where women constitute less than a fifth of the formal technology workforce, deploying such generative tools in school textbooks, government communication and advertising risks normalising the visual defaults that limit aspiration before it forms. And as generated images increasingly appear in future training data, the bias is no longer merely reflected but is progressively ingrained.
Who builds the system determines whose failures get fixed
The third stage, and the one least visible in public discussion, is deployment. Early voice-recognition systems exhibited higher error rates for women, but users were instructed to modulate their speech rather than fixing the defect. Pulse oximeters, calibrated on predominantly lighter-skinned people, produced systematic errors for darker-skinned patients Content-moderation systems on major platforms have been shown to flag material from women and minority users at disproportionate rates, with limited recourse available.
These patterns reveal a structural question: who decides what counts as a malfunction? Women constitute approximately a quarter to a third percent of professionals working in AI globally as well as in the Indian technology sector, with significantly lower representation at senior research levels. A recent study by LeanIn, a gender non-profit, found that women get less recognition for using AI tools at work, less managerial support, and are more likely to fear displacement.
Where the population's identifying failures is demographically narrow, failures that do not affect that population are noticed later, studied less, and remediated more slowly. A 2024 UNESCO evaluation of large language models found persistent gendered stereotyping in every model tested. Hence, AI workforce composition is not just a diversity signalling exercise; it is a technical quality-assurance question about whose problems the system will recognise.
India cannot afford to import bias at an infrastructure scale
None of this is an argument against technical work on algorithmic fairness. The argument is that technical methods address the symptoms and not the condition. The condition is that AI systems reproduce a social world that was already unevenly balanced; and do so on a scale no previous information technology has matched.
For India, this possesses a particular urgency. As AI-assisted diagnostics are rolled out through Ayushman Bharat, lending decisions are automated through fintech platforms, public employment schemes are filtered through algorithmic screening, the question must not be limited to aggregate model accuracy. The question is whose bodies, whose voices, and whose financial histories the models were calibrated on bears the cost when they are wrong.
The IndiaAI Mission and the Digital India framework provide a foundation. Three additions would make them adequate to the problem.
First, public procurement of AI systems should require disclosure of training data composition and documented performance across gender, regional and linguistic subgroups. Sub-group accuracy treated as a binding procurement criterion, not a supplementary disclosure.
Second, the Digital Personal Data Protection Act should provide meaningful redress for users affected by automated decisions, including human review and explanation in regional languages, not only English.
Third, gender representation in AI development teams should be treated as a component of technical quality assurance, with measurable targets at senior research levels and not just at entry-level cohortswhere diversity programmes are currently concentrated.
AI systems will continue to be deployed at speed, in contexts where the cost of failure falls unevenly across the population. The choice before policymakers is not whether to govern these systems but whether to govern them before the harm is already distributed. Frameworks that treat bias as a technical defect to be certified away, rather than a structural condition to be governed, will find themselves managing consequences long after the moment to prevent them has passed.
