The research by the London School of Economics’ Care Policy & Evaluation Centre (CPEC), published in BMC Medical Informatics and Decision Making, investigated potential gender bias by evaluating summaries of long-term care records generated by LLMs.

The social care sectors in the US and UK are using LLMs to generate summaries of extensive case notes or audio transcripts of care interventions, the paper noted. 

The study used LLMs, including Google’s Gemma and Meta’s Llama 3, to generate pairs of summaries based on case notes of 617 adult social care users from a London local authority. Each pair of summaries described the same individual, with their genders swapped.

The study found that the Gemma LLM displayed the most significant gender-based differences of the models researched, including consistently producing more negative summaries for men and tending to use more explicit language about men’s health conditions than women’s.

For example, the Gemma model frequently described women as managing well ‘despite’ their impairments, with the word ‘despite’ appearing significantly more for women, while the terms ‘disabled’ and ‘unable’ were used significantly more for men than for women.

The research was funded by the National Institute for Health and Care Research.

Dr Sam Rickman, lead author of the report and a researcher in CPEC, said: “If social workers are relying on biased AI-generated summaries that systematically downplay women’s health needs, they may assess otherwise identical cases differently based on gender rather than actual need. Since access to social care is determined by perceived need, this could result in unequal care provision for women.

“Large language models are already being used in the public sector, but their use must not come at the expense of fairness. While my research highlights issues with one model, more are being deployed all the time, making it essential that all AI systems are transparent, rigorously tested for bias and subject to robust legal oversight.”

• Rickman, S. Evaluating gender bias in large language models in long-term care. BMC Med Inform Decis Mak 25, 274 ( 2025 ).