Skip to main content
Science Catalogs
Social Sciences and Humanities

Large language models vary widely in checking ethics applications against standards

Large language models can check research ethics applications against published quality standards in minutes, but their reliability differs sharply between tools. A University of Bristol study, carried out with the Health Research Authority (HRA) and published in Scientific Reports, found that Claude 3.5 Sonnet clearly outperformed ChatGPT-4o and Microsoft Copilot on this administrative task.

The task being tested

Before a study can begin, its application must pass a series of checks. These can involve reading hundreds of pages and comparing information across several documents. The work is essential, but it takes up a large share of the approval process and falls on approvals staff and committee members.

The team asked whether language models could take on part of this load. They gave the three models the job of checking applications against the HRA's published quality standards. They then compared accuracy, speed and response quality, and examined the kinds of mistakes each model made.

Accuracy, speed and error detection

All three models could complete the checks, but the results were far from equal. Claude gave fully correct answers to 87.9% of checks, against 66.5% for ChatGPT and 62.8% for Copilot.

Speed also differed. Claude averaged just over seven minutes per application, ChatGPT 13 minutes and Copilot 23 minutes. Copilot also failed to answer data protection questions in most of the applications.

The gap was widest for finding problems that needed correcting:

  • Claude identified an average of eight errors per application.
  • ChatGPT identified an average of 2.3.
  • Copilot identified an average of 1.3.

Across all three models, failures in logical reasoning were the most common cause of wrong answers. Claude produced fewer hallucinations, meaning incorrect or unsupported statements, than the other two.

Checking is not judging

The authors stress that verifying whether an application meets documented requirements is a different activity from deciding whether a study is ethically acceptable. The second requires human judgement and cannot be reduced to a checklist. Lead author Dr Panos Maghsoudlou, an NIHR academic clinical lecturer in ophthalmology at Bristol, describes a possible role for AI as an assistant rather than a replacement for approvals staff, researchers or ethics committees.

In that role, a model might flag missing information and departures from standards before an application reaches a committee. Staff could then spend more of their time on the substantive ethical questions. The researchers also warn of the risks of relying on model outputs without verification, a point underlined by the hallucinations and reasoning errors observed.

Implications and next steps

For research administrators and ethics committees, the main lesson is that model choice matters. A tool that misses most errors or skips whole categories of questions, as Copilot did with data protection, could give false reassurance. The results also suggest that any deployment would need human verification built in.

Dr Jonathan Fennelly-Barnwell, director of approvals at the HRA and a co-author, said the results were promising but that further exploration is needed before the authority can make recommendations on adopting AI. The HRA is continuing to test and evaluate the safe and responsible use of such tools. The source does not give the number of applications tested, so the scale of the evaluation cannot be judged from this summary. Performance also reflects the specific model versions tested, which are likely to change.

The paper is titled "Evaluating large language models for quality control in research ethics review" (DOI: 10.1038/s41598-026-62027-y).

Source: AI could help speed up research ethics approvals (Phys.org, 17 November 2026).

Image: AI-generated illustration.

Working on this topic? Find a journal for your research or join the discussion on Scholars Post.