How accurate is ClinLit AI? 50 test questions, checked against the sources

An AI research tool is only useful if you know how often it’s right. So we put ClinLit AI through a strict test: 50 new questions, chosen at random, with every answer checked against sources outside its own pipeline.

ClinLit AI got the main answer right on 47 of the 50 questions, with no critical errors. On 37 of them, 74%, the graders found no error at all, even in side details. We held it to a hard standard: one wrong secondary fact was enough to cost an answer its “correct”. Every answer with an error is listed below.

Key findings

• No critical errors. Neither grader found a wrong drug, dose or safety fact in any answer.
• Trial history, FAERS statistics and the GCP guideline were clean. All 7 answers held up, including adverse event statistics recomputed from the raw reports.
• Documents, FDA data and single-trial facts were reliable. The main answer was right on all 17 of those questions, and direct quotations matched the source documents.
• Registry counts reproduce. The main answer was right on 16 of 18 registry questions, under the definitions the answers stated.
• It knows what it doesn’t know. Given a made-up trial ID, a made-up drug and a request for data it doesn’t hold, it said so each time and invented nothing.
• Its citations are real. Every trial and paper it cited exists, apart from the made-up ID we planted in a trap question. Of 132 cited records read in detail, one was misdescribed.

Putting these errors in context

Clinical trial data is messy. Conditions on ClinicalTrials.gov are largely free text, so one disease can appear under several names, spellings and word orders. Sponsors file under names that change over time, and the text of some posted PDFs is badly extracted. Some of the errors listed below come directly from that.

People working with the same data make the same kinds of mistakes, even with expertise and training: a search that misses a synonym, a miscount, a misread table. So 100% isn’t a realistic expectation of an AI tool, any more than it is of an expert analyst. We didn’t test people on these questions, so we can’t say how the two compare. What matters is being able to find the errors, which is why ClinLit AI states its definitions and shows the queries and sources behind each answer.

Help us get it right

If ClinLit AI gives you a wrong answer, please tell us. Rate the answer with the thumbs-down button beside it, use the Feedback button, or write to [email protected]. We will keep improving ClinLit AI based on what you find. It’s the only way to get better, and we’re grateful for your help.

The results

Two graders marked each answer, and the stricter verdict counts.

“No error found” includes answers that correctly said the data couldn’t answer the question. A medium error is a wrong count beyond 2%, a wrong trial or sponsor, invented detail, or a citation that doesn’t support the claim. A critical error could mislead a professional decision, such as a wrong drug or safety fact.

We weighted the test towards registry counts on purpose, because they’re the hardest questions to get exactly right: every count depends on how a condition, sponsor or status is defined.

How we tested

• Questions fixed in advance. All 50 were frozen before any answer was generated, and none had been used in earlier testing or tuning.
• Chosen at random, not by hand. Trials, documents, drugs and conditions came from a reproducible random selection, leaving out topics ClinLit AI’s instructions had been tuned on. Some are odd as a result, and we kept them.
• Traps. Some questions used a trial ID or drug that doesn’t exist, or asked for data ClinLit AI doesn’t hold.
• Run as a user’s question would be. Each question went through the relevance filter and then the agent, on the same model the app uses, and the first answer was scored. All 70 runs, including repeats of ten questions, finished without an error or hitting a usage limit.
• Graded by two AI graders working separately, neither seeing the other’s verdicts.
• Checked against outside sources. The graders used the live ClinicalTrials.gov and PubMed records, FDA’s data, the posted PDFs’ text and the guideline itself. Counts were recomputed with separately written queries, and FAERS statistics from the raw reports.
• A strict, fixed rubric. An answer was correct only if every key claim held up. A count passed if it was within 2% under the definition the answer stated, provided that definition was a reasonable reading of the question.

What we’re improving

• Side figures. Headline answers hold up well, and most remaining errors are in the numbers around them. The commonest cause is adding condition and MeSH counts together, which overstates a term’s size.
• Disease names in unusual word order. One count missed the MeSH term “Depressive Disorder, Major” because it searched for “major depress”.
• Date-bounded PubMed counts. Our data stores the journal issue date rather than the online publication date, so a count limited to one year can come in below PubMed’s own.
• Poorly extracted PDFs. Some posted documents have broken spacing in their text, which stops keyword search from finding passages in them.

Fifty questions can’t give a precise error rate, and this isn’t a comparison with other tools. The best way to use ClinLit AI is the way we tested it: get to the evidence fast, then open the sources it cites before you rely on anything. Its answers aren’t medical, investment or regulatory advice.

Every answer with an error

Medium errors (3)

• Sponsors of major depressive disorder trials. Missed the MeSH term “Depressive Disorder, Major”: 111 trials where there are 128, which changed the ranking after first place.
• Stopped phase 3 trials on suicidal ideation. The five trials and their reasons were right, but it named the wrong sponsor for one, left a trial out of a phase 2/3 extension, and two side counts were too high.
• Presbyopia RCTs published in 2023. 6 in our data against 9 in live PubMed, with no warning that it might be low. The graders disagreed: this is the stricter verdict.

Minor errors (10)

In each of these, the main answer was right and a secondary detail was wrong.

• Completed phase 3 trials on surgical wounds. Answer right. The sizes it gave for condition terms added two sources together.
• Drugs in venous thromboembolism trials. Main ranking right. Said two trials named only a strategy when they name aspirin and apixaban, and the placebo count was off by one.
• Recruiting trials of venous leg ulcers. Count right. Gave the whole set as 346 when it is 326.
• Trials of glucose metabolism disorders. Counts right. Gave 916 as starts in 2026 when that figure covers 2026 onward.
• Phase 2 chronic pain trials with results. Share right. A side figure for a MeSH term added two sources together.
• Primary outcomes of a hangover drug trial. Outcomes listed correctly, but it said nine where there are eight.
• Sample size in a chlorhexidine trial protocol. Figures right. Flagged an inconsistency in the protocol that isn’t one.
• Primary outcomes in lung disease trials. Counts right. Some measures labelled “first listed” weren’t.
• Orange Book patents on Alvaiz. Patent, expiry and exclusivity right. Listed use codes incompletely.
• Phase 4 trials of low back pain. Explained that phase 4 trials aren’t in the data, but opened with “Zero”. The graders disagreed: this is the stricter verdict.