How to benchmark a clinical trial design against earlier trials and their protocols: a repeatable method

We wrote this alongside How to design a phase 2b MASH trial, so that you can run the same analysis for another disease. It sets out the questions we asked, what each one needs, and the mistakes we caught. It was written on 10 November 2026, using ClinicalTrials.gov data as of 9 November; the data is updated daily.

The idea is simple. Every trial design rests on a few numbers: the response you expect on placebo and on the drug, how many people you must screen to randomise one, how many will drop out or miss the endpoint, how long it takes and at how many sites. Earlier trials in the same disease have measured all of these. Their registry records give the operational numbers, their posted protocols and statistical analysis plans (SAPs) give the assumptions they made, and their papers give what actually happened. Comparing the last two is the most useful thing you can do before writing your own sample size section.

You can do this with ClinLit AI, or directly on the public sources it uses: ClinicalTrials.gov (through AACT), the protocols and SAPs sponsors post there, PubMed, FDA labels and the FDA and EMA websites.

The MASH post took 24 questions. Each took between 38 seconds and under three minutes. Two of them taught us the most: one silently miscounted, and one gave a confident recommendation built on a number that turned out to be wrong.

Step 1: write down the numbers your design depends on

Before asking anything, list the assumptions your protocol will need. For a randomised trial with a binary endpoint they are usually:

• the response rate on placebo, and on the drug;
• the share of patients who will drop out or miss the endpoint assessment;
• the screen failure rate;
• the duration, the number of sites and countries, and the number of arms;
• the eligibility criteria that drive recruitment;
• how the endpoint is assessed (for MASH, who reads the biopsy) and how missing data are handled;
• what regulators accept as an endpoint.

Each becomes one or more questions below. Keeping the list in front of you stops the research drifting into a general landscape review.

Step 2: define the comparison set

This step decides every count after it. Choose the disease, the phases and the study type, then sort trials by what their primary endpoint measures, as an ordered list of categories where the first match wins. Paste the whole definition into the question:

Define [AREA] phase 2 trials as interventional studies in aact with phase in (‘PHASE2’,‘PHASE2/PHASE3’) and at least one condition (conditions.name) matching the case-insensitive regular expression ‘[CONDITION PATTERN]’, which leaves out [WHAT YOU EXCLUDE]. Classify each trial by its primary outcomes (design_outcomes with outcome_type = ‘primary’, measure and description together), first match wins: 1 ‘[CATEGORY 1]’ if they match ‘[PATTERN 1]’; 2 ‘[CATEGORY 2]’ if ‘[PATTERN 2]’; … ; 6 ‘Other’. How many [AREA] phase 2 trials are there, and how many by overall status? Then count them by primary endpoint category and start period ([PERIODS]), saying how many of the latest period’s trials have an ESTIMATED rather than ACTUAL start date.

For MASH we used the condition pattern nash|steatohepatitis|[[:<:]]mash[[:>:]] and six categories: liver histology, liver fat by imaging, liver stiffness, liver enzymes, safety or pharmacokinetics only, and other. The full question is in the MASH post.

Three things to get right:

• Write word boundaries as [[:<:]] and [[:>:]], not \m and \M. Short terms such as NAS, MRI or ALT need boundaries, or ALT matches “alternative”. When we wrote \m, ClinLit AI’s SQL doubled the backslash, which in PostgreSQL matches a literal backslash: those terms silently matched nothing, and it counted 72 histology trials instead of 77. The answer looked complete. The bracket form needs no backslashes and gave the right count.
• Order the categories so the one you care about comes first. A trial whose primary outcomes mention both a biopsy and liver fat then counts as a biopsy trial.
• Read what the definition leaves out. ClinLit AI ends a definition answer with what it may miss; for MASH, trials registered only as fatty liver disease. Decide whether that matters for your question before going on.

Follow-up questions in the same conversation reuse the definition. A conversation keeps its last 10 messages, so paste it again in a new one.

Step 3: get the operational benchmarks

In the same conversation:

For the ‘[CATEGORY 1]’ and ‘[CATEGORY 2]’ trials that are COMPLETED with enrollment_type ACTUAL, give for each category: the number of trials; the median and interquartile range of enrollment; the median number of arms (design_groups); the median months from start_date to primary_completion_date; the median number of facilities (calculated_values.number_of_facilities); the share with a US facility; the median number of countries; and the share that are randomized and the share with masking QUADRUPLE. Then repeat it for the same trials started in [RECENT YEAR] or later.

Use completed trials with actual enrolment, or you will be averaging plans. Repeat it for recent trials: in MASH, biopsy trials started since 2019 were bigger, more international and more uniform than earlier ones (median 201 patients at 53 sites, against 188 at 44 for all years). The older trials describe a different era.

ClinLit AI flagged its own caveat here, worth keeping: “masking QUADRUPLE” is a floor on blinding, because many double-blind trials are recorded as DOUBLE.

Step 4: choose which trials’ protocols to read

Not every trial posts its protocol, and fewer post an SAP too. Ask for the ones that have both:

List the ‘[CATEGORY 1]’ trials that started in [YEAR] or later, have enrollment of at least [N], and have both a study protocol and a statistical analysis plan posted (aact.study_documents). For each give NCT ID, drug, lead sponsor, start year, status, enrollment and primary endpoint, largest enrollment first.

For MASH this gave 21 trials, the same 21 our own query found. Read the list before choosing. Keyword categories catch oddities: our “liver histology” list included a trial whose primary measure is a blood test (it mentions histology in its description) and a breath-test device study.

We chose six of the 21 that also had a published primary results paper, so the assumptions could be compared with the outcome. That makes the sample larger and better resourced than most trials. Say so when you report it.

Step 5: ask each protocol for its assumptions, one trial at a time

One trial per question, asking for quotes and PDF pages:

For the phase 2 [DRUG] [AREA] trial [NCT ID]: From its posted protocol and statistical analysis plan, answer: (1) the primary endpoint and when it is measured; (2) how the sample size was justified: the assumed placebo and active response rates, power, alpha, multiplicity and dropout; (3) [HOW THE ENDPOINT IS ASSESSED: for MASH, how the baseline biopsy was handled: whether a historical biopsy was allowed and how old, central reading, how many pathologists and how disagreements were resolved]; (4) [WHAT SCREENING STEPS CAME FIRST: for MASH, which non-invasive tests or imaging were used to pre-screen before the biopsy, with thresholds]. Quote the key sentences and cite the document and PDF page for each answer. Say clearly if something is not in the documents.

These were the richest answers, and each took under three minutes. What we learned from six of them:

• The assumptions are usually in the protocol, not the SAP. Semaglutide’s SAP says only “See protocol section 17.1 for details”.
• One PDF can hold several protocol versions. ClinLit AI noticed that semaglutide’s posted protocol also contains version 1, which planned 372 patients and a 50% screen failure rate, against 288 and 65% in version 5. Changes between versions show what the sponsor learned during the trial.
• “Not in the documents” is an answer. Two of six protocols gave no rule for when pathologists disagree, and two more deferred it to a manual that was not posted. Those gaps are design lessons too.
• Some documents are redacted. Efruxifermin’s sample size paragraph and screening thresholds are black boxes in the posted PDF. ClinLit AI said it could not read them and guessed a text-extraction problem; the page image showed the redaction. When an answer says text is blank, open the PDF page before concluding anything. The registry’s eligibility text sometimes gives what the protocol hides: it states the FibroScan cut-off that the protocol blacks out.

Step 6: follow each assumption to the result

This is the step that turns a document review into evidence. In the same conversation as the protocol question:

Now find this trial’s primary results publication in PubMed (through the trial’s linked references or a search). What placebo and active response rates were observed for the endpoints the sample size was based on, and how many participants had a [ENDPOINT ASSESSMENT, e.g. biopsy] at the end of treatment? Compare them with the assumptions in the protocol or SAP. Cite the PMID.

ClinLit AI found each paper through the trial’s registry references, read it, and laid out assumption against observation. For MASH the placebo assumptions of 10% to 20% met observed rates of 2% to 33%, and planned dropout of 5% to 20% met 12% to 25% of patients without a final biopsy.

Check that the comparison is like for like. Papers often analyse differently from the plan:

• the population can differ (semaglutide planned its sample size on all patients but reported the confirmatory result in the F2–F3 subgroup);
• the handling of missing data can differ (lanifibranor’s paper used multiple imputation as its primary analysis, where the SAP said missing = non-responder);
• the paper may add a multiplicity adjustment the SAP did not plan (tirzepatide’s post hoc adjusted P values).

ClinLit AI pointed out all three. Keep those notes next to the numbers.

Step 7: ask the literature for the range

Single trials give you points; reviews give you the spread. Three questions covered what we needed:

In PubMed, find systematic reviews or meta-analyses of the placebo response in randomized trials of [AREA] with [ENDPOINT TYPE] endpoints. Report the pooled placebo rates for [ENDPOINT 1] and for [ENDPOINT 2], how many trials each pooled, and which trial features were associated with a higher placebo response. Cite PMIDs.

In PubMed, what screen failure rates have phase 2 and phase 3 [AREA] trials with [ENDPOINT TYPE] endpoints reported, and what drove them? Cite PMIDs.

In PubMed, find studies of how much [ASSESSORS, e.g. pathologists] disagree when [ASSESSING THE ENDPOINT], and what that means for trial endpoints. Cite PMIDs.

ClinLit AI read abstracts, and said so; the numbers you quote should be in the abstract, or you should open the paper. We checked every PMID it cited against PubMed, and the key numbers against the abstracts. The literature usually gives a centre (about 11% placebo response for MASH resolution) and the trials show how far from it a single trial can land.

Step 8: compare eligibility across trials

For the ‘[CATEGORY 1]’ trials with an INDUSTRY lead sponsor that started in [YEAR] or later and have enrollment of at least [N], read the eligibility criteria and make a table with one row per trial: [THE CRITERIA THAT MATTER, e.g. disease stage, severity score, how recent the qualifying test must be, screening thresholds, key comorbidities allowed, BMI limits]. Then summarise what most trials have in common and where they differ.

This is where an assistant saves the most time: it read the free-text criteria of 31 trials in under three minutes. Two cautions it gave itself: its tallies were “my hand-count of the table rows I read, not a re-run query”, and five registry records were abridged (“other criteria may apply”). Treat the summary as a reading, and check the rows you rely on.

Step 9: see why trials stopped

List the [AREA] phase 2 trials that are TERMINATED, WITHDRAWN or SUSPENDED, with their primary endpoint category, lead sponsor, start year, enrollment and why_stopped. Then group the reasons into: business or strategic decision; efficacy or futility; safety; slow enrolment or recruitment; COVID-19; other or not given. Give the count in each group, by primary endpoint category.

ClinLit AI grouped each reason and listed its judgement calls. Check its arithmetic in prose: it called 12 of 33 “nearly two-thirds”; it is 36% of all, or 43% of the 28 that gave a reason. Its table was right; the sentence wasn’t.

Step 10: check what regulators and labels say

As of [DATE], what does FDA guidance say about the endpoints for phase 2 and phase 3 trials in [AREA], and has it been updated or finalised? What does the EMA say? What has either agency said about [THE ALTERNATIVE ENDPOINT YOU ARE CONSIDERING]? Search official FDA and EMA sources and cite them.

From the FDA labels in our data for [APPROVED DRUGS IN THE AREA]: what is the indication, which trial supported it, what were the endpoints and the placebo and drug results, and is the approval accelerated? Quote the label sections.

Ask for official sources only, and open the links. Labels are an underused source for design: they give phase 3 placebo rates on the regulator’s own analysis. Resmetirom’s label reports each pathologist’s reading separately, which showed the reader effect more clearly than any paper we found.

Step 11: check the answers

Check	How	What it caught
Counts	Run the definition in your own SQL	The doubled backslashes: 72 against our 77
Cited pages	Open each protocol page cited, in the PDF	Every quote we checked was on the cited page or page range
Blank text	Look at the page image	Redaction, not an extraction failure
PMIDs	Look up each one, and the quoted numbers in the abstract	Numbers taken from full texts, which we then labelled as such
Quotes in the post	Compare each excerpt with the saved answer	Nothing, which is the point: keep the transcripts
Prose against tables	Recompute any share or ratio stated in words	“Nearly two-thirds” that was 36%

Our own SQL matched ClinLit AI’s counts exactly once the definition was fixed: the trial count, every cell of the endpoint table, the size and duration medians and the list of stopped trials. We kept every answer with its tool calls as JSON, so each excerpt can be checked against the original later.

The traps, in one place

Trap	What happened	How to avoid it
An open question repeats an assumption as fact	Our first, open question recommended planning for “~1.3 screened per randomised”, which was one protocol’s 25% assumption; its SAP recorded 72%	Ask for each number’s source, then follow it to the result (step 6)
Escaped regular expressions	\m became \\m and matched nothing	Use [[:<:]] and [[:>:]]
Keyword categories	A blood-test endpoint and a device study counted as “histology”	Read the trial list
Redacted protocols	Sample size and thresholds blacked out	Open the page; try the registry record
Unlike comparisons	Different populations, imputation and multiplicity between plan and paper	Note them next to each number
Hand counts	Tallies from reading free text	Check the rows you use
Record history	ClinLit AI’s history of study record changes starts in July 2024, so it can’t show amendments made earlier in most of these trials	Use it for trials still running; read protocol versions for older ones

From evidence to a design

The output is not one number per assumption but a range, with the trials that set its edges. For MASH the design sketch at the end of the main post puts each decision next to what earlier trials did and what it suggests: plan around the literature’s centre for the placebo response, and show the sample size at both ends of the range the trials actually saw. That table is where your team’s judgement, and your regulators’ advice, come in. Nothing here is regulatory or medical advice.