M4 · EVIDENCE SYNTHESIS — LESSON 1
Why a systematic review?
Two experts read the same literature on the same drug and reach opposite conclusions. Neither lied. The gap between them is the whole reason this module exists.
Dr. A has spent fifteen years treating this condition. She writes a review concluding Drug X offers a clear, meaningful benefit.
Dr. B, equally experienced, reviews the same drug and concludes the benefit is negligible.
They had the same published trials in front of them. They did the same arithmetic. So how do two careful experts end up on opposite sides?
The answer isn't expertise, and it isn't honesty. It's a single methodological choice — one you can usually see only if you go looking for it.
The evidence base
Here is every published trial of Drug X. Each marker is one trial; its position is the measured effect on a 0–100 quality-of-life score — drug minus placebo. Further right is better for the drug.
This scatter is normal. Trials differ in size, population, and the play of chance, so their results spread out. Every reviewer faces the same question: what do you do with a picture this messy?
The narrative reviewer
Dr. A is writing a narrative review — an expert summary of the field, in her own judgement. She finds Drug X promising, so she builds her review on the three trials she finds most convincing: the ones at +12, +9, and +6.
Compute the average effect across those three trials.
(+12 + +9 + +6) ÷ 3 = ?
Dr. A's review concludes: Drug X delivers a large, patient-noticeable benefit of about +9 points. Every trial she cites backs it up. Her review is internally consistent, well-written, and completely sincere.
The systematic reviewer
Dr. B starts somewhere else entirely. Before looking at any result, she writes down a rule: include every trial that meets the criteria — same drug, same comparator, same outcome — no matter what it found.
That rule pulls in all eight trials, not three.
positives +12 +9 +6 +1 = +28
negatives −1 −3 −6 −10 = −20
Compute the average across all eight trials.
total = +28 − 20 = +8
total ÷ 8 = ?
Dr. B's review concludes: about +1 point — inside the noise, no benefit a patient would feel.
a clear win
nothing
Same drug. Same trials in the world. Same arithmetic. The only thing that differed was which trials were allowed in — and Dr. A's review never told you it left five out.
Diagnosis
It's tempting to call this dishonesty. Usually it isn't.
Dr. A genuinely believes Drug X works. The trials that confirm that belief are the ones she finds "convincing" and remembers most easily. She isn't hiding the other five — to her they were "flawed," "underpowered," "not representative." Every exclusion felt justified at the time.
That's the real problem, and it's worse than dishonesty: her selection is invisible and unauditable. She never declared a rule for what gets in, so there's nothing for you to check. You can't separate a fair exclusion from a convenient one. Two sincere experts can curate the same literature into opposite stories, and the reader has no way to referee.
The fix
A systematic review fixes exactly this. The fix is not the statistics — it's the order of operations.
Before seeing any results, a systematic review writes down:
- the question, in PICO terms
- where it will search, and how
- what makes a study eligible
Then it applies those rules to whatever comes back. The set of included studies — the denominator — is fixed before anyone knows the answers. That's what "systematic" means: not "thorough," but reproducible. Hand the protocol to a stranger and they should land on the same set of studies you did.
A trap worth naming
"Systematic review" is not a synonym for "meta-analysis." The meta-analysis — the single pooled number — is optional and comes last. You can run that pooling math on a cherry-picked set and the arithmetic won't object; the statistics never protect you from a rigged denominator. And a perfectly good systematic review may pool nothing at all, when the studies are too different to combine. Systematic lives in the search and the selection, not the sums.
Spot the tell
None of this makes narrative reviews worthless. For mapping a young field, generating hypotheses, or an expert's broad tour of a topic, a narrative review is the right tool. The bar rises only when a review is used to justify a decision — to spend a health system's money, or to grant a patient access. At that point, "trust my judgement about which trials matter" is no longer enough.
So you learn to spot the difference on sight. Here are three things you might read in an evidence summary. Tag each one.
"We searched MEDLINE, Embase, and CENTRAL up to 31 March 2025; the full strategy is in Appendix 2."
"The most relevant and rigorous trials consistently show a benefit."
"Studies meeting our pre-specified criteria were included regardless of their findings."
Why this matters for HTA
A manufacturer's dossier lands on your desk. Section 4 is a polished summary of the clinical evidence: every cited trial points the same way, the prose is confident, the conclusion is that the drug clearly works.
- The manufacturer is structurally in Dr. A's position. They are not neutral; they want the product to look good, and a narrative-style selection is the easiest route there — often without a single false statement.
- Your first job as an assessor isn't to judge the trials. It's to check whether the evidence base was assembled systematically or curated: is there a reported search? Pre-specified inclusion criteria? A count of what was found, screened, and excluded — and why?
- A confident conclusion resting on an undeclared selection is the most common soft spot in a submission. You rarely need to disprove their trials; you need to ask what's missing from the denominator.
"A narrative review tells you what one expert remembered. A systematic review tells you what the evidence actually contains."
Systematic reviews, in one breath
- A narrative review filters the literature through one expert's judgement; a systematic review applies a pre-specified, reproducible rule for finding and selecting studies.
- The danger of narrative selection usually isn't dishonesty — it's that the selection is invisible, so you can't tell a fair exclusion from a convenient one.
- "Systematic" means the included set is fixed before the results are known: reproducible, not merely thorough.
- Systematic ≠ meta-analysis. Pooling is optional, comes last, can't rescue a cherry-picked denominator — and a good review may pool nothing.
- Narrative reviews keep legitimate uses; the bar rises sharply the moment a review drives a spending or access decision.
"Systematic isn't a synonym for thorough. It means the rules for what counts as evidence were written down before anyone looked at the answers."
You now know why the systematic review exists and what makes it trustworthy. The next lessons make it operational: how the search and selection are documented and counted so a reader can audit them (PRISMA and the study-flow diagram), how a PICO question becomes an actual search string, and — once you hold a clean, complete set of studies — how to combine their results properly, which is where meta-analysis finally earns its place.