Risk of Bias Comes After Extraction — and Must Not Be Faked
Appraisal judges whether an extracted study can support an inference. LitSynth does not implement Cochrane RoB 2, ROBINS-I, or GRADE. Report the order, use the table as a worksheet, and do not mint Low/High badges.
Institutions ask about risk of bias because that is where vendors overclaim. Screening asks whether a record is eligible. Extraction records what the study did. Appraisal asks whether that extracted study can be biased in a way that changes the answer. Mixing those three judgements is how a relevance score becomes a fake “low risk of bias” badge.
LitSynth does not implement Cochrane RoB 2, ROBINS-I, QUADAS-2, AMSTAR, or GRADE. That is intentional. A generated Low / Some concerns / High label without domain signalling questions would be worse than an honest gap. The citable page is risk-of-bias methods. This essay is the argument you need when a protocol, a librarian, or a procurement review asks anyway.
The method, in one paragraph
Study validity was considered using extracted study design, evidence level, and author-reported limitations. Formal domain-based risk-of-bias assessment (for example Cochrane RoB 2 or ROBINS-I) and GRADE certainty rating were not performed in LitSynth. Unless a separate appraisal is described, this review should not be interpreted as having completed those instruments.
Keep that sentence. Deleting it to sound more systematic is the failure mode methods reviewers are trained to catch.
Why appraisal cannot sit in screening
You cannot appraise a study you have not reduced to design, outcomes, and limitations. Title/abstract screening decides eligibility. Extraction produces the inputs. Only then is a RoB instrument meaningful, and even then it usually needs full text.
Doing RoB on abstracts invents certainty about allocation concealment, missing outcome data, and selective reporting that the abstract does not contain. Doing RoB on unscreened hits appraises ineligible research. Both look like diligence. Both are noise.
Separate three judgements that software likes to collapse
| Judgement | Question | What LitSynth records |
|---|---|---|
| Relevance | Does this address the review question? | Relevance score and screening decision |
| Directness | How closely does the PICO match? | Extracted population, comparator, endpoint, follow-up |
| Internal validity | Could bias change the answer? | Design, evidence level, limitations — not a RoB 2 assignment |
A paper can be highly relevant and still high risk of bias. A paper can be low risk of bias for an outcome you do not care about. Collapsing those into one traffic light is how AI review tools get procurement teams into trouble.
If the protocol requires RoB 2, use the table as a worksheet
Finish screening and extraction in LitSynth. Export the evidence table. Complete Cochrane RoB 2 on included randomized trials, or ROBINS-I on non-randomized studies, in a separate form using full text. Cite both steps. Do not write that LitSynth performed RoB 2.
Carry across: design, randomization clues, comparator, outcome, follow-up, and author-reported limitations. Do not relabel those limitations as RoB 2 domains. “The authors noted short follow-up” is not the missing-data domain.
What not to write
- “Low risk of bias” unless a named instrument produced that label.
- “We assessed quality with AI.”
- GRADE certainty language (high / moderate / low / very low) unless you ran GRADE.
- That extracted limitations equal a validated bias tool.
If you only intended a rapid or scoping review, say that formal RoB was out of scope. That is a legitimate design choice. Pretending the instrument ran is not.
Where this sits in the series
- Screen titles and abstracts you can defend
- Extract an evidence table
- This page — appraise, or honestly decline to
- Turn residual holes into the next research question
Gaps without extraction are marketing. RoB without extraction is theatre. The methods library exists so those sentences can be cited instead of inferred from a feature page.