When an appraiser, a journal editor, or a guideline panel rates your systematic review as low or critically low confidence on AMSTAR-2, the rating rarely comes with an explanation you can act on. The instinct is to assume you scored poorly across many items. That is almost never what happened. AMSTAR-2 does not produce a score. It produces an overall confidence rating driven almost entirely by seven critical domains, and a weakness in a single one of them can move a review from high confidence to low, regardless of how well the other fifteen items are addressed. Understanding which domains are critical, what each one requires, and how the rating algorithm actually works is the difference between a review that is relied upon and one that is set aside. Our systematic review writing service builds reviews to satisfy these domains from protocol to publication.
This guide reproduces the AMSTAR-2 rating logic exactly as its developers defined it, identifies the seven critical domains and what satisfies each, and explains the errors that most often cause an otherwise competent review to be assigned critically low confidence.
Quick Answers:
AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews, version 2) appraises the methodological quality of a systematic review of healthcare interventions. It has 16 items, of which 7 are designated critical. It does not generate a numeric score. Instead, it yields one of four overall confidence ratings: High, Moderate, Low, or Critically Low. The algorithm is driven by critical flaws: no critical flaws and up to one non-critical weakness is High; more than one non-critical weakness but no critical flaw is Moderate; one critical flaw is Low; more than one critical flaw is Critically Low. Because the rating hinges on critical domains, a single unaddressed critical item, such as an unregistered protocol or an incomplete search, can drop a review two full categories, no matter how rigorous the rest of the work is.
What AMSTAR-2 Is, and What It Is Not
AMSTAR-2 is a critical appraisal tool for systematic reviews that include randomized controlled trials, non-randomized intervention studies, or both. It was developed by Beverley Shea and colleagues and published in the BMJ in 2017 as a major revision of the original 2007 AMSTAR instrument. The original tool contained 11 items and applied only to reviews of randomized controlled trials. AMSTAR-2 expanded to 16 items, added the capacity to appraise reviews that include non-randomized studies, simplified the response categories, and, most importantly, introduced the concept of critical versus non-critical domains that govern the overall rating. The tool and its guidance are hosted by its developers at amstar.ca, and the development and validation are reported in full in the BMJ paper.
Two distinctions matter before going further, because conflating them is the source of much confusion. First, AMSTAR-2 appraises the review, not the primary studies within it. The methodological quality of the individual included studies is assessed with a risk-of-bias tool appropriate to their design, such as RoB 2 for randomized trials or ROBINS-I for non-randomized studies of interventions, a distinction our guide on how to critically appraise studies covers in detail. AMSTAR-2 asks whether the review used the tool appropriately, not whether the studies were low risk of bias.
Second, AMSTAR-2 is a tool for appraising methodological quality, and it is not interchangeable with ROBIS, the other principal instrument in this space. ROBIS is a risk-of-bias tool structured around the phases of the review and the domains through which bias enters. AMSTAR-2 is a broader methodological-quality tool structured around discrete items. Either may be mandated by a journal or a funder, and some methodological studies apply both to triangulate their judgments. The choice depends on whether the requirement is for a methodological-quality rating or a risk-of-bias judgment.
The tool is intended to be applied by two independent reviewers, with disagreements resolved by discussion or a third reviewer, in the same way screening and data extraction are conducted in duplicate. A single-reviewer appraisal is a departure from the intended method and is itself a limitation worth noting.
The Response Options: Yes, Partial Yes, No
Each of the 16 items is rated with a small set of response options: Yes, No, and for some items, Partial Yes. The Partial Yes category exists to recognize situations where a review meets some but not all of the elements required for a full Yes. For example, an adequate search (item 4) earns a full Yes only when a defined set of components is present, including searching at least two databases, providing the keyword or search strategy, and justifying publication restrictions. A review that searches two databases but omits the full search strategy may earn a Partial Yes rather than a full Yes. Understanding the elements behind each Yes and Partial Yes is essential, as the difference between them can determine whether a critical domain is adequately addressed or a critical flaw.
The Four Ratings, and Why It Is Not a Score
AMSTAR-2 deliberately does not sum the items into a total. Its developers were explicit that adding items and reporting a figure such as '13 out of 16,' or converting that to a percentage, misrepresents the tool. The reason is methodological: not all items carry equal weight in determining a review's conclusions. A review can address most items while still being fatally compromised by a single weakness in a critical domain, and a scoring approach would obscure exactly that.
Instead, AMSTAR-2 yields one of four ratings of overall confidence in the review's results. Reproducing the developers' rules precisely:
High confidence. The review has no critical weaknesses and no more than one non-critical weakness. It provides an accurate and comprehensive summary of the results of the available studies.
Moderate confidence. The review has more than one non-critical weakness but no critical flaws. It may provide an accurate summary of the results of the available studies. More than one non-critical weakness distinguishes Moderate from High; multiple such weaknesses may reduce confidence, and it may be appropriate to move an appraisal from Moderate toward Low as they accumulate.
Low confidence. The review has one critical flaw, with or without non-critical weaknesses. It may not provide an accurate and comprehensive summary of the available studies.
Critically low confidence. The review has more than one critical flaw, with or without non-critical weaknesses. It should not be relied upon to provide an accurate and comprehensive summary of the available studies.
Table 1: The Four AMSTAR-2 Confidence Ratings and the Rules That Produce Them
Rating | Condition | What It Means for the Review |
|---|---|---|
High | No critical flaws; up to one non-critical weakness | Accurate and comprehensive summary of the available studies |
Moderate | More than one non-critical weakness; no critical flaw | May provide an accurate summary of the available studies |
Low | One critical flaw, with or without non-critical weaknesses | May not provide an accurate, comprehensive summary |
Critically Low | More than one critical flaw, with or without non-critical weaknesses | Should not be relied upon for an accurate summary |
The practical consequence of this structure is stark and worth stating plainly. Because the rating is governed by critical flaws, the number of non-critical items a review satisfies cannot rescue it from a critical weakness. A review with a single critical flaw is Low confidence, even if all fifteen other items are fully met. A review with two critical flaws is Critically Low. This is why so many published reviews carry a critically low rating, and why understanding the seven critical domains is the entire task.
The Seven Critical Domains
Of the 16 items, seven are designated critical because a weakness in any of them undermines the validity of the review's conclusions. The developers identified these specific domains as the ones with the greatest capacity to distort a review's findings. Reproducing them by item number and stating what each requires and why it is critical:
Item 2: Protocol registered before commencement of the review. The review methods must be established before the review is conducted, and any significant deviation from the protocol must be justified. A prospectively registered protocol, on PROSPERO or an equivalent registry, is the mechanism that prevents post hoc changes to the question, eligibility criteria, or outcomes from being made to fit the emerging results. Without it, an appraiser cannot know whether the review's decisions were made a priori or shaped by the data. Our guide on how to write a systematic review protocol and register it on PROSPERO covers the mechanics, and our protocol and PROSPERO registration service handles it directly. This is the critical domain most often flagged, because a review conceived without a registered protocol can never retrospectively satisfy it.
Item 4: Adequacy of the literature search. The search must be comprehensive. A full Yes requires, at minimum, searching at least 2 relevant databases, providing the keyword or full search strategy, and justifying any restrictions by publication date or language. Comprehensive searching is critical because a review can only synthesize the evidence it locates; an incomplete search means the synthesis may rest on a biased subset of the literature. Supplementary methods, such as searching trial registries, reviewing reference lists, and consulting experts, strengthen the search. The construction of a reproducible, peer-reviewable search is exacting enough that many reviews falter here, which is why it is one focus of our search strategy service.
Item 7: Justification for excluding individual studies. The review must provide a list of studies excluded at the full-text stage and justify each exclusion. This is critical because it is the only way an appraiser or reader can verify that no relevant study was excluded improperly. A review that reports only the studies it included, without accounting for those it read in full and rejected, leaves the integrity of its selection unverifiable.
Item 9: Risk of bias assessment of individual included studies. The review must use a satisfactory technique to assess the risk of bias in each included study, appropriate to the study design. For randomized trials, this means a tool that addresses unblinded allocation and lack of blinding; for non-randomized studies, it means a tool that addresses confounding and participant selection. This domain is critical because the certainty of a review's conclusions depends directly on the internal validity of the studies feeding into it. A synthesis that does not assess the risk of bias in its inputs cannot know how much to trust its own output.
Item 11: Appropriateness of meta-analytical methods. If a meta-analysis was performed, the review must have used appropriate statistical methods to combine the results. This includes justifying the choice of effect measure, using an appropriate model, investigating heterogeneity, and handling it correctly. This domain is critical because an inappropriate pooling method can generate a precise-looking summary estimate that is meaningless, for example, by combining studies that should never have been combined.
Item 13: Accounting for risk of bias when interpreting results. The review must account for the risk of bias in the individual studies when interpreting and discussing its findings. It is not enough to assess the risk of bias at item 9 and then ignore it in the discussion. This domain is critical because a review that reports a favorable pooled result without acknowledging that the underlying studies were at high risk of bias overstates the certainty of its own conclusion.
Item 15: Investigation of publication bias. If a quantitative synthesis was performed, the review must have adequately investigated publication bias and small-study effects, and discussed their likely impact. When studies with null or unfavorable results go unpublished, the available literature overstates the effect, and a synthesis of it inherits that overstatement. This domain is critical because undetected publication bias can invert the practical conclusion of an otherwise sound review.
Table 2: The Seven AMSTAR-2 Critical Domains
Item | Critical Domain | Why a Weakness Is Fatal |
|---|---|---|
Item 2 | Protocol registered before commencement | Prevents post hoc changes to question, criteria, or outcomes; cannot be added later |
Item 4 | Adequacy of the literature search | An incomplete search means the synthesis rests on a biased subset of the evidence |
Item 7 | Justification for excluding individual studies | The only way to verify that no relevant study was excluded improperly |
Item 9 | Risk of bias in included studies | Certainty of conclusions depends on the internal validity of the inputs |
Item 11 | Appropriateness of meta-analytical methods | Wrong pooling produces a precise-looking but meaningless estimate |
Item 13 | Accounting for risk of bias in interpretation | A favorable result from high-risk studies overstates certainty |
Item 15 | Investigation of publication bias | Unpublished null results mean the literature overstates the effect |
The Nine Non-Critical Items
The remaining nine items are non-critical, which does not mean unimportant. They are items 1, 3, 5, 6, 8, 10, 12, 14, and 16, covering, among other elements, whether the research questions and inclusion criteria included the components of PICO (item 1), whether the review explained its selection of study designs (item 3), whether study selection and data extraction were performed in duplicate (items 5 and 6), whether the review described the included studies in adequate detail (item 8), whether it reported the funding sources of the included studies (item 10), the effect of risk of bias on the meta-analysis (item 12), whether it accounted for heterogeneity in interpreting results (item 14), and whether it reported conflicts of interest (item 16). Weaknesses in these items reduce confidence, and more than one non-critical weakness moves a review from High to Moderate, but no accumulation of non-critical weaknesses alone can produce a Low or Critically Low rating. Only critical flaws do that.
Why So Many Published Reviews Are Rated Critically Low
The tool's structure has an empirical consequence that surprises many authors: a large proportion of published systematic reviews, including those in high-impact journals, are rated low or critically low on AMSTAR-2. In one cross-sectional meta-research study of systematic reviews that explicitly claimed to have followed AMSTAR-2, the majority were nonetheless rated critically low, a small number were low, and only one was rated high. The lesson is that stating adherence to a standard is not the same as meeting it, and that the critical-domain structure is demanding by design.
The reason is arithmetic. There are seven independent ways to acquire a critical flaw, and two critical flaws are enough for the worst rating. A review that fails to register a protocol (item 2) and does not investigate publication bias (item 15) is already critically low, even if its search, appraisal, and synthesis are exemplary. Because protocol registration cannot be added retrospectively, and because publication bias investigation is often omitted when the number of studies is small, these two alone account for a substantial share of critically low ratings. Understanding this in advance, at the design stage, is the only reliable defense, which is the same reason reviews are so often sent back or rejected, a pattern our guide on why systematic reviews get rejected examines.
Will your review survive an AMSTAR-2 appraisal? |
|---|
Send us your protocol or draft, and a methodologist will appraise it against all seven critical domains, tell you which would currently flag, and show you what each needs before an editor or guideline panel applies the tool. Request an AMSTAR-2 readiness check and receive an itemized quote within 2 to 4 business hours, no obligation. |
How AMSTAR-2 Relates to GRADE and to Reporting Guidelines
AMSTAR-2 is frequently confused with two other frameworks it sits alongside, and distinguishing them demonstrates command of the evidence-synthesis toolkit.
AMSTAR-2 assesses the methodological quality of the review, meaning how well the review was conducted. GRADE assesses the certainty of the evidence for each outcome, meaning how much confidence to place in the effect estimates the review produces. The two are complementary: a well-conducted review, rated high on AMSTAR-2, can still yield low-certainty evidence on GRADE if the underlying studies are at high risk of bias, imprecise, or inconsistent. Conversely, a poorly conducted review cannot be rescued by the certainty of its inputs. Appraisers and guideline developers routinely apply both AMSTAR-2 to judge the review and GRADE to judge the evidence.
Reporting guidelines are a third, separate category. PRISMA 2020 governs how a systematic review is reported, meaning what information the manuscript must contain and how it should be presented. It is not a quality-appraisal tool. A review can be reported in full compliance with PRISMA and still be rated critically low on AMSTAR-2, because complete reporting of a flawed method documents the flaw rather than removing it. The distinction is that PRISMA asks whether the review is reported transparently, while AMSTAR-2 asks whether it was conducted soundly. Building a review correctly from the outset addresses both, which is the logic of our step-by-step guide to writing a systematic review.
The Errors That Produce a Critically Low Rating
A compact set of recurring errors accounts for most low and critically low ratings, and each maps to a critical domain.
The first is conducting the review without a prospectively registered protocol, which permanently forfeits item 2. The second is an incomplete search, most often a single-database search or an omission of the full search strategy, which forfeits or partially forfeits item 4. The third is failing to provide a list of excluded studies with reasons, forfeiting item 7. The fourth is assessing risk of bias with a tool inappropriate to the study design, or omitting it, forfeiting item 9. The fifth is pooling studies with inappropriate meta-analytical methods, forfeiting item 11. The sixth is assessing risk of bias but then failing to carry it into the interpretation, forfeiting item 13. The seventh is omitting any investigation of publication bias when a meta-analysis was performed, forfeiting item 15. Because two of these are enough for a critically low rating, and because several tend to co-occur, a review that neglects the critical domains at the design stage is very difficult to rehabilitate afterward. The appraisal outcome is largely determined before the first database is searched, whether or not the protocol is written.
Frequently Asked Questions
Is AMSTAR-2 scored out of 16?
No. AMSTAR-2 does not produce a numeric score; reporting a figure such as 13 out of 16 or a percentage is a misuse of the tool. Its developers were explicit that items are not summed because they do not carry equal weight. The output is one of four overall confidence ratings, High, Moderate, Low, or Critically Low, determined by the presence of critical flaws rather than by a count of satisfied items.
What are the seven critical domains in AMSTAR-2?
The seven critical domains are item 2 (protocol registered before the review commenced), item 4 (adequacy of the literature search), item 7 (justification for excluding individual studies), item 9 (risk of bias assessment of included studies), item 11 (appropriateness of meta-analytical methods), item 13 (accounting for risk of bias when interpreting results), and item 15 (investigation of publication bias). A weakness in any one of these can drop a review to Low confidence.
How does one critical flaw affect the rating?
A single critical flaw drops the overall rating to Low confidence, regardless of how well the other fifteen items are addressed. More than one critical flaw drops it to Critically Low. This is why the number of non-critical items a review satisfies cannot compensate for a weakness in a critical domain, and why a single omission, such as an unregistered protocol, has such a large effect.
What is the difference between AMSTAR-2 and ROBIS?
AMSTAR-2 is a methodological quality appraisal tool structured around 16 discrete items and an overall confidence rating. ROBIS is a risk-of-bias tool structured around the phases of a review and the domains through which bias enters. They serve overlapping but distinct purposes; either may be required by a journal or funder, and some methodological studies apply both to triangulate their assessment of a review.
Does AMSTAR-2 assess the primary studies or the review?
AMSTAR-2 assesses the review, not the primary studies within it. The risk of bias in the individual studies is assessed separately using a design-appropriate tool, such as RoB 2 or ROBINS-I. AMSTAR-2 item 9 asks whether the review used such a tool satisfactorily, but the tool itself does not appraise the individual studies.
Can a review compliant with PRISMA 2020 still be rated critically low?
Yes. PRISMA 2020 is a reporting guideline that governs what a review reports and how transparently it is reported, not how soundly it was conducted. A review can report a flawed method in full PRISMA compliance and still be rated critically low on AMSTAR-2, because transparent reporting of a critical flaw documents the flaw rather than removing it. Sound conduct and complete reporting are separate requirements.
Can I fix an AMSTAR-2 critical flaw after completing the review?
Some critical flaws can be remedied; others cannot. An incomplete search can be extended, a list of excluded studies can be compiled, and a publication bias investigation can be added if enough studies are available. But a protocol that was never registered before the review commenced (item 2) cannot be satisfied retrospectively, because prospective registration is the entire point of the domain. This is why the critical domains must be addressed at the design stage.
You May Also Find Useful
Designing for the Rating You Want
The reviews that earn high confidence on AMSTAR-2 are not the ones that scramble to satisfy the tool after the fact. They are the ones designed around the seven critical domains from the beginning: a protocol registered before any screening, a comprehensive and documented search, a full account of excluded studies, a design-appropriate risk-of-bias assessment carried through into the interpretation, sound meta-analytical methods, and an investigation of publication bias. Address those seven at the protocol stage, and the appraisal takes care of itself.
If your review is heading toward external appraisal and you want to know which critical domains would flag before an editor or panel applies the tool, send us your protocol or manuscript for an AMSTAR-2 readiness check. A methodologist will return an itemized quote within 2 to 4 business hours, with no obligation.
