Quality assessment of the review articles is performed using modified AMSTAR (Burda, Holmer and Noris 2016; Shea et al. 2007) and AMSTAR-2 (Shea et al.2017) checklist. The preliminary quality assessment is carried out using 16 quality assessment questions.
Based on the scoring for preliminary questions and also based on the subjective overall evaluation of the assessors, an overall rating (Quality Index) is assigned to each review article, highlighting whether major concerns arose during the quality assessment that may affect overall conclusions of the review article. Possible quality indices are:
Then a risk of bias score is assigned to each review article to represent how likely are the main conclusions of the review to be biased. Possible risk of bias scores are:
The list of questions used for the preliminary quality assessment is provided in the following table, along with the marking scheme.
| Question ID | Question description | Criteria for 'Yes' | Criteria for 'No' | Criteria for 'Cannot Answer' |
|---|---|---|---|---|
| 1 | Are the research questions and inclusion criteria for the review clearly delineated? | 1 = “Yes” = Who (Population), What (Intervention, Comparator group, Outcome), Where and When described. | 0 = ”No” = research question and inclusion criteria not outlined in detail. | 0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. Only partial description |
| 2 | Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? | 1 = “Yes” = The authors state that they had a written protocol or guide that included ALL the following: review question(s), a search strategy, inclusion/exclusion criteria, a risk of bias assessment. | 0 =”No” = no mention of a priori design of the systematic review, as listed above. | 0.5 = “Can’t answer / not sure / partially” = The authors state that they had a written protocol or guide that included some of the following: review question(s), a search strategy, inclusion/exclusion criteria, a risk of bias assessment. |
| 3 | Did the review authors explain their selection of the study designs for inclusion in the review? | 1 = “Yes” = explicit justification of the study designs/types included in the review. | 0 =”No” = no explicit justification of the study designs/types included in the review. | 0.5 = “Can’t answer / not sure / partially” =Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 4 | Did the review authors use a comprehensive literature search strategy? | 1 = “Yes” = searched at least 2 databases (relevant to research question), provided key word and/or search strategy, justified publication restrictions (e.g. language), AND searched the reference lists / bibliographies of included studies, searched trial/study registries, included/consulted content experts in the field, where relevant, searched for grey literature, conducted search within 24 months of completion of the review. | 0 =”No” = no information on search strategy, or not fulfilling criteria for “Yes” and “Partially”. | 0.5 = “Can’t answer / not sure / partially” = searched at least 2 databases (relevant to research question), provided key word and/or general search strategy, justified publication restrictions (e.g., language). |
| 5 | Did the review authors perform study selection in duplicate? | 1 = “Yes” = either ONE of the following: at least two reviewers independently agreed on selection of eligible studies and achieved consensus on which studies to include OR two reviewers selected a sample of eligible studies and achieved good agreement (at least 80%), with the remainder selected by one reviewer. | 0 =”No” = only one reviewer involved in the study selection or no description how many reviewers participated in study selection. | 0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 6 | Did the review authors perform data extraction in duplicate? | 1 = “Yes” = either ONE of the following: at least two reviewers achieved consensus on which data to extract from included studies OR two reviewers extracted data from a sample of eligible studies and achieved good agreement (at least 8 %), with the remainder extracted by one reviewer. | 0 =”No” = only one reviewer involved in the study selection or no description how many reviewers participated in data extraction. | 0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 7 | Did the review authors provide a list of excluded studies and justify the exclusions? | 1 = “Yes” = provided a list of all potentially relevant studies that were read in full-text form but excluded from the review AND justified the exclusion from the review of each potentially relevant study. | 0 =”No” = No list of studies excluded at a full-text stage. | 0.5 = “Can’t answer / not sure / partially” = only provided a list of all potentially relevant studies that were read in full-text form but excluded from the review, but not justified the exclusion from the review of each potentially relevant study that were read in full-text. |
| 8 | Did the review authors describe the included studies in adequate detail? | 1 = “Yes” = ALL the following: Who (Population), What (Intervention, Comparator group, Outcome), Where and When described in detail. | 0 =”No” = no, or partial description of the included studies | 0.5 = “Can’t answer / not sure / partially” = Who (Population), What (Intervention, Comparator group, Outcome), Where and When briefly described, or only some of these described in detail. Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 9 | Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review? | 1 = “Yes” = specifically mentions RoB assessment of individual included studies. | 0 =”No” = no mention of RoB assessment of individual included studies. | 0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. RoB mentioned or not sufficiently assessed (e.g. if multiple sources of bias potentially present, but not all assessed). |
| 10 | Did the review authors report on the sources of funding for the studies included in the review? | 1 = “Yes” = Must have reported on the sources of funding for individual studies included in the review. Note: Stating that the reviewers looked for this information but it was not reported by study authors, also qualifies. | 0 =”No” = no report of the sources of funding for individual studies included in the review. | 0.5 = “Can’t answer / not sure / partially” = sources of funding mentioned for individual studies included in the review, or reported only for some of the included studies. Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 11 | If meta-analysis was performed did the review authors use appropriate methods for statistical combination of results? | 1 = “Yes” = The authors justified combining the data in a meta-analysis AND they used an appropriate technique to combine study results and adjusted for heterogeneity if present AND investigated the causes of any heterogeneity or adjusted for heterogeneity or confounding if present. | 0 =”No” = no justification of meta-analysis or inappropriate statistical methods were used for quantitatively combining and analysing the data, heterogeneity not assessed. | 0.5 = “Can’t answer / not sure / partially” = Requirements for “Yes” only partially fulfilled. Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 12 | If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis? | 1 = “Yes” = included only low risk of bias studies OR the authors performed analyses to investigate possible impact of RoB on summary estimates of effect. | 0 =”No” = no assessment of the potential impact of RoB. | 0.5 = “Can’t answer / not sure / partially” = searched at least 2 databases (relevant to research question), provided key word and/or general search strategy, justified publication restrictions (e.g., language).0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 13 | Did the review authors account for RoB in individual studies when interpreting/ discussing the results of the review? | 1 = “Yes” = included only low risk of bias studies OR the review provided a discussion of the likely impact of RoB on the results. | 0 =”No” = no discussion of the potential impact of RoB in individual studies. | 0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 14 | Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? | 1 = “Yes” = There was no significant heterogeneity in the results OR if heterogeneity was present the authors performed an investigation of sources of any heterogeneity in the results and discussed the impact of this on the results of the review. | 0 =”No” = No explanation or discussion of heterogeneity present in the results. | 0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 15 | If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? | 1 = “Yes” = The authors performed graphical or statistical tests for publication bias and discussed the likelihood and magnitude of impact of publication bias. | 0 =”No” = The authors did not perform any tests for publication bias and did not discuss potential impact of publication bias. | 0.5 = “Can’t answer / not sure / partially” = more than one online source but no supplementary sources or one online source and one supplementary source. Cannot decide between “yes” and “no”, basing on the information provided in the paper. |
| 16 | Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? | 1 = “Yes” = The authors reported no competing interests OR the authors described their funding sources and how they managed potential conflicts of interest. | 0 = ”No” = The authors did not provide statement on competing interests and funding sources, and how they managed potential conflicts of interest. | 0.5 = “Can’t answer / not sure / partially” = Cannot decide between “yes” and “no”, basing on the information provided in the paper. |