Hello!
When I was previously at USDA National Agricultural Library, we adopted a version of AMSTAR2 for critically appraising existing reviews and determining which could be adopted for use in setting dietary reference values instead of commissioning new reviews through AHRQ (methods publication forthcoming).
From my experience guiding a team of nutritionists through application of the tool, we definitely found the search strategy evaluation piece lacking in AMSTAR2. As a checklist, it really emphasized dichotomous and quantifiable aspects (at least 2 databases? searched registries and for grey literature?). In terms of the actual search string/construction, AMSTAR2 asks whether keywords were provided in the manuscript or appendix, but it's not able to measure whether they were the right keywords, or a complete set of keywords, or even if controlled vocabulary were employed, or if there were any mistakes in the search syntax, etc. See question 4 in the
AMSTAR2 checklist linked here for more.
In addition, while many important supplemental search techniques are called out in the "Yes" (as opposed to "Partial Yes") section of question 4, it is exceedingly rare and not always appropriate that all of these are employed when that is what is required to attain a full "Yes." And, again, the focus is on whether the technique was used not whether it was done well or applied correctly - and this is often the important difference that we want to tease out. Finally, some items are also difficult to operationalize. We had a lot of trouble understanding how to interpret "included/consulted content experts in the field." Who qualifies as an expert? And what does included/consulted mean? Do authors count as people included? And reporting of this would almost always be lacking unless one contacted authors for further information.
AMSTAR2 is a great tool but, if I'm understanding correctly that your focus is on the search strategy's quality, it may be broader than you would find useful. I was unfamiliar with the two papers from Rethlefsen and Koffel and thanks to Amy for sharing those as they will make for interesting reading!
Hopefully you find that useful in identifying the best approach. This is a really important area to be developed within evidence synthesis, so thank you for your attention and future scholarship!
Best,
Eric
Eric Toole (he/him) | Evidence Synthesis Librarian
Science & Engineering Library
University of Massachusetts, Amherst
(413) 545-6151
Original Message:
Sent: 7/18/2024 11:48:00 AM
From: Amy Riegelman
Subject: RE: Searching for Published Critiques of SR Searches
You could consider AMSTAR 2. See item 4 in the fulltext and supplemental files:
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., ... & Henry, D. A. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. bmj, 358.
https://doi.org/10.1136/bmj.j4008
Koffel, Rethlefsen, and others have analysed search methods of published reviews. You could use their studies as models---
- Koffel, J. B., & Rethlefsen, M. L. (2016). Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One, 11(9), e0163309. https://doi.org/10.1371/journal.pone.0163309
- Rethlefsen, M. L., Brigham, T. J., Price, C., Moher, D., Bouter, L. M., Kirkham, J. J., ... & Zeegers, M. P. (2023). Systematic review search strategies are poorly reported and not reproducible: a cross-sectional meta-research study. Journal of clinical epidemiology, 111229. dx.doi.org/10.1016/j.jclinepi.2023.111229
-- Amy Riegelman (she, her, hers)
Social Sciences & Evidence Synthesis Librarian, University of Minnesota
Original Message:
Sent: 7/17/2024 12:02:00 PM
From: Elizabeth Sanders
Subject: Searching for Published Critiques of SR Searches
Hello everyone,
I'm working on a team project (link opens to new window) with another librarian to evaluate and to critique searches published in a series of systematic reviews (link opens to new window) that were used to inform related to the Cass Review (link opens in new window).
Our current draft evaluation checklist incorporates components of PRESS, PRISMA-S and the peer review guidance (link opens in new window) included in: Townsend, W.A., MacEachern, M. P. & Song, J. (2020). Analyzing reviewer responses to systematic review search methodology through open peer review [Data set], University of Michigan - Deep Blue Data. https://doi.org/10.7302/acjm-cz18
As we're developing this checklist, we're wondering if others are using or could share examples of established post-publication search review tools/guidance? (As opposed to peer-reviewing a search prior to publication).
Also, does anyone know of any examples of published critiques like the one we're proposing? We're going to search the literature, but just in case thought it'd be good to ask the group.
Thanks so much!
------------------------------
Elizabeth Sanders
Research, Engagement, and Learning Librarian
Lamar University, Mary and John Gray Library
She/Her/Hers
------------------------------