Hello!
When I was previously at USDA National Agricultural Library, we adopted a version of AMSTAR2 for critically appraising existing reviews and determining which could be adopted for use in setting dietary reference values instead of commissioning new reviews through AHRQ (methods publication forthcoming).
From my experience guiding a team of nutritionists through application of the tool, we definitely found the search strategy evaluation piece lacking in AMSTAR2. As a checklist, it really emphasized dichotomous and quantifiable aspects (at least 2 databases? searched registries and for grey literature?). In terms of the actual search string/construction, AMSTAR2 asks whether keywords were provided in the manuscript or appendix, but it's not able to measure whether they were the right keywords, or a complete set of keywords, or even if controlled vocabulary were employed, or if there were any mistakes in the search syntax, etc. See question 4 in the
AMSTAR2 checklist linked here for more.
In addition, while many important supplemental search techniques are called out in the "Yes" (as opposed to "Partial Yes") section of question 4, it is exceedingly rare and not always appropriate that all of these are employed when that is what is required to attain a full "Yes." And, again, the focus is on whether the technique was used not whether it was done well or applied correctly - and this is often the important difference that we want to tease out. Finally, some items are also difficult to operationalize. We had a lot of trouble understanding how to interpret "included/consulted content experts in the field." Who qualifies as an expert? And what does included/consulted mean? Do authors count as people included? And reporting of this would almost always be lacking unless one contacted authors for further information.
AMSTAR2 is a great tool but, if I'm understanding correctly that your focus is on the search strategy's quality, it may be broader than you would find useful. I was unfamiliar with the two papers from Rethlefsen and Koffel and thanks to Amy for sharing those as they will make for interesting reading!
Hopefully you find that useful in identifying the best approach. This is a really important area to be developed within evidence synthesis, so thank you for your attention and future scholarship!
Best,
Eric
Eric Toole (he/him) | Evidence Synthesis Librarian
Science & Engineering Library
University of Massachusetts, Amherst
(413) 545-6151