Background: Stakeholder engagement efforts in the Evidence-based Practice Center (EPC) program initially focused on defining opportunities and developing materials to involve stakeholders in systematic reviews. Over time, the basic mechanics of working with stakeholders have become a routine part of the systematic review process, allowing the program to begin to explore how to improve stakeholder engagement and make it more effective
Purpose: The purpose of this research paper is to examine the benefits and challenges of engaging stakeholders in the process of developing and performing systematic reviews, drawing upon findings from the literature and Key Informant (KI) interviews with program leaders, systematic reviewers and stakeholders from within the EPC program and other international evidence-based programs.
Methods: We searched a range of databases, including Ovid MEDLINE, EBM Reviews, and Scopus. A gray literature search identified relevant guidance issued by internationally recognized organizations, and additional citations were identified both via research team members and by pearling of relevant article bibliographies. Abstracts and full text articles were reviewed for relevance by three investigators. In addition to the literature search we conducted a series of interviews with those who know the most about this process: 60 KIs were invited and one investigator facilitated all of the interviews. With permission from all participants we recorded and transcribed all calls; two investigators and two research assistants conducted analysis using NVivo to identify and synthesize recurring themes. In addition to identifying overarching themes, we conducted a more detailed exploratory analysis of the KI interviews with an eye towards articulating the benefits of engaging specific types of stakeholders at each phase of the systematic review.
Results: Of the 299 abstracts and 80 full-text articles reviewed, 24 addressed in some fashion the benefits, challenges, measurement or evaluation of stakeholder engagement. Benefits cited included identifying and prioritizing topics for research; providing pragmatic feedback on the research protocol; aiding in recruitment of research participants; helping the researchers understand the research subject's perspective; ensuring that findings are interpreted with the end user in mind and that final products are readable and accessible; and facilitating wider dissemination and uptake of research findings. There was almost no discussion of measurement or evaluation of the impact of stakeholder engagement. Of the 60 KIs we invited, 34 agreed to participate, we conducted 12 discussion sessions (60 to 90 minutes per session) with between one and four participants each. Indeed, it was not uncommon for stakeholders to represent more than one perspective. Overarching themes from our KI interviews were organized according to the three guiding questions of this white paper: (1) What are the potential or expected benefits of involving stakeholders in systematic reviews? (2) What are the challenges of involving stakeholders in systematic reviews? (3) How can we measure the impact of stakeholder engagement in systematic reviews?
Conclusions/Recommendations: Although it is recognized by many as an important next step, to date there have been few efforts to measure the benefits/tradeoffs of specific stakeholder engagement processes or differing approaches to selecting and engaging differing stakeholder types. In order to refine our processes for efficiently and effectively engaging stakeholders, we need to develop methods to evaluate the impact of stakeholder engagement based on a more concrete understanding of the specific benefits we are hoping to achieve. Toward this end, we reviewed the existing literature and conducted a series of KI interviews in an effort to more explicitly define the expected benefits of engaging stakeholders in systematic reviews.