Monitoring implementation and evaluating impacts
This stage aims to monitor the implementation of a given option and to evaluate the impact of a given option in a concrete setting. It is structured in two different goals that may need to be achieved (A and B). In total, 7 different types of questions that may need to be answered to achieve the goal are included in this stage.
Open the goal that is mostly related to your specific query, and a list of types of questions will be displayed. More details about each type of questions will be provided, including examples and the methodological approaches to address each question.
A. Identifying measurement strategies for populations and outcomes
How can we measure populations and results?
This goal aims to identify and select measurement strategies for ascertain the right population and accurately measure the outcomes of interest. This is related to finding instruments to facilitate measurement in order to monitor implementation and evaluate impact. In ascertaining populations and measuring outcomes, the questions are split in first identifying available instruments, followed by the selection of the most suitable instrument.
Question A1. Identifying instruments to identify or categorize populations
This type of question aims to find measurement instruments to identify the right population to monitor the implementation and/or evaluate the impact of an option or intervention.
Some examples of this type of question are:
- What diagnostic tests are available to detect COVID-19 infection?
- What instruments are available to identify the population living in poverty?
- What instruments are available to determine the presence of domestic gender violence?
Study designs to address question A1
- Review to find measurement strategies that have been used by other studies (e.g., scoping review)
- Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people’s opinions on measurement strategies
- Jurisdictional scan (comparative analysis) to understand what measurement strategies have been used by other jurisdictions
Methodological approaches (forms of evidence) to address question A1
 |
| Big picture reviews include scoping reviews, evidence maps, and evidence gap maps, that identify and map the breadth of evidence available on a particular issue. |
Question A2. Choosing the most accurate instruments to identify or categorize populations
Diagnostic accuracy
This type of question aims to evaluate the accuracy of the instruments to ascertain the right populations.
Some examples of this type of question are:
- What is the diagnostic accuracy of rapid antigen tests for detecting current COVID-19 infection?
- What are the most accurate ways to identify populations living under the line of poverty?
Study designs to address question A2
No consensus has yet been reached on what methodological approaches are most suitable to address this question. Learn more about our approach to reach consensus.
Methodological approaches (forms of evidence) to address question A2
|
| Reviews of diagnostic test accuracy aim to collate evidence about the accuracy (i.e., sensitivity and specificity) of a single test. |
Question A3. Identifying measurement instruments for outcomes of interest
This type of question aims to find instruments to measure outcomes, which can be used to understand a problem or measure the impact of an option, at one or multiple points in time.
Some examples of this type of question are:
- What scales can be used to measure depression?
- What instruments are available to measure clinical improvement in patients with COVID-19?
- What outcomes are there to measure health inequalities?
- How multidimensional poverty can be measured?
Study designs to address question A3
- Review to find measurement strategies that have been used by other studies (e.g., scoping review)
- Jurisdictional scan (comparative analysis) to understand what measurement strategies have been used by other jurisdictions
- Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people’s opinions on measurement strategies
Methodological approaches (forms of evidence) to address question A3
 |
| |
|
| Big picture reviews include scoping reviews, evidence maps, and evidence gap maps, that identify and map the breadth of evidence available on a particular issue. |
Question A4. Determining the best instruments to measure outcomes of interest
Reliability, validity
This type of question identifies how accurate a given instrument (e.g., measurement or scale) is to measure a given outcome of interest.
Some examples of this type of question are:
- How should emergency room waiting times be measured?
- Is the Beck index a valid and reliable score to measure depression severity?
- What are the best indicators to measure the impact of police violence and police lethality?
Methodological approaches to address question A4
No consensus has yet been reached on what methodological approaches are most suitable to address this question. Learn more about our approach to reach consensus.
Methodological approaches (forms of evidence) to address question A4
|
|
B. Monitoring and evaluating populations and outcomes of interests
Is it doing what is supposed to be doing?
This goal aims to monitor and evaluate populations and options or implementation strategies, including its sustainability over time. Here, we focus on scenarios where an option has been already implemented (as opposed to the questions included in stage 2).
Question B1. Monitoring the implementation of an option or implementation strategy
This type of question provides the results of the monitoring of an intervention
Some examples of this type of question are:
- How many targets have been achieved by the intervention?
Methodological approaches to address question B1
Not available yet. We will upload a consensus methodological approach very soon. Learn more about our approach to reach consensus.
Question B2. Evaluating the impact of an option or implementation strategy
This type of question aims to measure the impact of a given option or implementation strategy, which includes the assessment of potential unanticipated harms (or spillover effects) of an intervention, and it could be done in both the monitoring and the evaluation processes.
Some examples of this type of question are:
- What have been the impacts of implementing a national program promoting breastfeeding?
- What have been the impacts of a national program to make available laptops for every student in schools?
- What have been the impacts of a program implemented to use technology to facilitate referrals from primary care centres?
Study designs to address question B2
No consensus has yet been reached on what methodological approaches are most suitable to address this question. Learn more about our approach to reach consensus.
Methodological approaches (forms of evidence) to address question B2
 |
| Implementation sciences can provide a more summative assessment to learn about the impact of the implementation of specific interventions. |
|
| This type of question would benefit from either a theory-based or participatory approach, emphasizing or not the role of stakeholder engagement in all of the stages of an evaluation design. |