Monitoring implementation and evaluating impacts

Problem Options Implementing M & E

Spanish version

This stage aims to monitor the implementation of a given option and to evaluate the impact of a given option in a concrete setting. It is structured in two different goals that may need to be achieved (A and B). In total, 7 different types of questions that may need to be answered to achieve the goal are included in this stage.

Open the goal that is mostly related to your specific query, and a list of types of questions will be displayed. More details about each type of questions will be provided, including examples and the methodological approaches to address each question.

A. Identifying measurement strategies for populations and outcomes

How can we measure populations and results?

M&E

This goal aims to identify and select measurement strategies for ascertain the right population and accurately measure the outcomes of interest. This is related to finding instruments to facilitate measurement in order to monitor implementation and evaluate impact. In ascertaining populations and measuring outcomes, the questions are split in first identifying available instruments, followed by the selection of the most suitable instrument.

Open the question that is mostly related to your specific query, and more details about the question, examples and methodological approaches will be displayed.

Question A1. Identifying instruments to identify or categorize populations

This type of question aims to find measurement instruments to identify the right population to monitor the implementation and/or evaluate the impact of an option or intervention.

Some examples of this type of question are:

  • What diagnostic tests are available to detect COVID-19 infection?
  • What instruments are available to identify the population living in poverty?

Methodological approaches to address question A1

  • Review to find measurement strategies that have been used by other studies (e.g., scoping review)
  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people’s opinions on measurement strategies
  • Jurisdictional scan (comparative analysis) to understand what measurement strategies have been used by other jurisdictions

Question A2. Choosing the most accurate instruments to identify or categorize populations

Diagnostic accuracy

This type of question aims to evaluate the accuracy of the instruments to ascertain the right populations.

Some examples of this type of question are:

  • What is the diagnostic accuracy of rapid antigen tests for detecting current COVID-19 infection?
  • What are the most accurate ways to identify populations living under the line of poverty?

Methodological approaches to address question A2

No consensus has yet been reached on what methodological approaches are most suitable to address this question. Learn more about our approach to reach consensus.

Question A3. Identifying measurement instruments for outcomes of interest

This type of question aims to find instruments to measure outcomes, which can be used to understand a problem or measure the impact of an option, at one or multiple points in time.

Some examples of this type of question are:

  • What scales can be used to measure depression?
  • What instruments are available to measure clinical improvement in patients with COVID-19?
  • What outcomes are there to measure health inequalities?

Methodological approaches to address question A3

  • Review to find measurement strategies that have been used by other studies (e.g., scoping review)
  • Jurisdictional scan (comparative analysis) to understand what measurement strategies have been used by other jurisdictions
  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people’s opinions on measurement strategies

Question A4. Determining the best instruments to measure outcomes of interest

Reliability, validity

This type of question identifies how accurate a given instrument (e.g., measurement or scale) is to measure a given outcome of interest.

Some examples of this type of question are:

  • How should emergency room waiting times be measured?
  • Is the Beck index a valid and reliable score to measure depression severity?

Methodological approaches to address question A4

No consensus has yet been reached on what methodological approaches are most suitable to address this question. Learn more about our approach to reach consensus.

B. Monitoring and evaluating populations and outcomes of interests

Is it doing what is supposed to be doing?

M&E

This goal aims to monitor and evaluate populations and options or implementation strategies, including its sustainability over time. Here, we focus on scenarios where an option has been already implemented (as opposed to the questions included in stage 2).

Open the question that is mostly related to your specific query, and more details about the question, examples and methodological approaches will be displayed.

Question B1. Monitoring the implementation of an option or implementation strategy

This type of question provides the results of the monitoring of an intervention

Some examples of this type of question are:

  • How many targets have been achieved by the intervention?

Methodological approaches to address question B1

Not available yet. We will upload a consensus methodological approach very soon. Learn more about our approach to reach consensus.

Question B2. Evaluating the impact of an option or implementation strategy

This type of question aims to measure the impact of a given option or implementation strategy, which includes the assessment of potential unanticipated harms (or spillover effects) of an intervention, and it could be done in both the monitoring and the evaluation processes.

Some examples of this type of question are:

  • What have been the impacts of implementing a national program promoting breastfeeding?
  • What have been the impacts of a national program to make available laptops for every student in schools?

Methodological approaches to address question B2

No consensus has yet been reached on what methodological approaches are most suitable to address this question. Learn more about our approach to reach consensus.

Question B3. Interpreting the findings of monitoring implementation or evaluating the impact of an option or implementation strategy

This type of question aims to ask why the option or implementation strategy is having the results that are showing and compare them with the expected impact. It also includes understanding the pathway (and intermediate outcomes) for a given option or implementation strategy to produce impact, and to draw lessons learned in this process.

Methodological approaches to address question B3

  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people's experiences (not asking about hypothetical scenarios)
  • Qualitative deductive (from general to particular i.e., testing theory) methods to describe a phenomenon (e.g., qualitative description, narrative approaches)
  • Qualitative deductive (from general to particular i.e., testing theory) methods to describe/critically analyze a phenomenon (e.g., qualitative case studies)

Monitoring implementation and evaluating impact can be done at the short, medium or long-term, and can also be conducted on problems (monitoring problems and evaluating their impacts) and on options or implementation strategies (monitoring option’s implementation and evaluating its impacts).

Take a look at the demand-driven approach used to create the Matching Q-M tool, as well as a list and explanation of all methodological approaches included.