Clarifying a societal problem, its causes and potential impacts

This type of question aims to describe or quantify a problem or issue at a given point in time. Depending on the type of variable that would need to be measured accumulated frequencies, central tendencies, and/or distributions can be measured.

Some examples of this type of question are:

• What are the most common symptoms of people with active COVID-19 infection?
• What is the mean age of patients living with diabetes in a given country?
• What is the 90th percentile of BMI for a given population?

Study designs to address question A1

Three clusters of study designs (in order of suitability) can be used:

• First choice/higher rank: 
  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for this purpose (i.e., primary data).
  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for other purposes (i.e., secondary data).
• Second choice/middle rank:
  • Cross-sectional study on other jurisdictions
• Bottom rank:
  • Delphi study (to get consensus from experts)
  • Prospective longitudinal study of individual-level data (panel study)
  • Retrospective longitudinal study of individual-level data (panel study)

Methodological approaches to address question A1

Evidence syntheses Prevalence reviews

This type of question aims to describe a problem or issue during a period of time, through frequencies, central tendencies, and/or distributions.

Some examples of this type of question are:

• What is the number of new patients that entered a waiting list for elective surgery during 2022?
• What is the median age of people that entered a new social program during 2022?
• What is the number of people that have been diagnosed with HIV in the last year?

Study designs to address question A2

• Review of outcomes that have been used by other studies (e.g., scoping review)
• Retrospective cohort study of individual-level data (retrospective or historical longitudinal, or panel study)
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for this purpose (i.e., primary data)
• Delphi study (to get consensus from experts)

Methodological approaches (forms of evidence) to address question A2

Evidence syntheses Incidence reviews

Depending on the complexity of a given problem, conceptual frameworks might be useful to understand a problem, which might lead to better ways on how to address it. This type of question aims to find and select frameworks or taxonomies to describe and conceptualize a complex problem or issue.

Some examples of this type of question are:

• What do we understand by the concept of health authority and how this is connected to the Essential Public Health Functions?
• How can we classify the reasons behind vaccine hesitancy?
• How can we describe decision-making processes and their interaction with different contextual variables?
• What factors are contributing to inaction in global warming and climate change and how do they interact with each other?

Study designs to address question B1

• Review to identify existing frameworks (conceptual analysis)
• Qualitative inductive (from particular to general i.e., creating theory) methods to interpret/critically analyze a phenomenon (e.g., ethnographic approaches, phenomenology)
• Review to build a new framework (critical interpretive synthesis)
• Qualitative inductive (from particular to general i.e., creating theory) methods to describe a phenomenon (e.g., grounded theory)

Methodological approaches (forms of evidence) to address question B1

	Behavioural/implementation research Knowledge syntheses to conceptually understand a phenomenon	In the context that the specific problem or issue is connected with the implementation of a given intervention
	Evaluation Contribution analysis to test an existing theory of change.
	Evidence syntheses Expert opinion/policy review Qualitative evidence syntheses (e.g., framework synthesis, critical interpretive synthesis)

• What do cancer patients think about the disruption of cancer due to COVID-19?
• How do teachers perceive the educational system in a given country?
• What are citizens’ views regarding climate change?

Study designs to address question B2

• Qualitative inductive (from particular to general i.e., creating theory) methods to interpret/critically analyze a phenomenon (e.g., ethnographic approaches, phenomenology) 
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people's experiences (not asking about hypothetical scenarios)
• Qualitative inductive (from particular to general i.e., creating theory) methods to describe a phenomenon (e.g., grounded theory)
• Qualitative deductive (from general to particular i.e., testing theory) methods to describe/critically analyze a phenomenon (e.g., qualitative case studies)

Methodological approaches (forms of evidence) to address question B2

	Behavioural/implementation research
	Evaluation Outcome Mapping Outcome Harvesting Most significant change
	Evidence syntheses Qualitative evidence syntheses Reviews of preferences and values

Contexts and/or settings can bring unique challenges/opportunities for a certain problem or objective to be developed or achieved. This type of question aims to understand the role that the context or the specific setting might have in the problem and its magnitude.

Some examples of this type of question are:

• What is the role that the context has in the lack of affordable care for patients with cancer in a given country?
• What contextual factors are most important to reduce the higher inflation rates?
• How does the context of a student interact with his/her learning process?

Study designs to address question B3

Three clusters of study designs (in order of suitability) can be used:

• First choice/higher rank: 
  • Qualitative inductive (from particular to general i.e., creating theory) methods to describe a phenomenon (e.g., grounded theory)
  • Qualitative inductive (from particular to general i.e., creating theory) methods to interpret/critically analyze a phenomenon (e.g., ethnographic approaches, phenomenology)
• Second choice/middle rank:
  • Qualitative deductive (from general to particular i.e., testing theory) methods to describe a phenomenon (e.g., qualitative description, narrative approaches)
  • Qualitative deductive (from general to particular i.e., testing theory) methods to describe/critically analyze a phenomenon (e.g., qualitative case studies)
• Bottom rank:
  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people's experiences
  • Cross-sectional study on other jurisdictions
  • Social network analysis (mapping network analysis)

Methodological approaches (forms of evidence) to address question B3

	Behavioural/implementation research Knowledge syntheses to conceptually understand the role of context	In cases in which the context of a problem/issue created by the implementation of a given intervention matters.
	Evaluation Contribution analysis Most significant change Qualitative comparative analysis
	Evidence syntheses Expert opinion/policy review Qualitative evidence syntheses (e.g., framework synthesis, critical interpretive synthesis, realist syntheses) Complexity oriented quantitative systematic reviews

This type of question looks for different outcomes that are available and suitable to characterize or measure a given problem.

Some examples of this type of question are:

• What outcomes are available to measure clinical improvement in patients with COVID-19?
• What outcomes can be used to measure health inequalities?
• What outcomes or measurements are available to describe poverty in a country"

Study designs to address question C1

• Review of outcomes that have been used by other studies (e.g., scoping review)
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people’s experience
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of experts' opinion

Methodological approaches (forms of evidence) to address question C1

	Evaluation Outcome Mapping  Outcome Harvesting Stakeholders (expert) workshop to identify suitable outcomes
	Evidence syntheses Big picture reviews to identify potential outcomes.	Big picture reviews include scoping reviews, evidence maps, and evidence gap maps, that identify and map the breadth of evidence available on a particular issue.

People could have different values on the uses of different outcomes to characterize or measure a problem. Also, different stakeholders could have different values regarding outcomes (e.g., citizens may value different the outcomes than government policymakers). This type of question aims to describe the different values that different stakeholders could have regarding existing outcomes to measure a problem.

Some examples of this type of question are:

• How important is the outcome progression-free survival for patients with breast cancer?
• How do family members of patients with Alzheimer's disease value the quality of life?
• What are parents’ views on learning outcomes when choosing a school for their child's education?

Study designs to address question C2

Most suitable designs selected:

• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people’s perceptions about outcomes (not asking about hypothetical scenarios)
• Qualitative inductive (from particular to general i.e., creating theory) methods to interpret/critically analyze a phenomenon (e.g., ethnographic approaches, phenomenology).

Other designs that are still suitable but not the first choice:

• Qualitative deductive (from general to particular i.e., testing theory) methods to describe/critically analyze a phenomenon (e.g., qualitative case studies)
• Qualitative deductive (from general to particular i.e., testing theory) methods to describe a phenomenon (e.g., qualitative description, narrative approaches)
• Discrete choice experiment (stated preferences)

Methodological approaches (forms of evidence) to address question C2

	Evaluation Discrete choice experiments to distinguish revealed vs stated preferences	This type of question would benefit from a participatory approach, which emphasizes stakeholder engagement in all of the stages of an evaluation design.
	Evidence syntheses Qualitative evidence syntheses Reviews of preferences and values

This type of question aims to prioritize outcomes in their ability to characterize or measure a problem and considering the different values and preferences of individuals regarding them.

Some examples of this type of question are:

• What are the most suitable outcomes or scales to measure depression symptoms?
• What are the best ways to measure emergency room waiting times?
• What evaluation scale would be more reliable and accurate to measure students’ science learning?

Study designs to address question C3

• Delphi study (to get consensus from experts)
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for this purpose (i.e., primary data)
• Jurisdictional scan (comparative analysis) to understand what other jurisdictions are using
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for other purposes (i.e., secondary data)

Methodological approaches (forms of evidence) to address question C3

	Evaluation Outcome Mapping  Outcome Harvesting Stakeholders (expert) workshop to prioritize outcomes	This type of question would benefit from a participatory approach, which emphasizes stakeholder engagement in all of the stages of an evaluation design.
	Evidence syntheses Psychometric/measurement properties reviews

This type of question aims to measure the evolution of an indicator or measurement over time. Assessing the variability of a problem or issue when compared against time could be done over limited (e.g., 2 points) or multiple (e.g., time series) point of time.

Some examples of this type of question are:

• Has maternal mortality been reduced in the last 10 years?
• How the low ratio of nurses per 1000 habitants has evolved in the last 10 years?
• How has the average earth temperature varied in the last 3 decades?

Study designs to address question D1

• Longitudinal study of individual-level data (retrospective or historical longitudinal, or panel study)
• Descriptive (not predicting) time-series analysis (including trend analysis)
• Modelling to predict future scenarios (e.g., system dynamics, ARIMA models, etc.)
• Single before-and-after study of aggregated data (pre-post or pretest-posttest study)

Methodological approaches (forms of evidence) to address question D1

	Evaluation Qualitative comparative analysis
	Evidence syntheses Prevalence and/or incidence reviews

This type of question aims to identify what populations or sub-populations are affected (or most affected) by a certain problem, which could entail assessing equity considerations.

Some examples of this type of question are:

• What is the average waiting time in the emergency room vary depending on patients’ age?
• What is the difference in police violence episodes against white vs black communities?
• What is the income difference between citizens with high and low socio-economic status?
• How have wildfires affected different geographic communities?

Study designs to address question D2

• First choice/higher rank: 
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for this purpose (i.e., primary data
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for other purposes (i.e., secondary data)
• Second choice/Middle rank:
• Ecological study (population-based study, including spatial analysis)
• Prospective cohort study of individual-level data (prospective longitudinal or panel study)
• Bottom rank:
• Jurisdictional scan (comparative analysis) to understand variability across populations in other jurisdictions
• Retrospective cohort study of individual-level data (retrospective or historical longitudinal, or panel study)
• Qualitative deductive (from general to particular i.e., testing theory) methods to describe a phenomenon (e.g., qualitative description, narrative approaches)

Methodological approaches (forms of evidence) to address question D2

	Behavioural/implementation research Hybrid designs to identify specific target populations Knowledge syntheses to identify populations mostly affected	In cases in which the problem or issue is created by the implementation of a given intervention, a behavioural/implementation research approach could help to identify differences in population groups.
	Evidence syntheses Systematic reviews with a focus on healthy equity

This type of question aims to assess the variability or the magnitude of a problem when compared against other problems, by measuring the relative importance (i.e., weight) of an issue compared to others.

Some examples of this type of question are:

• What is the burden that hypertension creates compared to diabetes?
• For a given city, how important homelessness is compared to a poor educational system?"

Methodological approaches to address question D3

Three clusters of study designs (in order of suitability) can be used:

• First choice/higher rank: 
  • Modelling to estimate an indicator that allows comparisons in common units (e.g., DALYs)
  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of quantitative data (e.g., demographic information)
• Second choice/Middle rank:
  • Delphi study (to get consensus from experts)
  • Cross-sectional study (survey, point-in-time or snapshot study or analysis) of people's (affected by the problem) experiences (not asking about hypothetical scenarios)
• Bottom rank:
  • Discrete choice experiment (stated preferences)
  • Qualitative deductive (from general to particular i.e., testing theory) methods to describe a phenomenon (e.g., qualitative description, narrative approaches)

This type of question aims to find to what extent a certain variable can be associated with a certain problem and can be identified as a potential cause or aggravating factor. Identifying potential causes or aggravating factors of a problem require assessing whether a factor is associated with a certain outcome.

• Is having hypertension associated with a higher risk of myocardial infarction?
• Is being >60 years old associated with an increase in the risk of severe COVID-19?
• Is unemployment associated with worse health outcomes?

Study designs to address question E1

• Prospective cohort study of individual-level data (prospective longitudinal or panel study)
• Retrospective cohort study of individual-level data (retrospective or historical longitudinal, or panel study)
• Case-control study (case-comparison study)
• Interrupted time-series analysis (including joint-point regression)
• Regression discontinuity study (regression kink study or analysis)
• Ecological study (population-based study, including spatial analysis)
• Single before-and-after study of aggregated data (pre-post or pretest-posttest study)
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for this purpose (i.e., primary data)
• Instrumental variables study (two-stage least-squares study or regression)

Methodological approaches (forms of evidence) to address question E1

	Behavioural/implementation research Optimization designs to identify specific components or causes to enhance the implementation	In cases in which the problem or issue is created by the implementation of a given intervention, a behavioural/implementation research approach could help to identify potential causes or enabling factors in the implementation of a given intervention.
	Evaluation Contribution analysis to understand the role of potential components or causes on the problem
	Evidence syntheses Etiology and/or risk/exposure reviews Prognosis factor systematic reviews	Prognostic factors reviews are synthesis of studies that identify factors whose values are associated with changes in the outcome's risk or expected value.

This type of question aims to find to what extent a certain variable is a cause or aggravating factor of a certain problem. Different type of causes or aggravating factors might have variable effects on certain outcomes, and the level of exposure to a certain factor might also vary the effects on these outcomes.

Some examples of this type of question are:

• Does having hypertension increase the risk of myocardial infarction?
• Is the mental health status a potential cause explaining the use of psychoactive drugs among youth?
• Does being >60 years old increases the risk of severe COVID-19?
• Does unemployment produce worse health outcomes?

Methodological approaches to address question E2

• Prospective cohort study of individual-level data (prospective longitudinal or panel study)
• Retrospective cohort study of individual-level data (retrospective or historical longitudinal, or panel study)
• Case-control study (case-comparison study)
• Ecological study (population-based study, including spatial analysis) 
• Regression discontinuity study (regression kink study or analysis)
• Instrumental variables study (two-stage least-squares study or regression) 
• Interrupted time-series analysis (including joint-point regression)
• Single before-and-after study of aggregated data (pre-post or pretest-posttest study)

Methodological approaches (forms of evidence) to address question E2

	Behavioural/implementation research Optimization designs to identify specific components or causes to enhance the implementation	In cases in which the problem or issue is created by the implementation of a given intervention, a behavioural/implementation research approach could help to identify potential causes or enabling factors in the implementation of a given intervention.
	Evaluation Contribution analysis to understand the role of potential components or causes on the problem
	Evidence syntheses Etiology and/or risk/exposure reviews Prognosis factor systematic review	Prognostic factors reviews are synthesis of studies that identify factors whose values are associated with changes in the outcome's risk or expected value.

After having identified causes or aggravating factors of a problem, this type of question aims to assess the relative importance that causes and/or aggravating factors have based on their contribution to the problem, and to identify in what population groups the association between these causes and/or aggravating factors and the outcome is stronger.

Some examples of this type of question are:

• What are the most important risk factors that predict the incidence of myocardial infarction?
• What are the most important factors that explain a high infant mortality rate?
• What are the most important factors that explain the lack of access to prescription drugs?

Study designs to address question E3

• Prospective cohort study of individual-level data (prospective longitudinal or panel study)
• Case-control study (case-comparison study)
• Cross-sectional study (survey, point-in-time or snapshot study or analysis) of data collected for this purpose (i.e., primary data)
• Ecological study (population-based study, including spatial analysis)
• Retrospective cohort study of individual-level data (retrospective or historical longitudinal, or panel study)
• Regression discontinuity study (regression kink study or analysis)
• Delphi studies (to get consensus from experts)

Methodological approaches (forms of evidence) to address question E3

	Behavioural/implementation research Optimization designs to identify specific components or causes to enhance the implementation	In cases in which the problem or issue is created by the implementation of a given intervention, a behavioural/implementation research approach could help to identify potential causes or enabling factors in the implementation of a given intervention.
	Evaluation Most significant change to prioritize the most important causes of a problem
	Evidence syntheses Etiology and/or risk/exposure reviews using network meta-analysis or meta-regression techniques Prognosis factor systematic review using network meta-analysis or meta-regression techniques	Prognostic factors reviews are synthesis of studies that identify factors whose values are associated with changes in the outcome's risk or expected value.

This type of question aims to find whether and to what extent a given scenario is a consequence of a certain problem.

Some examples of this type of question are:

• What are the impacts of homelessness on the health system of a given country?
• What is the impact of violent police actions in citizens perceptions about the local government and democracy?
• What is the natural prognosis of a given disease?
• Can unemployment produce a higher cancer mortality rate?

Study designs to address question F1

• Controlled before-and-after study of aggregated data (including difference-in-differences study and non-equivalent control group designs)
• Review to find causes or aggravating factors that have been identified by other studies (e.g., scoping review)
• Prospective cohort study of individual-level data (prospective longitudinal or panel study)
• Retrospective cohort study of individual-level data (retrospective or historical longitudinal, or panel study)

Methodological approaches (forms of evidence) to address question F1

	Evaluation Contribution analysis to understand the role of the problem in producing other effects through theory of change. Quasi-experimental methods (such as controlled-before and after studies)
	Evidence syntheses Etiology and/or risk/exposure reviews Prognosis systematic reviews	Prognostic factors reviews are synthesis of studies that identify factors whose values are associated with changes in the outcome's risk or expected value.

This type of question aims to determine what are the most important impacts or spillover effects of a problem.

Some examples of this type of question are:

• What are the most important impacts of homelessness?
• What are the most important impacts of unemployment?
• What are the most important health impacts of undernutrition?"

Study designs to address question F2

• Review to find causes or aggravating factors that have been identified by other studies (e.g., scoping review)
• Delphi studies (to get consensus from experts)
• Prospective cohort study of individual-level data (prospective longitudinal or panel study)
• Retrospective cohort study of individual-level data (retrospective or historical longitudinal, or panel study)

Methodological approaches (forms of evidence) to address question F2

	Evaluation Most significant change to prioritize the most important spillover effects of a problem Stakeholder consultation (using Delphi) to prioritize the most important spillover effects.
	Evidence syntheses Etiology and/or risk/exposure reviews Prognosis systematic reviews Preferences and values reviews	Prognostic factors reviews are synthesis of studies that identify factors whose values are associated with changes in the outcome's risk or expected value.