Skip to main content
Download PDF
Download ePub

A review of implementation and evaluation frameworks for public health interventions to inform co-creation: a Health CASCADE study

Health Research Policy and Systems volume 22, Article number: 39 (2024) Cite this article

Abstract

Background

By including the needs and perspectives of relevant stakeholders, co-creation is seen as a promising approach for tackling complex public health problems. However, recommendations and guidance on how to plan and implement co-creation are lacking. By identifying and analysing existing implementation and evaluation frameworks for public health, this study aims to offer key recommendations for professional stakeholders and researchers wanting to adopt a co-creation approach to public health interventions.

Methods

Firstly, PubMed and CINAHL databases were screened for articles introducing original implementation and evaluation frameworks for public health interventions. Backwards snowballing techniques were applied to the included papers. Secondly, identified frameworks were classified and relevant data extracted, including steps and constructs present in the frameworks. Lastly, recommendations were derived by conducting thematic analysis on the included frameworks.

Results

Thirty frameworks were identified and data related to their nature and scope extracted. The frameworks’ prominent steps and constructs were also retrieved. Recommendations related to implementation and evaluation in the context of co-creation were included.

Conclusion

When engaging in co-creation, we recommend including implementation considerations from an early stage and suggest adopting a systems thinking as a way to explore multiple levels of influence, contextual settings and systems from an early planning stage. We highlight the importance of partnering with stakeholders and suggest applying an evaluation design that is iterative and cyclical, which pays particular attention to the experience of the engaged co-creators.

Peer Review reports

Background

Implementation science has been defined as the transfer of clinical research findings and evidence-based results into the real world and hence how a study can affect or hinder its uptake in the routine practice [1,2,3,4]. Thus, implementation science is set to observe and study the gap between, on one side, a solution developed in a controlled environment and, on the other, the specific context where the intervention is applied by looking at contextual factors that may act as barriers or facilitators.

However, interventions and solutions built in a controlled setting and transferred to specific context, have been argued to obtain limited success, mostly in the long term. For instance, the misconsideration for complex systems and factors related to settings and the targeted population have been said to influence the lack of effectiveness [5, 6]

Taking into account the relevance and inclusion of stakeholders’ knowledge in research production as been put forward as a possible way to address the research-practice gap [2, 3]. For this reason, more recently, implementation science has been experiencing a shifts from this type of linear and controlled production models to more iterative participatory and complex models [7,8,9] with the design and creation of solutions and interventions directly in the real world.

Involving relevant stakeholders from the earliest stage of intervention design and/or implementation has been considered a way to increase uptake and positively affect not only patient satisfaction but also the quality of the service [10,11,12,13,14,15]. In line with this considerations, co-creation has been brought to the forefront as an opportunity to increase the successful uptake of evidence-based interventions and practices through meanginful and deep engagement of key stakeholders [16,17,18,19].

Co-creation is a collaborative approach of creative problem solving engaging diverse stakeholders at all project stages, from determining and defining the problem through to the final stages of a project [20]. By facilitating collaboration among key stakeholders, co-creation aims to taking into account social determinants and contextual factors that may influence the intervention’s feasibility and acceptability from the earliest stage of intervention design.

Considering co-creation’s intention to work within real-life settings and conditions, systems thinking seems to be a valuable approach to explore and potentially adopt when designing and evaluating co-creation. Adopting a system thinking approach would allow assessing contextual elements from an early stage of the intervention and gathering considerations around systemic factors that may influence the public health issue [21].

The need for formative evaluation in co-creation has been argued to be crucial to co-creation processes. An evaluation is intended to be formative when the implementation team and/or staff use data to improve or adapt the process of implementation [1]. Van Dijk-de Vries [22] argues that researchers, when co-creating, should assess the stakeholders’ engagement to ensure their perceptions are captured, suggesting this happens throughout the implementation. Formative evaluation would enable, if needed, to adapt and adjust the intervention.

Despite research advancement in the field, implementation guidance and recommendations for the planning and implementation of co-creation processes are lacking as existing implementation and evaluation frameworks have not been designed specifically for such approaches. The need to develop dedicated implementation and evaluation guidelines for co-creation lies in the distinctive nature of co-creation approaches, involving collaborative efforts with diverse stakeholders, emphasizing shared decision-making, innovation and creativity.

Implementation and evaluation guidance needs to be further developed to address the dynamic and participatory nature of these processes and the unique challenges of fostering meaningful partnerships, navigating diverse perspectives and harnessing collective creativity. Closing this gap is essential not only for the successful implementation of co-creation initiatives but also for unlocking the full potential of these collaborative efforts within the broader landscape of public health interventions. This study aims to address this gap by reviewing existing frameworks and offering an overview of recommendations that may guide the design and implementation of co-created interventions.

Methods

Eligibility criteria

To be included, studies had to describe or introduce an implementation or evaluation framework, therefore describing or introducing a framework representing key stages, factors, constructs or variables that explain or influence the implementation and/or evaluation of programs/interventions. Frameworks had to be generalizable, and therefore designed to be applicable for all public health topics. For the scope of this review, an intervention was defined as a set of actions with a coherent objective to bring about change or produce identifiable outcomes [26]. We identified the Promoting Action on Research Implementation in Health Services (PARIHS), designed in 1998, as our start date search, as one of the first frameworks to make explicit the multi-dimensional, complex nature of implementation and highlight the central importance of context [27]. Table 1 includes further details on applied criteria.

Table 1 Eligibility criteria
Full size table

Search strategy

This scoping review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines for producing the PRISMA flow diagram (see Additional file 1). A specialist librarian was involved in developing the search strategy on PubMed and CINAHL. These were equal search strategies adapted for each database. The detailed search is available in Additional file 2. In addition, we applied a snowballing technique of pursuing backwards references cited in selected publications.

Process of selection

At least two authors were independently involved in all review studies through each stage [23]. D.A., K.M., J.Z., M.B., M.G. and G.L. were involved in the title and abstract screening. All the latter co-authors, with the addition of K.G., conducted the full-text screening. D.A., K.M., J.Z., M.B., M.G., G.L. and K.G. took part in the data extraction, and K.G. and G.L. conducted the data analysis. Authors conducted individual and independent reviews through the software Rayyan.Footnote 1 Discrepancies were resolved by consensus and, if unresolved, by the involvement of a third reviewer (M.G., G.L., G.C.).

Included frameworks are reported in Fig. 1. From each framework, we extracted prominent constructs (Additional file 3) and steps (Additional file 4). For extraction purposes, we defined constructs as a fundamental unit of thought, smaller than a judgment or theory but integral to them [24]. Identified steps were presented by categorizing them into the Leask’s et al. [25] framework included stages of planning, conducting, evaluating and reporting [25].

Fig. 1
figure 1

Classification of extracted frameworks

Full size image

Data extraction

An Excel template was developed to extract information relate to the framework’s nature, type and scope, positioning of implementation and evaluation considerations within the frameworks, and type of evaluation covered and other elements (see Table 2 for full list of data extracted). For each framework, if applicable, we extracted the main constructs (Additional file 3) and main phases (Additional file 4). The data extraction sheet was developed and piloted by two reviewers (G.L., M.G.) on two eligible papers and reviewed accordingly. D.A., K.M., J.Z., M.B., M.G., K.G. and G.L. extracted all data independently and blind-folded.

Table 2 Data extracted
Full size table

Data analysis

With the scope of aligning findings and come to a final set, extracted data was then cross-checked between G.L. and K.G. G.L. and K.G. sought consensus and, if in disagreement, they involved MG as a third reviewer. Final data was then plotted by G.L. in Fig. 1, which includes the framework’s classification data and steps and constructs reported in Additional files 2 and 3. K.G. and G.L. further conducted thematic analysis of included frameworks, as described below.

Thematic analysis

Through thematic analysis, we aimed to identify from included frameworks recommendations that would be relevant to the context of co-creation. This process followed the six stages as outlined by Braun and Clarke [58], including the following steps: (1) G.L. and K.G. familiarized with the data and wrote familiarization notes; (2) G.L. and K.G. developed a coding; (3) G.L. and K.G. independently generated initial themes from coded and collated data; (4) finally met to develop and reviewing themes; as well as to (5) refine, define and name themes; and (6) they applied the thematic framework to the remaining frameworks. [24]. The two researchers (G.L. and K.G.) independently coded four frameworks and then met to develop the thematic framework, which was then applied to the remaining frameworks. The themes and related recommendations reported in the results section emerged as the result of the data coding and iterative theme development [58].

Coding themes included the following: (a) early implementation considerations—exploring how the frameworks were including early implementation consideration; (b) system thinking—understand in what way frameworks were framed within a systems thinking paradigm [21, 29, 30]; (c) partnering with stakeholders—retrieving information on how the frameworks partnered with the stakeholders; (d) experience—how frameworks were pointing towards an assessment of the users’ experience throughout the process; or, finally, (e) iterative and cyclical evaluation—we explored how frameworks were accounting for this aspect.

Results

Summary of identified frameworks

From a total of 9061, after removing duplicates, 5284 papers were screened at title and abstract and 425 retrieved for full-text screening. We identified 30 articles and related implementation and evaluation frameworks (Table 3).

Table 3 List of frameworks
Full size table

As shown in Fig. 1, among frameworks, we identified 18 process models, 11 determinant frameworks and 16 evaluation frameworks. Most frameworks (16) are implementation frameworks, including evaluation elements, while 9 frameworks exclusively look at implementation or evaluation. There are 13 descriptive and 18 prescriptive frameworks. A total of 12 frameworks include implementation considerations from the stage of intervention development. Frameworks focus on various evaluation elements, with most frameworks (21) including effectiveness and 16 looking at impact evaluation, 15 at efficacy, 17 frameworks concern process evaluation and 11 related to an evaluation of elements related to the user experience. A total of 3 evaluation frameworks were concerned with evaluation of the planning phase, 8 were concerned with the conducting phase, 11 were concerned with post-execution and 12 were used throughout all stages.

Prominent constructs and steps were extracted from selected frameworks to give an overview of constructs (Additional file 3) and steps (Additional file 4) included in and steps that made part of the frameworks. Constructs that have been mentioned in frameworks include evidence, feasibility, acceptability, etc. and steps reported were moments of the process that the framework suggested carrying out.

Thematic analysis results

Below, we share findings derived from thematic analysis. Our intention is for these insights to represent recommendations that might be relevant for future research and the implementation of co-creation in practice. Figure 2 represents key themes and sub-themes identified through the thematic analysis of the identified frameworks.

Fig. 2
figure 2

Recommendations

Full size image

Early implementation considerations in extracted frameworks

A total of 12 frameworks include reference to early implementation considerations. Eslava-Schmalback et al. [37], recommend identifying critical factors for implementing equity focus recommendations and exploring barriers and facilitators of the intervention from the design phase. Kitson et al. [40] pay attention to preparing the intervention for context application, and Wimbush and Watson [44] call out the possibility of significant inconsistency between an intervention developed in experimental conditions and implementation in the real world.

Among frameworks, Cambon and Alla [33] focus on the context in which the intervention is implemented and argue that this should be viewed as an “interventional system”. In most frameworks, taking into account potential barriers to implementation takes the form of attention to the acceptability of its final users. It is claimed that by assessing its acceptance rate with users, the intervention might address potential barriers to its real-world application. In its description of the intervention cycle, Campbell et al. [31], for instance, advises adopting an iterative process through which the potential recipients’ acceptability of the intervention is assessed and re-examined if needed.

Similarly, Gonot-Schoupinsky and Garip [46] framework dedicates special attention to appropriateness and morality and how the user feels about the intervention as it may impact the intervention’s scalability potential. Assessing acceptability to both end-users and to stakeholders early in the process may be a crucial consideration for large-scale intervention implementation because of its potential to identify potential contextual barrier, enablers and motivations to participation in interventions [46].

Systems thinking in frameworks

Among the frameworks identified, seven explicitly reference a systems thinking perspective. Best et al. [43] advocate for a systems-grounded frame to be built with key stakeholders. Lo and Karn [41] view complex health programme interventions as systems composed of interdependent features and factors. The latter includes interdependent features and characteristics, such as human behaviours/perceptions, skills and capacity, and governmental and physical structures.

Similarly, Titler [51] recommends finding ways to account for systemic factors in the study design and randomized controlled designs. Zucca et al. [52] place intervention within a systems approach, making a distinction between a complex systems approach, in which variables are so intertwined that the cause and effect relation is uncertain, versus a complicated system, where numerous elements and relationships exist but their relationship can be unveiled and understood. Within the same perspective, Eslava-Schmalback et al. [37] stress the importance of understanding complex systems to advance and enhance implementation.

Partnerning with stakeholders

Co-creation is considered an approach that promotes engagement in partnership with stakeholders throughout the intervention. We reported on the level of engagement by identifying frameworks that had involved their stakeholders in a partnership, meant, according to Arnstein’s ladder, as the commitment to share planning and decision-making responsibilities through a set structure with its key stakeholders [53].

Six frameworks include elements of partnership with stakeholders. Racher and Annis [48] and Leask et al.’s [25] framework define partnership as the instance in which the stakeholders experience ownership while also (b) providing directional guidance and (c) being invested with responsibilities for activities and outcomes [48].

Partnerships are to be established according to frameworks, with different groups, including with (a) people involved in programme operations, (b) those served or affected by the programme, and (c) primary users of the evaluation [54] or, when community partnerning, between (a) multidisciplinary researchers, (b) the health researchers and community practitioners, and (c) and community health organizations at an international, national and local level [43].

Stakeholders’ views and experience are considered equal to other types of knowledge by Kitson et al. [40] Chen [38], which include patient preferences, views and experience as equally valuable and crucial to evaluate whether an intervention is practical, affordable, suitable, evaluable, and helpful in the real world.

Evaluating the experience of the co-creators

Benefits of joining the co-creation process might include cognitive, social and personal benefits [55]. To maintain and assert the value of co-creation to the co-creators involved, assessing their experience seems essential.

We reviewed the extent to which frameworks were evaluating the experience of the stakeholder’s involvement in the intervention. Gonot-Schoupinsky and Garip [46] include the assessment of acceptability to include reflections on appropriateness or morality, and how the user might have felt about the intervention, while Jolley et al. [50] suggest investigating barriers to participation and state the importance of ensuring the process is inclusive and values diversity.

Hennesy Lavery et al. [37] and Masso et al. [47], when evaluating the intervention, assess whether the level of agency of participants has increased and regard it as crucial to achieving the sustainability of the intervention. Marckmann et al. [35] stress the importance of evaluating the impact on autonomy while including the elements of health-related empowerment, such as health literacy, respect for individual autonomous choice and protection of privacy and confidentiality.

Iterative and cyclical evaluation

To comprehensively account for influential implementation elements, iterative evaluation at the planning and conducting phases allows researchers to address and prevent implementation obstacles by assessing the stakeholders’ perceptions and views and adjusting the intervention as needed.

Among frameworks, seven studies recognize the need to perform a more cyclical and iterative evaluation to allow for an intervention to be sustainable within its actual context and replicability to others [36, 44, 47, 50, 54]. Wimbush and Watson, for instance, suggest iterative evaluation as a way to review the intervention's feasibility, practicability, acceptability and for adjusting the programme’s initial design.

Discussion

This review identified 30 implementation and evaluation frameworks, classified according to their types and according to the categories specified in the data extraction. By analysis the frameworks through thematic analysis, it also offered insights into considerations for when implementing and evaluating future research and practice of co-creation.

Recommendations included accounting for early implementation considerations. Anticipating implementation questions has in fact been argued to be a way to increase the sustainability and maintenance of the intervention in the real-world setting [42] and considered by Moore et al. as crucial for future intervention development and evaluation [28]. Considering the adoption of a systems thinking approach was included as a key facet. Interventions, it is argued, need to be contextualized and understood in, rather than isolated from, the systems they operate within and co-creating interventions with its relevant stakeholders and intended target population, who hold deep knowledge of the systems they are situated within, can ensure a closer tie between theory and context.

By working with multiple levels of influence and with related contextual settings and systems, systems thinking [56]  seems to fit well the scope and intention of co-creation. With its intention to map the larger environment and to identify obstacles and challenges impacting and affecting the public health matter in question, systems thinking enables co-creation to address beyond the isolated causal effect but rather to explore and identify the multiplicity of real-world systematic factors that collate and contribute to the complex problem.

Evaluation is essential and crucial to co-creation and key is formative and cyclical evaluation, as suggested by Anneke van Dijk-de Vries et al. [22]. This, in fact, allows researchers to address and prevent implementation obstacles by assessing the stakeholders’ perceptions and views and adjusting the intervention as needed throughout.

By fostering reflection moments to ensure that end users’ perceptions are continuously captured [25], iterative process evaluation can represent a powerful tool in placing the voice and perception of the co-creators at the core of the intervention cycle. Doing so is particularly relevant when co-creating, as the process and how the co-creators are involved throughout, become part of a co-created intervention’s major outcomes and value in itself  [22].

Partnering with stakeholders and evaluating the co-creators experience is key as the co-created solution is expected to be developed jointly and provide benefits to the co-creators. Valuing the co-creators’ perceived level of co-ownership has been previously regarded essential to the co-creation process [25] and a way to ensure the co-created solution is developed through meaningful engagement.

This review identifies 30 implementation and evaluation frameworks for co-creation and offers recommendations for the planning and evaluating of co-creation for public health. Recommendations emphasize the importance of early implementation considerations, adopting a systems thinking approach, and prioritizing formative and cyclical evaluation. Iterative process evaluation is suggested as a powerful tool to centre co-creators’ voices in intervention cycles, posing and recognizing the value of the co-creation process itself, and not only on the implementation of the co-created solution. To underscore the significance of meaningful engagement and co-ownership when developing co-created solutions, the review highlights attention on the partnering with stakeholders and on an evaluation of the co-creators’ experiences.

This scoping review is conducted as part of the Health CASCADE study and findings will be used to inform the development of further guidance on planning and evaluating co-creation for public health. Authors will dive deeper into the framework by Leask et al. 2019 [25] identified through this review to identify strengths and weakness and expand on the available guidance, through a scoping review, and qualitative interviews with key stakeholders. Authors will also conduct a scoping review on process evaluation studies for co-creation and qualitative interviews with key stakeholders to develop an evaluation framework for co-creation.

Limitations of the study

Firstly, as we intended to explore the broader implementation field, we included several types of implementation frameworks within our definitions. This meant we captured several non-primary studies, presenting frameworks that had been developed for and/or applied to specific settings and contexts. These studies were later excluded at full-text screening. This might have caused the loss of frameworks that were specific to a context and setting but relevant to the scope of the study. We, however, as previously explained, performed snowballing on the identified frameworks to reduce this possible loss.

Secondly, while we set the search strategy with a specialist librarian, the review might have missed terms used for the same scope by other professionals (e.g. reporting guidelines, checklist or step-by-step how-to).

As part of our analysis, we scoped for implementation intervention development frameworks and did not include public health intervention development frameworks as we were interested in frameworks built to help guide the implementation of the intervention. Therefore this means the search lacks frameworks supporting the intervention development. We applied the search strategy to databases focussing on public health per the review’s scope. Therefore, this review might lack frameworks in databases from social sciences, although the snowballing exercise aimed to reduce the bias as it was performed with no limitations to the databases’ field. It is also worth acknowledging that the frameworks’ classification and data extraction were extracted independently by two reviewers and agreed upon by consensus to ensure the analysis was accurate. Nevertheless, interpretations made as part of the frameworks’ analysis were based on the reviewers’ subjective appraisal.

Conclusions

This review identified, classified and analysed 30 implementation and evaluation frameworks and offered recommendations for professional stakeholders and researchers wanting to adopt a co-creation approach.

The study recommends, when co-creating, to (a) include implementation considerations from an early stage and at the stage of intervention planning, (b) adopt a systems thinking approach when co-creating, and (c) form a partnership relationship with stakeholders to (d) plan for an iterative and cyclical evaluation and (e) focus on evaluating the co-creators’ experiences.

Contributions to literature

  • This scoping review identifies and classifies 30 implementation and evaluation frameworks for the development, implementation and evaluation of interventions in public health.

  • The analysis suggests positioning implementation considerations from an early start of the intervention and adopting a systems thinking approach to the implementation and evaluation of co-created interventions.

  • The authors highlight the importance of partnering with stakeholders and recommend carrying out an evaluation that is iterative and cyclical and focusses on the experience of the co-creators.

Availability of data and materials

Further data and materials are included in the additional files.

Notes

  1. https://www.rayyan.ai/

References

  1. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3:32.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract JPHMP. 2008;14:117–23.

    PubMed  Google Scholar 

  3. Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manage Rev. 1996;21:1055–80.

    Article  Google Scholar 

  4. Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006. https://0-doi-org.brum.beds.ac.uk/10.1186/1748-5908-1-1.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Trenchard-Mabere E. The emergence of systems thinking in behaviour change: a public health focus. In: Spotswood F, editor. Behav Change Key Issues Interdiscip Approaches Future Dir. Policy Press; 2016 https://0-doi-org.brum.beds.ac.uk/10.1332/policypress/9781447317555.003.0013

  6. Finegood DT, Johnston LM, Steinberg M, Matteson CL, Deck PB. Complexity, systems thinking, and health behavior change. Health Behav Change Popul. Baltimore, MD, US: Johns Hopkins University Press; 2014. p. 435–58.

  7. Verloigne M, Altenburg T, Cardon G, Chinapaw M, Dall P, Deforche B, et al. Making co-creation a trustworthy methodology for closing the implementation gap between knowledge and action in health promotion: the Health CASCADE project. Zenodo; 2022. https://zenodo.org/record/6817196

  8. Messiha K. D1.1—ESR1 Preliminary Synthesis. 2021; https://zenodo.org/record/6818098. Accessed 23 Sep 2022.

  9. Gibbons M. Mode 2 society and the emergence of context-sensitive science. Sci Public Policy. 2000;27:159–63.

    Article  Google Scholar 

  10. Martin LR, Williams SL, Haskard KB, DiMatteo MR. The challenge of patient adherence. Ther Clin Risk Manag. 2005;1:189–99.

    PubMed  PubMed Central  Google Scholar 

  11. Green LW, O’Neill M, Westphal M, Morisky D. The challenges of participatory action research for health promotion. Promot Educ. 1996;3:3–5.

    Article  CAS  PubMed  Google Scholar 

  12. Lyon AR, Munson SA, Renn BN, Atkins DC, Pullmann MD, Friedman E, et al. Use of Human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: protocol for studies applying a framework to assess usability. JMIR Res Protoc. 2019;8: e14990.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Dopp AR, Parisi KE, Munson SA, Lyon AR. Aligning implementation and user-centered design strategies to enhance the impact of health services: results from a concept mapping study. Implement Sci Commun. 2020;1:17.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Dopp AR, Parisi KE, Munson SA, Lyon AR. A glossary of user-centered design strategies for implementation experts. Transl Behav Med. 2019;9:1057–64.

    Article  PubMed  Google Scholar 

  15. Dopp AR, Parisi KE, Munson SA, Lyon AR. Integrating implementation and user-centred design strategies to enhance the impact of health services: protocol from a concept mapping study. Health Res Policy Syst. 2019;17:1.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Flaspohler PD, Meehan C, Maras MA, Keller KE. Ready, willing, and able: developing a support system to promote implementation of school-based prevention programs. Am J Community Psychol. 2012;50:428–44.

    Article  PubMed  Google Scholar 

  17. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38:44–53.

    Article  PubMed  Google Scholar 

  18. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Commun Psychol. 2008;41:171–81.

    Article  Google Scholar 

  19. Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94:392–429.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Vargas C, Whelan J, Brimblecombe J, Allender S. Co-creation, co-design, co-production for public health—a perspective on definition and distinctions. Public Health Res Pract. 2022;32:3222211.

    Article  PubMed  Google Scholar 

  21. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of medical research council guidance. BMJ. 2021;374: n2061.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Anneke van Dijk-de Vries, Anita Stevens, Trudy van der Weijden, Anna Beurskens. How to support a co-creative research approach in order to foster impact. The development of a Co-creation Impact Compass for healthcare researchers. PLOS ONE. 2020;

  23. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343: d5928.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Kolb WL. A dictionary of the social sciences. New York: Free Press of Glencoe; 1964.

    Google Scholar 

  25. Leask CF, Sandlund M, Skelton DA, Altenburg TM, Cardon G, Chinapaw MJM, et al. Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions. Res Involv Engagem. 2019;5:2.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Polese F, Mele C, Gummesson E. Value co-creation as a complex adaptive process. J Serv Theory Pract. 2017;27:926–9.

    Article  Google Scholar 

  27. Smith N, Georgiou M, Jalali MS, Chastin S. Planning, implementing and governing systems-based co-creation: the DISCOVER framework. Health Res Policy Syst. 2024;22:1–16.

    Article  Google Scholar 

  28. Bednarczyk RA, Chamberlain A, Mathewson K, Salmon DA, Omer SB. Practice-, provider-, and patient-level interventions to improve preventive care: development of the P3 Model. Prev Med Rep. 2018;11:131–8.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Best A, Stokols D, Green LW, Leischow S, Holmes B, Buchholz K. An integrative framework for community partnering to translate theory into effective health promotion strategy. Am J Health Promot AJHP. 2003;18:168–76.

    Article  PubMed  Google Scholar 

  30. Cambon L, Alla F. Understanding the complexity of population health interventions: assessing intervention system theory (ISyT). Health Res Policy Syst. 2021;19:95.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321:694–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2:40.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Chen HT. The bottom-up approach to integrative validity: a new perspective for program evaluation. Eval Program Plann. 2010;33:205–14.

    Article  PubMed  Google Scholar 

  34. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Green LW, Kreuter MW. Health promotion planning: an educational and environmental approach. Mountain View, CA: Mayfield Pub. Co.; 1991.

    Google Scholar 

  36. Gurewich D, Garg A, Kressin NR. Addressing social determinants of health within healthcare delivery systems: a framework to ground and inform health outcomes. J Gen Intern Med. 2020;35:1571–5.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Hennessey Lavery S, Smith ML, Esparza AA, Hrushow A, Moore M, Reed DF. The community action model: a community-driven model designed to address disparities in health. Am J Public Health. 2005;95:611–6.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Hyner GC. A procedural model for planning and evaluating behavioral interventions. Methods Inf Med. 2005;44:299–302.

    Article  CAS  PubMed  Google Scholar 

  39. Jolley G, Lawless A, Hurley C. Framework and tools for planning and evaluating community participation, collaborative partnerships and equity in health promotion. Health Promot J Aust Off J Aust Assoc Health Promot Prof. 2008;19:152–7.

    Article  Google Scholar 

  40. Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care QHC. 1998;7:149–58.

    Article  CAS  PubMed  Google Scholar 

  41. Lo K, Karnon J. in-DEPtH framework: evidence- in formed, co-creation framework for the D esign, E valuation and P rocuremen t of H ealth services. BMJ Open. 2019;9: e026482.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Marckmann G, Schmidt H, Sofaer N, Strech D. Putting public health ethics into practice: a systematic framework. Front Public Health. 2015;3:23.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Masso M, Quinsey K, Fildes D. Evolution of a multilevel framework for health program evaluation. Aust Health Rev Publ Aust Hosp Assoc. 2017;41:239–45.

    Article  Google Scholar 

  44. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42.

    Article  PubMed  PubMed Central  Google Scholar 

  45. MMWR. Framework for program evaluation in public health. MMWR Recomm Rep Morb Mortal Wkly Rep Recomm Rep. 1999; 48: 1–40.

  46. Nguyen DTK, McLaren L, Oelke ND, McIntyre L. Developing a framework to inform scale-up success for population health interventions: a critical interpretive synthesis of the literature. Glob Health Res Policy. 2020;5:18.

    Article  PubMed  PubMed Central  Google Scholar 

  47. O’Connor-Fleming ML, Parker E, Higgins H, Gould T. A framework for evaluating health promotion programs. Health Promot J Aust Off J Aust Assoc Health Promot Prof. 2006;17:61–6.

    Article  Google Scholar 

  48. Racher FE, Annis RC. Community health action model: health promotion by the community. Res Theory Nurs Pract. 2008;22:182–91.

    Article  PubMed  Google Scholar 

  49. Titler MG. Translation science and context. Res Theory Nurs Pract. 2010;24:35–55.

    Article  PubMed  Google Scholar 

  50. Wilson KM, Brady TJ, Lesesne C, NCCDPHP Work Group on Translation. An organizing framework for translation in public health: the knowledge to action framework. Prev Chronic Dis. 2011;8:A46.

    PubMed  PubMed Central  Google Scholar 

  51. Zucca C, Long E, Hilton J, McCann M. Appraising the implementation of complexity approaches within the public health sector in Scotland an assessment framework for pre-implementation policy evaluation. Front Public Health. 2021. https://0-doi-org.brum.beds.ac.uk/10.3389/fpubh.2021.653588.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Wimbush E, Watson J. An evaluation framework for health promotion: theory. Quality Effectiveness Evaluat. 2000;6:301–21.

    Google Scholar 

  53. Arnstein SR. A ladder of citizen participation. J Am Inst Plann. 1969;35:216–24.

    Article  Google Scholar 

  54. Verleye K. The co-creation experience from the customer perspective: its measurement and determinants. Dr Elina Jaakkola AH and DLA-S, editor. J Serv Manag. 2015;26:321–42.

    Article  Google Scholar 

  55. Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health. 2017;3:132–5.

    Article  PubMed  Google Scholar 

  56. Planning, implementing and governing systems-based co-creation: the DISCOVER framework

  57. Amit G, Singal Peter DR, Higgins Akbar K, Waljee,. A primer on effectiveness and efficacy trials. Clin Transl Gastroenterol. 2014;5(1):e45. https://0-doi-org.brum.beds.ac.uk/10.1038/ctg.2013.13

    Article  CAS  Google Scholar 

  58. Braun V, Clarke V. Usingthematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. https://0-doi-org.brum.beds.ac.uk/10.1191/1478088706qp063oa.

    Article  Google Scholar 

  59. Moullin JC, Dickson KS, Stadnick NA, Albers B, Nilsen P, Broder-Fingert S, Mukasa B, Aarons GA. Ten recommendations for using implementation frameworks in research and practice. Implement Sci Commun. 2020;1:42. https://0-doi-org.brum.beds.ac.uk/10.1186/s43058-020-00023-7.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Evaluation: A Systematic Approach

  61. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Quarterly. 2004;82(4):581–629. https://0-doi-org.brum.beds.ac.uk/10.1111/j.0887-378X.2004.00325.x.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Special acknowledgement to Dr Ivan Sola, who has supported the search strategy development for this study.

Funding

This study has been funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement no. 956501. The views expressed in this paper are the author’s views and do not necessarily reflect those of the funders.

Author information

Authors and Affiliations

  1. Faculty of Psychology, Education and Sport Sciences, Universitat Ramon Llul, Blanquerna, Barcelona, Spain

    Giuliana Raffaella Longworth

  2. Department of Physical Activity and Sport Sciences, FPCEE Blanquerna, Universitat Ramon Llull, Carrer del Císter, 34, 08022, Barcelona, Spain

    Jorge Raul Zapata-Restrepo & Maria Giné-Garriga

  3. Department of Public and Occupational Health, Amsterdam Public Health Research Institute, Amsterdam UMC, Vrije Universiteit Amsterdam, Meibergdreef 9, 1105 AZ, Amsterdam, The Netherlands

    Kunshan Goh & Katrina Messiha

  4. School of Health and Life Sciences, Glasgow Caledonian University, Cowcaddens Road Glasgow, Scotland, G4 0BA, UK

    Danielle Marie Agnello & Sebastien Chastin

  5. Department of Movement and Sports Sciences, Physical Activity and Health, Ghent University and Policy Research Center Sport, Krijgslaan 281 - S2, 9000, Ghent, Belgium

    Greet Cardon & Sebastien Chastin

  6. School of Psychology, Artevelde Hogeschool, Ghent, Belgium

    Melanie Beeckman

Authors
  1. Giuliana Raffaella Longworth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kunshan Goh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Danielle Marie Agnello
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Katrina Messiha
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Melanie Beeckman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jorge Raul Zapata-Restrepo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Greet Cardon
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sebastien Chastin
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Maria Giné-Garriga
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

G.L. and M.G. developed the protocol and search strategy with the support of the specialist librarian and input from G.C. D.A., K.M., J.Z., M.B., M.G. and G.L. were involved in the title and abstract screening. They all performed, with the addition of KG, the full-text screening. D.A., K.M., J.Z., M.B., M.G., G.L. and K.G. took part in the data extraction, and K.G. and G.L. conducted the data analysis. K.G. contributed to the framework appraisal and conceptualization while G.L. drafted the manuscript. M.G., S.C. and G.C. contributed to the manuscript’s first draft. The manuscript was then reviewed and approved by all authors.

Corresponding author

Correspondence to Giuliana Raffaella Longworth.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing of interests

The authors have no competing interests to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

PRISMA.

Additional file 2.

Search strategy.

Additional file 3.

A summary of the constructs found in the frameworks.

Additional file 4.

A summary of steps found in frameworks.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Longworth, G.R., Goh, K., Agnello, D.M. et al. A review of implementation and evaluation frameworks for public health interventions to inform co-creation: a Health CASCADE study. Health Res Policy Sys 22, 39 (2024). https://0-doi-org.brum.beds.ac.uk/10.1186/s12961-024-01126-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12961-024-01126-6

Keywords

Health Research Policy and Systems

ISSN: 1478-4505

Contact us