Using and implementing individual-level outcome measures in palliative care settings: a reflective commentary

ANDY BRADSHAW, MARK PEARSON, FLISS MURTAGH

Wolfson Palliative Care Research Centre, Hull York Medical School, University of Hull, Hull, UK

Received August 19, 2021. Accepted August 31, 2021.

Summary. Individual-level outcome measures are an important aspect of palliative care. They are tools that can drive high-quality, person-centred care through providing healthcare professionals and services with valuable information about the patients we see, supporting us in making important decisions on patient care, and helping us to evaluate the effectiveness of clinical interventions. By reflecting on contemporary evidence from within palliative care, this commentary has three purposes. Firstly, we describe what individual-level outcome measures are and make the case for their importance within palliative care. Secondly, we highlight how we may get the most out of these outcome measures through using them in different ways. Finally, we reflect on the challenges to implementing outcome measures and advocate for the adoption of a ‘whole-systems’ approach that is complemented by implementation science when integrating them into practice. We provide practical advice and considerations on how this approach may be adopted. Accordingly, we hope that researchers working in this area, as well as those in clinical practice who are involved in using or implementing outcome measures across different settings of care, will reflect and critically engage with these suggestions in order to inform their implementation efforts and use of outcome measures.

Key words. Outcome measures, implementation science, palliative care.

Utilizzo e implementazione di misure di esito individuale nei contesti di cure palliative: una riflessione.

Le misure di outcome a livello individuale sono un aspetto importante delle cure palliative. Sono strumenti che possono favorire un’assistenza di alta qualità e incentrata sulla persona. Da una parte possono fornire agli operatori e ai servizi sanitari informazioni preziose sui pazienti che assistiamo, dall’altra possono supportare nelle decisioni importanti sulla cura dei pazienti, oltre ad aiutare nel valutare l’efficacia degli interventi clinici. Partendo dalle attuali evidenze derivanti dalle cure palliative, questa riflessione ha tre obiettivi. In primo luogo, descrivere quali sono le misure di esito a livello individuale e confermare la loro importanza all’interno delle cure palliative; in secondo luogo, indicare come ottenere il massimo da queste misure di esito utilizzandole in diversi modi; e infine, riflettere sulle sfide per l’attuazione di tali misure e sostenere l’adozione di un approccio di sistema che sia complementare all’esercizio della scienza quando integrato nella pratica. Si riportano di seguito consigli pratici e considerazioni su come adottare un simile approccio. L’augurio è che i ricercatori che lavorano in quest’area, così come quelli che si occupano di pratica clinica, e che sono coinvolti nell’uso o nell’implementazione di misure di esito nei diversi contesti di cura, riflettano e raccolgano criticamente questi suggerimenti al fine di implementare e utilizzare le misure di esito.

Parole chiave. Misure di esito, scienza dell’implementazione, cure palliative.

Introduction

By 2060, the global need for palliative care is projected to increase by 87%1. To meet the needs of this increased number of patients, it is essential that we are able to systematically identify, monitor, and address their symptoms and concerns. One way of achieving this is through using patient-level outcome measures. In this commentary, we demonstrate the value of outcome measures for palliative care and suggest how to implement them into routine clinical practice. As such, this article is split into three sections. The first section outlines what outcome measures are within palliative care and draws on available evidence in demonstrating the value of using them to inform patient care. The second section builds on the first by highlighting how we may collect outcome measures appropriately and then use them in different ways in order to get the most out of them. The final section outlines some common implementation challenges associated with integrating outcome measures into everyday clinical practice, and argues for the importance of adopting a whole-systems approach when attempting to implement them.

What are outcome measures?

Within healthcare, an outcome is defined as ‘the change in a patient’s current and future health status that can be attributed to preceding healthcare’2. In capturing change, therefore, outcomes require individual-level information about patients to be collected at two or more timepoints (figure 1).




To collect this information, measurement tools are needed. In ensuring that clinical decision-making is driven by a person-centred approach, it is advocated that these measurement tools are designed in ways that assesses the perspectives of patients directly3-5. In achieving this, Patient-Reported Outcome Measures (PROMs) are considered as the ‘gold standard’ for outcome measurement within palliative care6. PROMs require patients to fill out standardised and validated questionnaires which provide health-care professionals with information regarding their own perceptions of their health status and well-being7. It is often the case, however, that many patients with palliative care needs lack the capacity to complete outcome measures (due to impaired cognition or being too unwell)8, and so patient-centered outcome measures (PCOMs) are often used in clinical practice. Whilst these include PROMs, they also include proxy-reported ratings which, whilst still endeavor to focus on and evaluate concerns most important to patients, are completed by others (e.g., by healthcare professionals and/or a patient’s family member)7.

Within palliative care, most core outcome measures that are used can be split into two types: (i) functional status measures; and (ii) symptom measures. Functional status measures assess the physical performance/functional status of an individual, and often are centred on their ability to perform common activities of daily living. The most commonly used functional status measures in palliative care are the Australia-modified Karnofsky Performance Status (AKPS)9 and The Modified Barthel Score for Palliative care10-12. Symptom measures aim to assess the different illness-related physical, psychological, social, and spiritual symptoms that a person may experience. A multitude of symptom measures are used within palliative care, including the palliative Phase of Illness13, the Edmonton Symptom Assessment Scale (ESAS)14, the Memorial Symptom Assessment Scale (MSAS)15, the Palliative Care Problem Severity Score (PCPSS)16, the Symptom Assessment Scale (SAS)17-19, the European Organization for Research and Treatment of Cancer Quality of Life 15 items Questionnaire for Palliative Care (EORTC QLQ-C15-PAL)20, 21, and the Integrated Palliative care Outcome Scale (IPOS)22,23. Some of these measures focus exclusively on symptoms, and others focus on the wider range of symptoms and other concerns which affect those with advanced progressive illness. IPOS is a particularly good example of a measure which assesses the full range of concerns that a person may have (not just their symptoms). It also has both patient and proxy versions; it can be completed by patients themselves (via the self-reported version) or healthcare staff (via the proxy-reported version)23.

Why are outcome measures important?

To improve the standard of palliative care that is provided to patients and their families, measurement is important. This is so that we are able to understand whether or not current practice is working or getting better, or whether it is not24. In assessing the quality of healthcare, the Donabedian framework is the most widely adopted model. This framework consists of three components: structures, processes, and outcomes25,26. Assessing structures allows us to understand the effectiveness of resources, people, equipment, and buildings, whereas measuring processes assesses the effectiveness of how these structures/resources are used24. Most research within palliative care has focused on examining structures and processes of care6,24. Whilst these are necessary prerequisites for good palliative care, neither can guarantee, nor provide indicators of, good quality care. This is because they do not tell us anything about the patients we see (i.e., their needs or concerns) or whether the interventions that we use in clinical practice are effective at addressing these.

Thus, measuring outcomes is important because they provide healthcare professionals with valuable information about the patients we see, support us in making important decisions on their care, and help us to evaluate if the clinical interventions that we use are effective or not3,27. Indeed, there is strong evidence7,24,28-31 to support the use of PCOMs in routine palliative care. At a patient level, they act as tools that can support:

■ Improved communication between patients and clinicians.

■ The identification of unrecognised symptoms.

■ Monitoring of symptoms and concerns.

■ Increase the amount of clinical action taken based on data.

■ Improve patient satisfaction and experience.

■ Reduce reports of debilitating physical symptoms at subsequent visits.

Moreover, aggregating data from individual PCOMs also allows for benchmarking and auditing (i.e., setting standards to compare the quality of care to) so that we are able to highlight areas in which health services/organisations are doing well and identify areas for improvement and refinement32. An example of this can be seen from the Australia Palliative Care Outcome Collaboration, who have demonstrated the value of benchmarking and auditing to systematically improve clinical outcomes at a service level through routine outcomes data collection and feedback to hospice services33.

For these reasons, in working towards high quality palliative care, the European Association for Palliative Care recommended that PCOMs are implemented into routine clinical practice across all settings in which palliative care is delivered. 6 Moreover, they also advocate for the ‘establishment of National and international outcome collaborations that work towards benchmarking to establish and improve care standards.’ Despite these recommendations, PCOMs are not always used in clinical practice. One reason for this may be that healthcare professionals often report difficulties in understanding what they are and how they should best be used to benefit patient care3,34-36.

Getting the most out of outcome measures

Whilst there are numerous potential benefits of using outcome measures, their value is not derived through simply using them more often. Rather, in getting the most out of outcome measures, using them appropriately is crucial. This entails two major considerations: (i) collecting outcome measures correctly; and (ii) using the data we have collected effectively.

Collecting outcome measures correctly

Collecting outcome measures correctly involves understanding how to collect data using the right measures, at the right time, and in the right settings. Many of the outcome measures that have been developed and designed specifically for use in palliative care contain ‘rules’ with regards to how and when they should be collected across different settings of care. For example, in the UK and much of Europe, a core set of outcomes measures (palliative Phase of Illness, AKPS, and IPOS) are used in clinical practice. These measures inter-relate with one another, and the frequency and timing of their use depends on whether they are being used within inpatient hospice or community settings (table 1 for an example).




Collecting each outcome measure appropriately within the setting they are being used is important; failing to do so means that the data collected is less likely to be helpful in guiding clinical decisions or facilitating service development. Conversely, ensuring the appropriate methods of collection of outcomes information is achieved provides the foundation on which these measures may be used to inform clinical practice in meaningful ways.

Using outcomes information effectively

Getting the most out of outcome measures, however, requires much more than simply collecting them correctly. The real value of collecting this information is dependent on the different ways in which we use this data. A helpful way to understand the different types of value that we may achieve through using outcomes data is through the application of Greenhalgh’s framework38. Greenhalgh describes how we may use individual level patient data, or the aggregation of this into group data, either directly with or away from patients. Table 2 provides an adaptation of this framework that has been contextualised within palliative care. It provides four ‘quadrants’, each of which summarises the different ways in which outcome measures may be used.




One way in which we may apply outcome measures is through using and discussing individual level outcomes data directly with patients and their families (quadrant 1). This may be through using them as part of our initial assessments or as a conversation opener to create a person-centred dialogue about the things that matter to individual’s the most. We may also use outcome measures at this level to screen for a wide range of symptoms and concerns that cover multiple domains of well-being (physical, psychological, social, spiritual), alongside monitor whether the interventions that we use help to improve them over time.

Another application of outcome measures is through using individual level outcomes data but away from the patient interface (i.e., at a team and/or organisation level; quadrant 2). Within multidisciplinary teams, using measures in these ways can facilitate communication and more efficient working through providing a common language. This may efficiently focus discussions or help in prioritising time and resources on the symptoms and concerns that are most important to patients and their families. Moreover, at this level, outcomes data can also be used between teams and organisations as a way of passing on important information during referrals, handovers, and discharge.

The third way in which we may apply outcome measures is through using group level data with patients (third quadrant). This refers to applying a standard approach to the whole group of patients seen, usually to screen the group of patients for a specific issue or to trigger a specific action once a symptom or concern is identified. Examples might be to formally assess for depression (with a full clinical assessment of mood and mental state) any patients who reports ‘depressed mood’ above a certain level within a measure. Another example might be that all patients below a certain level of function (or with deterioration in function) might be automatically reviewed by a physiotherapist or an occupational therapist. Electronic scoring can also readily be used to embed decision aids for the professionals or to ‘trigger’ automatic alerts or referrals, although it is important to understand how such decision aids or alert systems might work and if they are effective, including their safety.

The final way in which we may apply outcome measures is through using group level data away from the patient interface (quadrant 4). Using data in these ways is particularly helpful in assessing quality of care. This is through using aggregated data to monitor who accesses services and assess the impact of these services through demonstrating whether or not they are effective at improving/managing patient symptoms and concerns. Moreover, using data in this way is helpful for developing business cases either for highlighting to funders that more resources and funding is needed either to improve and/or maintain already-existing services, or make funding cases for developing newer and better services.

Despite the potential value of using outcome measures in different ways, they are used inconsistently (if at all). Some palliative care organisations do not collect outcomes data, and many that do often only apply them in ways that algin with only one or two of these quadrants. Whilst this can support patient care, to maximise the value of using outcome measures, it is crucial that they are used across all of these quadrants. One reason for the inconsistent collection and use of outcome measures within palliative care is that they are often difficult to implement.

Whole-systems approach to implementation

The challenges of implementing outcome measures into practice

There is a growing body of work within palliative care that has explored the barriers/facilitators that underpin the implementation of outcome measures3,34-36,39,40. Common barriers that affect implementation include:

■ perceived time constraints

■ lack of training and education

■ tools being perceived as burdensome

■ negative attitudes towards outcome measures

■ fear of added work

■ top-down approach to implementation

■ lack of i.t. infrastructure within organisations

■ no feedback of outcomes data

■ availability of champions to drive change.

When viewed in silo, each of these issues represents individual, interpersonal, team, or organisational factors that impact implementation. Each of these factors interact in different, and often complex, ways that are important to understand and address when implementing outcome measures into routine practice. For example, our recent study exploring the processes underpinning the successful implementation of outcome measures in palliative care found that efficient I.T. systems that allowed staff to easily input, view, and extract outcomes data (so that they could be fed back to those that used them) were fundamental to successful implementation 34. When combined with strong leaders who championed their use, this allowed healthcare professionals to see the different values of using outcome measures to inform patient care, helped them to feel involved in implementation, and motivated them to continue learning about and using outcome measures as part of their everyday practice. Conversely, when these systems were not in place, and outcome measures were collected without feedback, many saw them as a ‘tickbox’ exercise. Understanding and addressing these challenges is important if PCOMs are to be implemented into practice and their benefits realised.

A whole-systems approach to implementation

Effective implementation requires an understanding of how to integrate PCOMs in a systematic, skilled, and consistent manner across the different settings in which palliative care is delivered. To do this involves consideration of the different ‘wrap-around’ factors that underpin implementation. These include thinking about how to demonstrate the importance and purposes of PCOMs to those who are using them, using the right measures at the right time, ensuring follow-up assessments, involving all teams/team members, having efficient feedback systems in place, and embedding PCOMs into the ‘cultural fabric’ of how teams and services operate. Given that these factors exist at multiple levels of practice, we argue that adopting a ‘whole-systems approach’ to implementation is essential in the planning and rollout phases of implementing outcome measures.

A whole systems approach to implementation appreciates the relationships between individual, interpersonal, team, and organisational factors that impact the scale-up and diffusion of complex interventions (such as PCOMs) into specific/local contexts. In adopting this ethos, those interested in implementation may wish to draw on the ideas of Hawe41, Lanham42, May43, McLeroy44, and Sallis45. A central feature that unifies these the theories and models proposed by these scholars is the appreciation that the implementation of PCOMs is affected by complex sociocultural processes, structures, and contexts that are likely to naturally evolve over time. However, rather than attempting to iron out and remove complexity within these contexts, it is important that complexity is accepted and embraced as an unavoidable feature of working within ‘real-world’ settings.

To complement the adoption of a whole-systems approach to implementation, we argue that there is merit in drawing on the principles and methods of implementation science. Implementation science is the systematic study of methods that are used to facilitate the integration of evidence-based practices/interventions (such as PCOMs) into routine clinical practice46, 47. This area of study comes equipped with a menu of different theories and frameworks that may be selected to provide a theoretical foundation on which to plan and perform the implementation of PCOMs. This is through drawing on specific theories and frameworks as a basis through which we can make evidence-based and theoretically informed assumptions on how, why, and in which contexts our implementation strategies/efforts are likely to work46.

There are a few examples from within the palliative care literature of where a whole-systems approach has been complemented by different implementation theories and frameworks. For example, in their systematic review, Antunes et al.3 drew on Promoting Action on Research Implementation in Health Services’ (PARIHS) framework to highlight the facilitators and barriers to implementing PCOMs that existed across individual, management/organisational, and setting specific levels. Moreover, Pinto et al 36 used the Consolidated Framework for Implementation Research (CFIR) to explore how the implementation of PCOMs across different palliative care settings were affected by individual-level factors (i.e., attitudes, beliefs, and knowledge) alongside the structural, political and cultural context within and outside of the organisations in which implementation occurred. Whilst the specific theories or frameworks that are adopted in research or clinical practice will depend on which is the best fit for answering our research questions, or which is likely to be most helpful in the context that implementation will occur, they ensure a robust and informed way of capturing and considering the multilevel factors that impact implementation.

Using existing evidence on the most important factors that underpin the implementation of PCOMs into palliative care, table 3 provides a summary (yet not exhaustive) set of questions that are designed to help those using PCOMs, or those who are involved in their rollout, to embrace complexity through reflecting on the multilevel factors that should be considered before and during attempts to implement PCOMs across different palliative care settings.




Summary

To summarise, outcome measures are an important part of evidence-based palliative care. The aim of this commentary article was to describe what outcome measures are, make the case for their importance within palliative care, demonstrate the different ways through which they can be used to better the quality of care that we provide to patients, and provide practical advice and considerations on how we may successfully implement them into routine practice*. We hope that researchers working in this area, as well as those in clinical practice who are involved in using or implementing outcome measures across different settings of care, will reflect and critically engage with this article in order to inform their implementation efforts and use of outcome measures.

Notes. For those interested, as part of the RESOLVE project, we have developed a set of training resources (including instructional videos, handbooks, and interactive quizzes) that have been designed to help healthcare professionals in their understandings of what outcome measures are, why they are important, and how they should be used across different settings of care. These may be found through following this link: https://www.hyms.ac.uk/research/research-centres-and-groups/wolfson/resolve/resolve-training-resources

Conflict of interests: the authors have no conflict of interests to declare.

Author contributions. AB drafted the manuscript and MP and FEM provided critical revision of the manuscript for important intellectual content.

Funding statement. Andy Bradshaw is funded by Yorkshire Cancer Research (Award reference number L412). Professor Fliss Murtagh is a National Institute for Health Research (NIHR) Senior Investigator. The views expressed in this article are those of the author(s) and not necessarily those of the NIHR, or the Department of Health and Social Care.

References

1. Sleeman KE, de Brito M, Etkind S, et al. The escalating global burden of serious health-related suffering: projections to 2060 by world regions, age groups, and health conditions. Lancet Global Health 2019; 7: e883-e892.

2. Donabedian A. Explorations in quality assessment and monitoring. Ann Arbor, MI: Health Administration Press, 1980.

3. Antunes B, Harding R, Higginson IJ, et al. Implementing patient-reported outcome measures in palliative care clinical practice: a systematic review of facilitators and barriers. Palliat Med 2014; 28: 158-75.

4. Dawson J, Doll H, Fitzpatrick R, et al. The routine use of patient reported outcome measures in healthcare settings. BMJ 2010; 340: c186.

5. Higginson IJ and Carr AJ. Measuring quality of life: using quality of life measures in the clinical setting. BMJ 2001; 322: 1297-1300.

6. Bausewein C, Daveson BA, Currow DC, et al. EAPC White Paper on outcome measurement in palliative care: Improving practice, attaining outcomes and delivering quality services–Recommendations from the European Association for Palliative Care (EAPC) Task Force on Outcome Measurement. Palliat Med 2016; 30: 6-22.

7. Etkind SN, Daveson BA, Kwok W, et al. Capture, transfer, and feedback of patient-centered outcomes data in palliative care populations: does it make a difference? A systematic review. J Pain Symptom Manage 2015; 49: 611-24.

8. Hill N. Use of quality-of-life scores in care planning in a hospice setting: a comparative study. Int J Palliat Nurs 2002; 8: 540-7.

9. Abernethy AP, Shelby-James T, Fazekas BS, et al. The Australia-modified Karnofsky Performance Status (AKPS) scale: a revised scale for contemporary palliative care clinical practice [ISRCTN81117481]. BMC Palliative Care 2005; 4: 7.

10. Gupta A. Measurement Scales Used in Elderly Care. Milton Keynes: Radcliffe Publishing, 2008.

11. Collin C, Wade DT, Davies S, et al. The Barthel ADL Index: a reliability study. Int Disabil Stud 1988; 10: 61-3.

12. Wade DT, Collin C. The Barthel ADL Index: a standard measure of physical disability? Int Disabil Stud 1988; 10: 64-7.

13. Masso M, Allingham SF, Banfield M, et al. Palliative care phase: inter-rater reliability and acceptability in a national study. Palliat Med 2015; 29: 22-30.

14. Bruera E, Kuehn N, Miller MJ, et al. The Edmonton Symptom Assessment System (ESAS): A Simple Method for the Assessment of Palliative Care Patients. J Palliat Care 1991; 7: 6-9.

15. Portenoy RK, Thaler HT, Kornblith AB, et al. The Memorial Symptom Assessment Scale: an instrument for the evaluation of symptom prevalence, characteristics and distress. Eur J Cancer 1994; 30a: 1326-36.

16. Masso M, Allingham SF, Johnson CE, et al. Palliative Care Problem Severity Score: reliability and acceptability in a national study. Palliat Med 2015; 30: 479-85.

17. Aoun SM, Monterosso L, Kristjanson LJ, et al. Measuring symptom distress in palliative care: psychometric properties of the Symptom Assessment Scale (SAS). J Palliat Med 2011; 14: 315-21.

18. Kristjanson L, Pickstock S, Yuen K, Blight J, Cummins A, Dean AA. Development and Testing of the Revised Symptom Assessment Scale: Final report. Perth, Australia: Edith Cowan University, 1999.

19. Daveson BA, Allingham SF, Clapham S, et al. The PCOC Symptom Assessment Scale (SAS): A valid measure for daily use at point of care and in palliative care programs. PLOS ONE 2021; 16: e0247250.

20. Groenvold M, Petersen MA, Aaronson NK, et al. The development of the EORTC QLQ-C15-PAL: a shortened questionnaire for cancer patients in palliative care. Eur J Cancer 2006; 42: 55-64.

21. Groenvold M, Petersen MA, Aaronson NK, et al. EORTC QLQ-C15-PAL: the new standard in the assessment of health-related quality of life in advanced cancer? Palliat Med 2006; 20: 59-61.

22. Hearn J, Higginson I. Development and validation of a core outcome measure for palliative care: the palliative care outcome scale. Palliative Care Core Audit Project Advisory Group. Qual Health Care 1999; 8: 219-27.

23. Murtagh FE, Ramsenthaler C, Firth A, et al. A brief, patient- and proxy-reported outcome measure in advanced illness: Validity, reliability and responsiveness of the Integrated Palliative care Outcome Scale (IPOS). Palliat Med 2019; 33: 1045-57.

24. Dudgeon D. The impact of measuring patient-reported outcome measures on quality of and access to palliative care. J Palliat Med 2018; 21: S-76-S-80.

25. Ayanian JZ and Markel H. Donabedian’s Lasting Framework for Health Care Quality. N Engl J Med 2016; 375: 205-7.

26. Porter ME, Larsson S and Lee TH. Standardizing patient outcomes measurement. N Engl J Med 2016; 374: 504-6.

27. De Vet HC, Terwee CB, Mokkink LB, et al. Measurement in medicine: a practical guide. Cambridge: Cambridge University Press, 2011.

28. Chen J, Ou L and Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC health services research 2013; 13: 211.

29. Luckett T, Butow PN and King MT. Improving patient outcomes through the routine use of patient-reported data in cancer clinics: future directions. Psychooncology 2009; 18: 1129-38.

30. Howell D, Molloy S, Wilkinson K, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol 2015; 26: 1846-58.

31. Boyce MB, Browne JP. Does providing feedback on patient-reported outcomes to healthcare professionals result in better outcomes for patients? A systematic review. Qual Life Res 2013; 22: 2265-78.

32. Antunes B, Rodrigues PP, Higginson IJ, et al. Outcome measurement: a scoping review of the literature and future developments in palliative care clinical practice. Ann Palliat Med 2018; 7: S196-S206.

33. Currow DC, Allingham S, Yates P, et al. Improving national hospice/palliative care service symptom outcomes systematically through point-of-care data collection, structured feedback and benchmarking. Support Care Cancer. 2015; 23: 307-15.

34. Bradshaw A, Santarelli M, Mulderrig M, et al. Implementing person-centred outcome measures in palliative care: An exploratory qualitative study using Normalisation Process Theory to understand processes and context. Palliat Med; 0: 0269216320972049.

35. Dunckley M, Aspinal F, Addington-Hall JM, et al. A research study to identify facilitators and barriers to outcome measure implementation. Int J Palliat Nurs 2005; 11: 218-25.

36. Pinto C, Bristowe K, Witt J, et al. Perspectives of patients, family caregivers and health professionals on the use of outcome measures in palliative care and lessons for implementation: a multi-method qualitative study. Ann Palliat Med 2018; 7: S137.

37. Witt J, Murtagh F, de Wolf-Linder S, et al. Introducing the Outcome Assessment and Complexity Collaborative (OACC) suite of measures-a brief introduction. London: Kings College London 2014.

38. Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual Life Res 2009; 18: 115-23.

39. Bausewein C, Simon ST, Benalia H, et al. Implementing patient reported outcome measures (PROMs) in palliative care-users’ cry for help. Health and quality of life outcomes 2011; 9: 27.

40. Bausewein C, Schildmann E, Rosenbruch J, et al. Starting from scratch: implementing outcome measurement in clinical practice. Ann Palliat Med 2018; 7: S253-S261.

41. Hawe P, Shiell A and Riley T. Theorising interventions as events in systems. Am J Community Psychol 2009; 43: 267-76.

42. Lanham HJ, Leykum LK, Taylor BS, et al. How complexity science can inform scale-up and spread in health care: understanding the role of self-organization in variation across local contexts. Soc Sci Med 2013; 93: 194-202. 2

43. May CR, Johnson M, Finch T. Implementation, context and complexity. Implementation Science 2016; 11: 141.

44. McLeroy KR, Bibeau D, Steckler A, et al. An ecological perspective on health promotion programs. Health Educ Q 1988; 15: 351-77.

45. Sallis JFO, N., Fisher, E. Ecological models of health behaviour. In: Glanz K, Rimer, B. K., Viswanath, K (eds). Health behaviour and health education: theory, research, and practice. San Franscisco, CA: John Wiley & Sons, 2008.

46. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci  2015; 10: 53.

47. Stover AM, Haverman L, van Oers HA, et al. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 2020; doi: 10.1007/s11136-020-02564-9.