The aim of this study was to explore the range and nature of influences on safety in decision-making by ambulance service staff (paramedics). A qualitative approach was adopted using a range of complementary methods. The study has provided insights on the types of decisions that staff engage in on a day-to-day basis. It has also identified a range of system risk factors influencing decisions about patient care. Although this was a relatively small-scale exploratory study, confidence in the generalisability of the headline findings is enhanced by the high level of consistency in the findings, obtained using multiple methods, and the notable consensus among participants.
The seven predominant system influences identified should not be considered discrete but as overlapping and complementary issues. They also embody a range of subthemes that represent topics for future research and/or intervention.
The apparently high level of consistency across the participating trusts suggests that the issues identified may be generic and relevant to other ambulance service trusts.
In view of the remit of this study, aspects relating to system weaknesses and potential threats to patient safety dominate in the account of findings. However, it should be noted that respondent accounts also provided examples of systems that were said to be working well, for example specific care management pathways, local roles and ways of working and technological initiatives such as IBIS and the ePRF.
Implications for health care
The NHS system within which the ambulance service operates is characterised in our study as fragmented and inconsistent. For ambulance service staff the extent of variation across the geographical areas in which they work is problematic in terms of knowing what services are available and being able to access them. The lack of standardisation in practice guidelines, pathways and protocols across services and between areas makes it particularly challenging for staff to keep up to date with requirements in different parts of their own trust locations and when crossing trust boundaries. Although a degree of consistency across the network is likely to improve the situation, it is also desirable to have sufficient flexibility to accommodate the needs of specific local populations. There was some concern over the potential for further fragmentation with the increased number of CCGs.
Ambulance services are increasingly under pressure to focus on reducing conveyance rates to A&E; this arguably intensifies the need to ensure that crews are appropriately skilled to be able to make effective decisions over the need to convey or not to convey if associated risks to patients are to be minimised. Our findings highlight the challenges of developing staff and ensuring that their skills are utilised where they are most needed within the context of organisational resource constraints and operational demands. Decisions over non-conveyance to A&E are moderated by the availability of alternative care pathways and providers. There were widespread claims of local variability in this respect. Staff training and development, and access to alternatives to A&E, were identified as priorities for attention by workshop attendees.
One of the difficulties for ambulance services is that they operate as a 24/7 service within a wider urgent and emergency care network that, beyond A&E, operates a more restricted working day. The study findings identify this as problematic for two reasons. First, it fuels demand for ambulance service care as a route to timely treatment, when alternatives may involve delay. Second, it contributes to inappropriate conveyance to A&E because more appropriate options are unavailable or limited during out-of-hours periods. Ultimately, this restricts the scope for ensuring that patients are getting the right level of care at the right time and place. Study participants identified some patient populations as particularly poorly served in terms of alternatives to A&E (e.g. those with mental health issues, those at the end of life, older patients and those with chronic conditions).
The effectiveness of the paramedic role in facilitating access to appropriate care pathways hinges on relationships with other care providers (e.g. primary care, acute care, mental health care, community health care). An important element relates to the cultural profile of paramedics in the NHS, specifically, the extent to which other health professionals and care providers consider the clinical judgements/decisions made by paramedics as credible and actionable. Staff identified this as a barrier to access where the ambulance service is still viewed primarily as a transport service. Consideration could be given to ways of improving effective teamworking and communication across service and professional boundaries.
Although paramedics acknowledged the difficulties of telephone triage, they also identified how the limitations of this system impact on them. Over-triage at the initial call-handling stage places considerable demands on both staff and vehicle resources. A related concern is the limited information conveyed to crews following triage. Initial triage was suggested as an area that warrants attention to improve resource allocation.
The findings highlight the challenges faced by front-line ambulance service staff. It was apparent that the extent and nature of the demand for ambulance conveyance represents a notable source of strain and tension for individuals and at an organisational level. For example, there were widespread claims that meeting operational demands for ambulance services limits the time available for training and professional development, with this potentially representing a risk for patients and for staff. Staff perceptions of risk relating to patient safety extend to issues of secondary risk management, that is, personal and institutional liabilities, in particular risks associated with loss of professional registration. The belief that they are more likely to be blamed than supported by their organisation in the event of an incident was cited by staff as a source of additional anxiety when making more complex decisions. This perceived vulnerability can provoke excessively risk-averse decisions. These issues merit further attention to examine the workforce implication of service delivery changes, including how to ensure that staff are appropriately equipped and supported to deal effectively with the demands of their role.
Paramedics identified a degree of progress in relation to the profile of patient safety within their organisations but the apparent desire within trusts to prioritise safety improvement was felt to be constrained by service demands and available resources. Attempts to prioritise patient safety appear to focus on ensuring that formal systems are in place (e.g. reporting and communication). Concerns were expressed over how well these systems function to support improvement, for example how incident reports are responded to and whether lessons learned are communicated to ambulance staff within and between trusts. Consideration could be given to identifying ways of supporting ambulance service trusts to develop the safety culture within their organisation.
Service users attributed the increased demand for ambulance services to difficulties in identifying and accessing alternatives. They were receptive to non-conveyance options but felt that lack of awareness of staff roles and skills may cause concern when patients expect conveyance to A&E.
Recommendations for research
The workshop attendees identified a range of areas for attention in relation to intervention and research, which are provided in Chapter 6 (see Suggestions for potential interventions and research). The following recommendations for research are based on the study findings:
Limited and variable access to services in the wider health and social care system is a significant barrier to reducing inappropriate conveyance to A&E. More research is needed to identify effective ways of improving the delivery of care across service boundaries, particularly for patients with limited options at present (e.g. those with mental health issues, those at the end of life and older patients). Research should address structural and attitudinal barriers and how these might be overcome.
Ambulance services are increasingly focused on reducing conveyance to A&E and they need to ensure that there is an appropriately skilled workforce to minimise the potential risk. The evidence points to at least two issues: (1) training and skills and (2) the cultural profile of paramedics in the NHS, that is, whether others view their decisions as credible. Research could explore the impact of enhanced skills on patient care and on staff, for example the impact of increased training in urgent rather than emergency care. This would also need to address potential cultural barriers to the effective use of new skills.
Research to explore the impact of different aspects of safety culture on ambulance service staff and the delivery of patient care (e.g. incident reporting, communication, teamworking, and training) could include comparisons across different staff groups and the identification of areas for improvement, as well as interventions that could potentially be tested.
The increased breadth of decision-making by ambulance service crews with advanced skills includes more diagnostics; therefore, there is a need to look at the diagnostic process and potential causes of error in this environment.
There is a need to explore whether there are efficient and safe ways of improving telephone triage decisions to reduce over-triage, particularly in relation to calls requiring an 8-minute response. This could include examining training and staffing levels, a higher level of clinician involvement or other forms of decision support.
There is a need to explore public awareness of, attitudes towards, beliefs about and expectations of the ambulance service and the wider urgent and emergency care network and the scope for behaviour change interventions, for example communication of information about access to and use of services; empowering the public through equipping them with the skills to directly access the services that best meet their needs; and informing the public about the self-management of chronic conditions.
A number of performance measures were identified engendering perverse motivations leading to suboptimal resource utilisation. An ongoing NIHR Programme Grant for Applied Research (RP-PG-0609–10195; ‘Pre-hospital Outcomes for Evidence-Based Evaluation’) aims to develop new ways of measuring ambulance service performance. It is important that evaluations of new performance metrics or other innovations (e.g. Make Ready ambulances, potential telehealth technologies or decision-support tools) address their potential impact on patient safety.
How well have we achieved our original aim and objectives?
The initially stated overarching aim of this research was to identify the contextual factors and mechanisms that are regularly associated with effective and cost-effective public involvement in research. While recognising the limitations of our analysis, we believe we have largely achieved this in our revised theory of public involvement in research set out in Chapter 8. We have developed and tested this theory of public involvement in research in eight diverse case studies; this has highlighted important contextual factors, in particular PI leadership, which had not previously been prominent in the literature. We have identified how this critical contextual factor shapes key mechanisms of public involvement, including the identification of a senior lead for involvement, resource allocation for involvement and facilitation of research partners. These mechanisms then lead to specific outcomes in improving the quality of research, notably recruitment strategies and materials and data collection tools and methods. We have identified a ‘virtuous circle’ of feedback to research partners on their contribution leading to their improved confidence and motivation, which facilitates their continued contribution. Following feedback from the HS&DR Board on our original application we did not seek to assess the cost-effectiveness of different mechanisms of public involvement but we did cost the different types of public involvement as discussed in Chapter 7. A key finding is that many research projects undercost public involvement.
In our original proposal we emphasised our desire to include case studies involving young people and families with children in the research process. We recruited two studies involving parents of young children aged under 5 years, and two projects involving ‘older’ young people in the 18- to 25-years age group. We recognise that in doing this we missed studies involving children and young people aged under 18 years; in principle we would have liked to have included studies involving such children and young people, but, given the resources at our disposal and the additional resource, ethical and governance issues this would have entailed, we regretfully concluded that this would not be feasible for our study. In terms of the four studies with parental and young persons’ involvement that we did include, we have not done a separate analysis of their data, but the themes emerging from those case studies were consistent with our other case studies and contributed to our overall analysis.
In terms of the initial objectives, we successfully recruited the sample of eight diverse case studies and collected and analysed data from them (objective 1). As intended, we identified the outcomes of involvement from multiple stakeholders‘ perspectives, although we did not get as many research partners‘ perspectives as we would have liked – see limitations below (objective 2). It was more difficult than expected to track the impact of public involvement from project inception through to completion (objective 3), as all of our projects turned out to have longer time scales than our own. Even to track involvement over a stage of a case study research project proved difficult, as the research usually did not fall into neatly staged time periods and one study had no involvement activity over the study period.
Nevertheless, we were able to track seven of the eight case studies prospectively and in real time over time periods of up to 9 months, giving us an unusual window on involvement processes that have previously mainly been observed retrospectively. We were successful in comparing the contextual factors, mechanisms and outcomes associated with public involvement from different stakeholders‘ perspectives and costing the different mechanisms for public involvement (objective 4). We only partly achieved our final objective of undertaking a consensus exercise among stakeholders to assess the merits of the realist evaluation approach and our approach to the measurement and valuation of economic costs of public involvement in research (objective 5). A final consensus event was held, where very useful discussion and amendment of our theory of public involvement took place, and the economic approach was discussed and helpfully critiqued by participants. However, as our earlier discussions developed more fully than expected, we decided to let them continue rather than interrupt them in order to run the final exercise to assess the merits of the realist evaluation approach. We did, however, test our analysis with all our case study participants by sending a draft of this final report for comment. We received a number of helpful comments and corrections but no disagreement with our overall analysis.
What were the limitations of our study?
Realist evaluation is a relatively new approach and we recognise that there were a number of limitations to our study. We sought to follow the approach recommended by Pawson, but we acknowledge that we were not always able to do so. In particular, our theory of public involvement in research evolved over time and initially was not as tightly framed in terms of a testable hypothesis as Pawson recommends. In his latest book Pawson strongly recommends that outcomes should be measured with quantitative data,17 but we did not do so; we were not aware of the existence of quantitative data or tools that would enable us to collect such data to answer our research questions. Even in terms of qualitative data, we did not capture as much information on outcomes as we initially envisaged. There were several reasons for this. The most important was that capturing outcomes in public involvement is easier the more operational the focus of involvement, and more difficult the more strategic the involvement. Thus, it was relatively easy to see the impact of a patient panel on the redesign of a recruitment leaflet but harder to capture the impact of research partners in a multidisciplinary team discussion of research design.
We also found it was sometimes more difficult to engage research partners as participants in our research than researchers or research managers. On reflection this is not surprising. Research partners are generally motivated to take part in research relevant to their lived experience of a health condition or situation, whereas our research was quite detached from their lived experience; in addition people had many constraints on their time, so getting involved in our research as well as their own was likely to be a burden too far for some. Researchers clearly also face significant time pressures but they had a more direct interest in our research, as they are obliged to engage with public involvement to satisfy research funders such as the NIHR. Moreover, researchers were being paid by their employers for their time during interviews with us, while research partners were not paid by us and usually not paid by their research teams. Whatever the reasons, we had less response from research partners than researchers or research managers, particularly for the third round of data collection; thus we have fewer data on outcomes from research partners‘ perspectives and we need to be aware of a possible selection bias towards more engaged research partners. Such a bias could have implications for our findings; for example payment might have been a more important motivating factor for less engaged advisory group members.
There were a number of practical difficulties we encountered. One challenge was when to recruit the case studies. We recruited four of our eight case studies prior to the full application, but this was more than 1 year before our project started and 15 months or more before data collection began. In this intervening period, we found that the time scales of some of the case studies were no longer ideal for our project and we faced the choice of whether to continue with them, although this timing was not ideal, or seek at a late moment to recruit alternative ones. One of our case studies ultimately undertook no involvement activity over the study period, so we obtained fewer data from it, and it contributed relatively little to our analysis. Similarly, one of the four case studies we recruited later experienced some delays itself in beginning and so we had a more limited period for data collection than initially envisaged. Research governance approvals took much longer than expected, particularly as we had to take three of our research partners, who were going to collect data within NHS projects, through the research passport process, which essentially truncated our data collection period from 1 year to 9 months. Even if we had had the full year initially envisaged for data collection, our conclusion with hindsight was that this was insufficiently long. To compare initial plans and intentions for involvement with the reality of what actually happened required a longer time period than a year for most of our case studies.
In the light of the importance we have placed on the commitment of PIs, there is an issue of potential selection bias in the recruitment of our sample. As our sampling strategy explicitly involved a networking approach to PIs of projects where we thought some significant public involvement was taking place, we were likely (as we did) to recruit enthusiasts and, at worst, those non-committed who were at least open to the potential value of public involvement. There were, unsurprisingly, no highly sceptical PIs in our sample. We have no data therefore on how public involvement may work in research where the PI is sceptical but may feel compelled to undertake involvement because of funder requirements or other factors.
What would we do differently next time?
If we were to design this study again, there are a number of changes we would make. Most importantly we would go for a longer time period to be able to capture involvement through the whole research process from initial design through to dissemination. We would seek to recruit far more potential case studies in principle, so that we had greater choice of which to proceed with once our study began in earnest. We would include case studies from the application stage to capture the important early involvement of research partners in the initial design period. It might be preferable to research a smaller number of case studies, allowing a more in-depth ethnographic approach. Although challenging, it would be very informative to seek to sample sceptical PIs. This might require a brief screening exercise of a larger group of PIs on their attitudes to and experience of public involvement.
The economic evaluation was challenging in a number of ways, particularly in seeking to obtain completed resource logs from case study research partners. Having a 2-week data collection period was also problematic in a field such as public involvement, where activity may be very episodic and infrequent. Thus, collecting economic data alongside other case study data in a more integrated way, and particularly with interviews and more ethnographic observation of case study activities, might be advantageous. The new budgeting tool developed by INVOLVE and the MHRN may provide a useful resource for future economic evaluations.23
We have learned much from the involvement of research partners in our research team and, although many aspects of our approach worked well, there are some things we would do differently in future. Even though we included substantial resources for research partner involvement in all aspects of our study, we underestimated how time-consuming such full involvement would be. We were perhaps overambitious in trying to ensure such full involvement with the number of research partners and the number and complexity of the case studies. We were also perhaps naive in expecting all the research partners to play the same role in the team; different research partners came with different experiences and skills, and, like most of our case studies, we might have been better to be less prescriptive and allow the roles to develop more organically within the project.
Implications for research practice and funding
If one of the objectives of R&D policy is to increase the extent and effectiveness of public involvement in research, then a key implication of this research is the importance of influencing PIs to value public involvement in research or to delegate to other senior colleagues in leading on involvement in their research. Training is unlikely to be the key mechanism here; senior researchers are much more likely to be influenced by peers or by their personal experience of the benefits of public involvement. Early career researchers may be shaped by training but again peer learning and culture may be more influential. For those researchers sceptical or agnostic about public involvement, the requirement of funders is a key factor that is likely to make them engage with the involvement agenda. Therefore, funders need to scrutinise the track record of research teams on public involvement to ascertain whether there is any evidence of commitment or leadership on involvement.
One of the findings of the economic analysis was that PIs have consistently underestimated the costs of public involvement in their grant applications. Clearly the field will benefit from the guidance and budgeting tool recently disseminated by MHRN and INVOLVE. It was also notable that there was a degree of variation in the real costs of public involvement and that effective involvement is not necessarily costly. Different models of involvement incur different costs and researchers need to be made aware of the costs and benefits of these different options.
One methodological lesson we learned was the impact that conducting this research had on some participants’ reflection on the impact of public involvement. Particularly for research staff, the questions we asked sometimes made them reflect upon what they were doing and change aspects of their approach to involvement. Thus, the more the NIHR and other funders can build reporting, audit and other forms of evaluation on the impact of public involvement directly into their processes with PIs, the more likely such questioning might stimulate similar reflection.
Recommendations for further research
There are a number of gaps in our knowledge around public involvement in research that follow from our findings, and would benefit from further research, including realist evaluation to extend and further test the theory we have developed here:
In-depth exploration of how PIs become committed to public involvement and how to influence agnostic or sceptical PIs would be very helpful. Further research might compare, for example, training with peer-influencing strategies in engendering PI commitment. Research could explore the leadership role of other research team members, including research partners, and how collective leadership might support effective public involvement.
More methodological work is needed on how to robustly capture the impact and outcomes of public involvement in research (building as well on the PiiAF work of Popay et al.51), including further economic analysis and exploration of impact when research partners are integral to research teams.
Research to develop approaches and carry out a full cost–benefit analysis of public involvement in research would be beneficial. Although methodologically challenging, it would be very useful to conduct some longer-term studies which sought to quantify the impact of public involvement on such key indicators as participant recruitment and retention in clinical trials.
It would also be helpful to capture qualitatively the experiences and perspectives of research partners who have had mixed or negative experiences, since they may be less likely than enthusiasts to volunteer to participate in studies of involvement in research such as ours. Similarly, further research might explore the (relatively rare) experiences of marginalised and seldom-heard groups involved in research.
Payment for public involvement in research remains a contested issue with strongly held positions for and against; it would be helpful to further explore the value research partners and researchers place on payment and its effectiveness for enhancing involvement in and impact on research.
A final relatively narrow but important question that we identified after data collection had finished is: what is the impact of the long periods of relative non-involvement following initial periods of more intense involvement for research partners in some types of research, particularly clinical trials?