Transparent measurement |
This is the place to talk about transparent measurement. |
|
Re: Transparent measurement |
Interesting that this theme has not elicited much response. Here is a starter for 10. (or 6 in fact) Some blocks to transparent measurement 1 Transparent measurement is costly and difficult. |
|
Re: Transparent measurement |
Hi Nick, Thank you for your post – it’s really thought-provoking. The evidence tells us that up to 75% of change efforts fail to meet their objectives because they run out of energy. I’m not surprised when I consider the list of difficulties you describe. Each of the difficulties must make teams ask themselves - “Is it really worth it?”. Having said that, I’m sure that many teams and individuals continue to soldier on, because they have a strong sense of “why” they are measuring. I think the NHS Change Model draws our attention to the importance of putting upfront effort into creating the right conditions for change. Establishing Our Shared Purpose is about creating and sustaining a strong sense of “why”. But its also something that needs reviewing over time, so its not a one-off investment. Sirkin et al (2005) point out that successful change requires management-led formal review processes as well as influential leaders who show visible support for the change throughout its duration. This has some implications for 5 of your 6 "blocks": 1. Since Transparent Measurement is costly and difficult, we’d better be sure we are getting a Return on our Investment by being clear on why we are doing it. 2. Could we start with “why” we are measuring? Keep the “what” simple for the majority, and allow the experts to get into the complexities of determining the benefits management case. 3. Establishing cause and effect usually requires a balance of qualitative and quantitative investigation. The compelling message for change often comes from the qualitative data (or stories), so should we be making more use of that to keep ourselves grounded amid the noise? 4. We have access to HES and the business objects function allows us the flexibility to create our own tools – one tool will not be sustainable to meet changing needs anyway – raw data is best, the expertise to use it is what we need to invest in, perhaps? 5. If our data gives us the “wrong” answer, we need to reconsider “why” we are measuring – do the results change “why” we are measuring? Or do they just mean we need to change “what” we are measuring? In what ways, then, do you (and others) think we could we align Transparent Measurement with the other components of the NHS Change Model to make “numbers and measurement more sexy in the NHS”? Do you have any views on the practical steps that can be taken to start Transparent Measurement? Sirkin H, Keenan P, Jackson A, “The Hard Side of Change Management” Harvard Business Review, October 2005
|
|
Re: Transparent measurement |
Interesting! Let me respond. 2 Why are we measuring? We are measuring for one purpose alone. To determine whether or not what we are doing is working. We can then decide whether to do more, do the same or stop. I agree that determining the precise metrics for measuring whether objectives are being achieved can be left to the backroom. However that is only part of benefits management. Having clarity about what changes are expected to be achieved, what the value of those benefits is and to whom, how those changes and benefits will manifest themselves and ensuring that collateral damage is minimised are surely core to a change leaders skill set. 3 The need to use both qualitative as well as quantitative information is a valid one. However qualitative research is a rigourous process (as my qualitative researcher colleague reminds me on a regular basis) this is different from using patient stories as a powerful pursuasive tool to illustrate the changes that are intended and to garner support. The reason I say this is that there is a natural tendency to find stories that support the intuitive view. As the paper refered to above about Randomised Controlled Trials makes clear, the intuitively obvious has a nasty habit of turning out not to be true. Stories are excellent for knowing and sharing what you want to achieve. They are less useful in working out whether you have achieved it 4 HES data is only a limited and obfuscated sub-set of the data supplied by Trusts and other to the centre. Dr Foster, CHKS amd other can licence to use of all of that data to create "value added" products which they can then sell on. Local NHS organisations only have access to their own information unless they buy into these products. That is bizarre 5 By the "wrong answer" I meant the answer we don't want not an uninformative answer. We need to be much more open to trying things and then stopping if the don't work. If we put in an intiative to reduce emergency admissions and they go up, we need to stop. We don't blame anybody because it didn't work, we don't bin all other initiatives because they don't work but, if that data is giving us bad news, we live with it and start again. and you haven't answered 6 you have just thrown it back at me! Who could become the Dr Brian Cox of Health Informatics? |
|
Re: Transparent measurement |
Why are we measuring? Solberg et al (1997) point out that its not just to understand how whether what we are doing is working, but its also to motivate people involved in the change process. If numbers and measurement just aren’t sexy in the NHS, however, we have a big job to do on getting better at presenting data well and engaging people. I believe that leaders can create a compelling message with data – and improvements demonstrated through the data can be used to create a great deal of motivation to support the change. There are bound to be things we can learn from the experts all the way over there in the “Engage to Mobilise” component. Shall we pop over to their knowledge share and ask the question? Could their expertise even help us overcome some of the blocks as well? What about the fear of getting the “wrong answer” – its an important point Nick raises here because Transparent Measurement means we should be sharing what we measure, not hiding it for fear of “blame” and the possibility that “it may make it impossible to focus on systematic improvement of the process” (Solberg et al 1997). Is it all in the way we frame our results for different audiences, perhaps? Solberg, Mosser & McDonald (1997) The Three Faces of Performance Measurement: Improvement, Accountability and Research Journal on Quality Improvement, 23, 3. |
|
Re: Transparent measurement |
Hello Rosanna, All, I've come to your knowledge share from Engagement to Mobile ... Re Nick's #6: Storytelling is well recognised as a way of contextualising data and measurement and as a way to bring the numbers to life in a way with which people can relate. May I suggest take a look at the short piece on Public Narrative in the Enagagement to Mobilse knowledge share And take a look at Professor Hans Rosling of International Health at Karolinska Institute – Swedish medical doctor, academic, statistician and public speaker. 11 great minutes at http:/ |
|
Re: Transparent measurement |
Great stuff! Hans also talks about how the US is the world leader on making data transparent in another TED talk: "Let my dataset change your mindset" (available from the same website Stuart posted above). |
|
Re: Transparent measurement |
Hi everyone. Those of us who are not clinicians but have worked closely with them for long enough know well that they don't have the time for collecting yet more data to demonstrate improvement. I came up with a way to do this without troubling them whilst still providing them with a visual outcome of what is being achieved following a transformational change project. So, no data is collected at all to achieve this, although data quality reporting would obviously go some way to providing more detailed validation. In the case of demonstrating RiO benefits, I mapped 30 or so RiO functions to both operational and organisational benefits (a mixture of outcomes and objectives). Since the Programme office knows what functionality the clinical services have taken, we can fill in the tool for the services. It imediately shows which benefits. The approach has its foundation in MSP (Managing Successful Programmes) which gives it credibility. It also performs a task which is very difficult to do quickly - connecting activities or functions to operational and strategic benefits and objectives. For operational managers, it informs which operational benefits are being achieved whilst also providing them with a means to see how those also contribute to the organisations strategic obejectives. For large transformation programmes in particular, this sort of understanding really should be developed. The concept is that the matrix also acts as a diagnostic tool. If, after discussion with the services, it is agreed that certain benefits are not being achieved, because we know the background mappings we can look to see what might be the cause and advise services accordingly. One final point, the method also reflects partial achievement of benefits (e.g. 50%, 75%). Several functional aspects (in the case of RiO) often combine to generate higher benefits. Enabling the reflection of partial achievement of individual benefits allows for circumstances where staff may be aware that they are using 3 out of 4 elements that produce a specific benefit so that they are still credited with this rather than a complete non-recognition because the benefit has not been fully achieved. It's a framework that, with a bit of thought, could be adapted for most transformational projects.
|
|
Re: Transparent measurement |
Often when we measure, the results obtained do not show what was intended -e.g. the Sept. 2012 analysis of the 'End of Life Care Pathway' seemed to show no clear benefits could be obtained in terms of cost, and as for 'care quality', the abysmal training in roll-out has been responsible for extremely bad press for the NHS. |
