star-ratings

Published: 12/09/2002, Volume II2, No. 5822 Page 26 27

Changing a chief executive has little impact on a trust's star-rating, at least in the first 12 months, a survey has shown.

David Dawes reports What effect does a chief executive's experience have on a trust's star-rating? Can a change of chief executive be expected to improve the organisation's performance? And if so, how quickly?

Following the recent publication of the second round of NHS performance ratings, the NHS Leadership Centre undertook a study to try to identify the contribution that chief executives make to their organisation's star-rating.

In September 2001, the government published the first NHS performance ratings for trusts providing acute hospital services. The six areas covered by the performance assessment framework are:

improving people's health;

fair access to services;

delivering effective healthcare;

efficiency;

the experiences of patients and their carers;

health outcomes.

Taken with the key target measures, the measures of clinical, staff and patient focus constitute a 'balanced scorecard' approach. A trust that has received a performance rating of three stars will have demonstrated high standards of performance against the key targets and the three areas above.

Similarly, trusts with a performance rating of two stars will have demonstrated good standards of performance in most, if not all, of these areas.

Those trusts with a performance rating of one star have demonstrated some areas for concern in either the key targets or the balance scorecard. The trusts with a performance rating of zero stars have shown significant areas for concern in the key targets.

In 2000-01, 173 acute trusts were rated and 35 received three stars, 103 two stars, 23 one stars and 12 zero stars. In 2001-02, 304 trusts were rated for their performance, with 68 achieving three stars, 172 two stars, 54 one stars and 10 zero stars.

Unfortunately, due to mergers and reconfigurations, only 143 of the original 173 trusts remained in existence as individual organisations between the two years. Although it is possible to compare the ratings of organisations that have merged, it was felt that for the purposes of this study, only those organisations that we could uniquely identify in both years would be analysed.

Overall, 46 NHS acute trusts (32 per cent) received an improved performance rating, 63 stayed the same (44 per cent) and 34 (24 per cent) received a lower rating.

We contacted the 143 trusts to find out who their chief executive was, how long they had been in post and whether this was their first chief executive appointment. This data was compared with existing chief executive databases to ensure that the data was as reliable as possible.

Changes in chief executives Twenty-seven of the trusts had changed their chief executive since the first performance ratings assessment, representing 19 per cent of the trusts.

Only 23 of the trusts had a female chief executive (16 per cent of all trusts), and the first analysis was to see whether there was any relationship between the gender of the chief executive and the change in the trusts' star-rating. Not surprisingly, a chi-squared analysis showed that there was no significant variation between male and female chief executives (p=0.928, not significant at the 10 per cent level).

Does having a more experienced chief executive affect performance?

The second analysis was whether more experienced chief executives were more (or less) successful than less experienced colleagues. On average, chief executives had been in post 3.8 years. Interestingly, a chi-squared analysis showed that there was no significant relation between the change in the trusts' rating and the length of time that the chief executive had been in post.

This analysis simply looked at length of time in the current post, so we then analysed the results according to whether this was the individual's first chief executive post or not.

For 64 of those questioned, this was their first chief executive appointment (45 per cent of the trusts), which suggests that there is not only a high turnover at this level, but that almost half are firsttime appointments. Again, a chi-squared analysis showed no significant relation between the change in the trusts' rating and whether this was a chief executive's first appointment.

Does changing your chief executive affect performance?

Twenty-seven trusts had changed their chief executive between the 2000-01 performance rating exercise and the one in 2001-02. Even though we failed to find any relationship between experience in post and previous chief executive experience, we expected to find some impact from changing the chief executive. In fact, of the 27 trusts, eight had performed worse, 10 had remained the same and nine had improved. Again, a chi-squared analysis showed that these results were statistically insignificant (p = 0.583, not significant at the 10 per cent level).

This finding was the most surprising and potentially the most important in light of the government's franchising proposals for nostar trusts.

Franchising involves finding the best available managers to take on the role of chief executive and, where necessary, supporting teams.

But this study casts doubt on the impact of a new chief executive on the star-rating of their trust, and further research is probably necessary to understand why this is so.One factor that may be significant is why the previous chief executive left and whether these were for positive reasons such as promotion or retirement, or for negative reasons such as poor performance.

This was not something that was analysed and it would probably require a different methodological approach, as well as a great deal of sensitivity and tact.

Statistics can never explain complex issues, but they can raise questions and challenge assumptions. Intuitively, we are confident that chief executives have a major impact on their organisations, otherwise why do they command the salaries they do and why is so much effort and so many resources spent on ensuring that we continue to develop and support them?

The results of this study were surprising as we were expecting experience in the role and previous roles, and a change of chief executives, to have a measurable impact.

There may be many reasons for these findings.

Can one individual make a significant impact on an organisation as large and complex as an acute trust within less than 12 months? Were improvements already in the pipeline when the previous chief executive left? What of the impact of the entire executive team?

An alternative hypothesis is that chief executives are not significant in changing the performance of an organisation.

Given the high turnover and the relative lack of previous experience, it is possible that chief executives have very little to do with how their organisation as a whole functions. If this was true, it would challenge many of our assumptions about the relationship between chief executives and organisational leadership, and perhaps shift the focus from leadership as the characteristic of an individual to leadership as an organisational capacity.

In other words, perhaps leadership is not something that rests in a few talented, wellrewarded individuals, but is an aspect of a whole organisation and reflects how whole teams support each other, learn from mistakes and instigate service change.

The study suggests that changing the chief executive of an organisation is unlikely to have any significant impact within a 12-month period.This has serious implications for how franchises are evaluated and the timescale over which service improvements are to be measured, and suggests that holding new chief executives to account for their organisational performance within the first 12 months is unlikely to be effective.

It is hoped that this study raises debate about the relationship between the chief executive and the performance of their trust, and the way in which success and failure are evaluated. Further research is needed on what is clearly a complex organisational issue and may lead to a greater understanding of the impact of chief executives and/or the measures by which one judges success or failure in today's NHS.

Key points

Analysis of trusts that changed their star-rating over the past two years indicates that a change of chief executive was not a significant factor.

The length of time in post and the experience of the chief executive were also insignificant.

This has serious implications for the theory behind franchising and the evaluation of franchised trusts.

Holding chief executives to account for the organisation's performance within their first 12 months is unlikely to be effective.

David Dawes is the e-learning development manager at the NHS national nursing leadership programme, NHS Leadership Centre.