Overcoming complex NHS management challenges - insularity, short-termism and a sometimes crippling hierarchy - will require major changes to the way managers operate, argues Nigel Edwards

Quite a bit of our time at the NHS Confederation is spent defending NHS organisations and management against various types of attack. Much of this is ill-informed and, in the worst cases, based on little more than opinions of columnists' dinner party guests.

There is a good deal of evidence to suggest NHS management at its best is as good as the best in other sectors. People coming into senior management roles from the private sector can confirm that healthcare is a complex field and running an organisation populated with so many professional staff is particularly challenging.

But one of the effects of being continuously attacked is the development of a siege mentality. The key to developing and improving is having time for honest reflection and learning to identify weaknesses and areas to work on. The culture of the NHS in the last few years has not been conducive to self-examination.

As we move to a more devolved system, it is going to be even more important to take a critical look at NHS leadership and management. We asked a range of leaders to do this and they identified a number of areas where NHS leaders could do better.

Some of these relate to technical skills required to do the job, such as using data, strategic planning, working with local authorities and, of course, commissioning.

NHS North West chief executive Mike Farrar agrees there is a need to improve technical skills but emphasises that the new world also demands a range of sophisticated ways of dealing with the emotional side of organisational life. He says many managers need to pay more attention to developing and using some of these so-called softer skills.

Far from being soft, he argues these are actually some of the toughest areas of management. 'These skills are essential if we want to create sustainable leaders. It's what differentiates the great from the good because it requires conversations about really difficult issues,' says Mr Farrar.

He says these skills will be vital in addressing the complex challenges now facing NHS management, particularly the need to change behaviour among clinicians, service users and the public.

Evidence gap

A second area where there is scope for improvement is in the use of evidence to inform decision-making. This includes more careful and rigorous analysis of arguments for changes and the use of evaluation to learn from previous decisions. This was highlighted as a potentially important way of improving the management legitimacy for clinicians, who are often suspicious of the lack of rigour in NHS decision-making.

The people we spoke to thought there was often too little leadership in NHS organisations. They were also concerned that there were significant weaknesses in middle management.

A number of factors can get in the way of the development and exercise of successful leadership. Most people we spoke to recognised the need for leaders to think in terms of systems rather than just the interests of their organisation, and to focus more on medium- and long-term goals.

It was generally felt that the design of many aspects of current accountability arrangements and system reform, particularly payment by results, tended to create insularity and short-termism. The heavy workload and general feeling of overload that many leaders experience may also contribute to these problems.

While some of these problems relate to the leaders and managers themselves, quite a lot of them are a product of the system in which leaders operate. The most notable source of problems seems to be related to hierarchy.

While hierarchy does have some significant advantages, it also comes with major problems. It hinders and distorts communication in both directions and introduces times lags in decision-making.

There is a tendency for people at the top to assume they have a clearer view about what needs to be done and to underestimate the difficulties of implementation. While it is quite likely that they have a different view, it is not necessarily better.

Even where hierarchy is effective, there is a question about the extent to which it is scalable and whether complexity and co-ordination costs outweigh any benefits gained from increasing scale.

For example, at what point does the system get so large that values become meaningless slogans? Large scale means small errors multiply into major problems.

It may also mean policy-makers tend to concentrate on making policy that at best is aimed at the average trust but often appears to be targeted at the worst performers.

Perhaps the most pernicious effect of hierarchy is the creation of a dependency culture. This is particularly true in a risk-averse setting such as the NHS.

If the culture works by top-down direction and boards and local leaders are expected to look upwards for instruction and performance management, then it is likely that this will create disempowerment and even a failure to pay attention to some key areas. The more the dependency culture develops, the more the upper levels of the hierarchy need to intervene.

Devolution to the front line and increased autonomy for trusts are clearly part of government policy. The anxiety is that hierarchy tends to be very resilient and to reassert itself at the first sign that things are not going to plan.

The arguments for increasing autonomy are partly the reverse of those against hierarchy but there are some additional aspects worth rehearsing. Greater autonomy tends to be associated with improved problem-solving and innovation, increased responsiveness, reduced stress and higher levels of discretionary effort. But the potential for too much autonomy to create chaos is considerable.

Misleading perspectives

The view of markets being based on adversarial relationships, which seems to underpin some views of the new system, is a caricature of real markets. In complex and dynamic market environments collaboration and competition co-exist.

Autonomy therefore has to be accompanied by a clear set of outcomes and objectives, responsibility to the wider system, appropriate behaviours and levels of effort contributed by each organisation and leader.

There is much to be proud of in NHS management but it is important that we are not complacent or afraid to point out where it is failing to deliver or not living up to the values that should underpin it.

There has not always been enough willingness to do this. Leadership development is important but not enough; the environment in which leaders operate also needs to change and it is this rather than debates about independence that is likely to really make a difference.

The foundation trust movement is a major step in this direction. We need to think about an equivalent for primary care trusts.