Published: 11/11/2004, Volume II4, No. 5931 Page 28 29

Senior managers are increasingly using individual accounts as indicators of success over traditional hard data. Jay Bevington and colleagues report on the rise of 'soft intelligence'

We still trust the soft stuff when making assessments about the performance and quality of our health services, according to new research by York University. In fact, less than one in four of those interviewed used hard data to inform their judgements of NHS trusts, preferring more informal sources of information such as the views of colleagues and reports from patients, family and friends, as well as their own experiences.

Soft information is important because 'people do business with other people, not with organisations', says Greater Manchester strategic health authority chief executive Neil Goodwin. Soft data gives the SHA important clues about the probability of success. 'If you're talking about financial recovery there is a fairly standard approach: a bit of analysis and then proposals for action. But you never completely form a view as to whether they will achieve that just by looking at their plan.You form a view based on the soft information - how effectively they engage clinicians, how good they are at inter-organisation relationships etc...' There is also increasing recognition that apparently hard, quantitative data can be unreliable. As history has shown, trusts can be made to appear sound on paper.

An organisation might also be good at consistently hitting targets, but the spirit of what it is trying to achieve might have disappeared. 'We never totally rely on hard performance data', says West Yorkshire SHA chief executive Richard Jeavons. 'We always test the soft, even for organisations getting the numbers right.'

Soft intelligence is information gathered from conversations, observations and experiences that allows judgements to be made about something (eg capability of a chief executive and staff morale). These sources of information can be direct: based on the first-hand experiences of the people making the judgement; and indirect: the accounts and views of colleagues, patients, public, friends and journalists.

The soft intelligence might be at odds with the hard information.Mr Jeavons says: 'Sometimes we know things are not right despite getting nice numbers.'

Under circumstances like these, 'my expectation based on experience is that if the soft intelligence is consistently good, the hard indicators will sort themselves out and improve'.

'But if the people and culture are not right, the ticks in the boxes will eventually fail. An organisation showing lots of positive soft-intelligence signs gets credit on the harder stuff. This means that you are happy to give it some space and wait for things to improve. An organisation with lots of hard targets ticked but no soft intelligence credits is under closer scrutiny than the hard intelligence says it should be, ' says Mr Jeavons.

Mr Goodwin agrees: 'A lot of soft intelligence is about relationships and the test of any relationship is when things go wrong. If people invest enough time and effort in building relationships in the good times, when difficulties do arise there is a history of trust.'

So what kinds of soft information are SHA chief executives most attentive to when making judgements about performance on their patch?

First, SHAs are attentive to the general levels of noise coming from their extensive managerial, clinical, professional and political networks at times of major change.Mr Goodwin says: 'There are trusts across the conurbation that deliver financial balance, targets, change, but they do it very effectively and quietly. They do it because they have very good internal relationships.

Others struggle. They deliver some change, but not without generating noise - among local MPs, trade unions, clinicians and others.'

He continues: 'When I ask these people about what the experience has been like they might say, 'We were put in a corner and told we have to support this' or 'We asked them questions and didn't get proper responses.'

If every time an organisation did something it generated reactions like these, I would say it is not spending sufficient energy building relationships with its people.'

Second, the behaviour of the senior management team also comes under scrutiny.Mr Goodwin explains:

'When you meet a high-performing executive management team you see contributions from all of them.You see people expressing contrary views.You see banter.You will see the more experienced chief executives allowing members of their team to lead the discussion and only intervene when appropriate.'

He adds: 'When I sit down with Greater Manchester chief executives, I focus on the difficult issues and ask them: 'If you were in the same position again, would they do anything differently?' If they say 'nothing' you know you're going to have difficulty.'

Similarly, Mr Jeavons states that self-aware chief executives 'are people who invest time and effort in reflecting on their practice, with help'.

'In their interactions with me or key others, they demonstrate a trusting and open approach.You see how they react to different situations: do they call on their networks for advice and support, or curl up and hide away?'

Finally, both chief executives agree that 'groupthink' - when two or more people think too much alike - is a very serious risk. They are attentive to various soft signs that this might be happening, especially between chairs and chief executives. 'If an organisation is pursuing one set of targets over and above another, if it is not able to demonstrate that it is giving equal attention to a range of financial and access targets, I would suspect groupthink because they would be trying to play a different game, believing they would win, ' says Mr Goodwin.

Since much soft data is collected via conversations with individuals that are often unrecorded, there is a risk of bias and distortion. The quality of information depends to a large extent on who is spoken to and the kinds of questions. There is a danger of basing major decisions on anecdotes that might sometimes be several years old rather than judgements based on personal experiences that are current. There is also the risk of people thinking that there is 'no smoke without fire' and the potential negative impact that this might have on the trust's reputation.

'Unless we can triangulate what We are hearing with other sources of information we would dismiss it, ' claims Dr Goodwin. Likewise, Mr Jeavons describes a process that begins with 'listening-station' activity. 'The station is permanently switched on, picking up signals and seeing if patterns emerge. For instance, four or five conversations over a couple of weeks might produce a common thread.You pick a lot up in the listening station, but I wouldn't believe anything unless I check it out directly.You have to filter and test. Four weeks worth of intelligence might produce a hypothesis. A few weeks later, I test it.'

In order to produce more rounded judgements on the performance of trusts, the NHS should work towards creating a collage of hard and soft data. Soft information is here to stay. The key to success is the degree of rigour and imagination involved in its collection, interpretation and use.

Dr Jay Bevington is associate director and Paul Stanton is director of the board development team for the Clinical Governance Support Team.Ron Cullen is director of the CGST and head of strategy for healthcare quality and standards at the Department of Health.

Key points

Soft intelligence is information gathered from conversations, observations and experiences that allow judgements to be made.

Research by York University shows that less than one in four of those interviewed used hard data to inform their judgements of NHS trusts.

Sources of soft information can be direct: based on first-hand experiences; and indirect: the accounts and views of colleagues, patients, public, friends, journalists.