The majority of managers have been found to support Agenda for Change even though it overspent by millions of pounds and has failed to increase productivity.
Research by the King's Fund found that the pay system, introduced in 2004 to improve career progression and patient care in the NHS, has not met expectations.
Realising the Benefits? Assessing the Implementation of Agenda for Change, concludes that most of the ambitions of the pay reforms 'remain unfulfilled'.
It highlights the scheme's massive overspend of more than£220m, comparing it to other maligned pay schemes such as the GP and consultants' contracts.
Despite pay rises of up to around 15 per cent, many staff are disillusioned after promised appraisals and personal development plans failed to materialise.
The knowledge and skills framework, which sets out competencies for each role, has been 'cumbersome and costly to implement', it says.
However, most of the managers interviewed were in favour of the new system, saying it was fair, offered protection against equal pay claims and provided scope for new roles and working practices.
James Buchan, who co-authored the report, said he hoped his work would spark an independent analysis of Agenda for Change.
He told HSJ: 'It's a central element of human resources policy and involves more than 1 million staff and clearly cost a lot of money, but there's never been any evaluation of where it's going.'
A major problem, he said, was the lack of time given to managers to implement the pay reforms, as well as inconsistencies at trust level.
NHS Employers said the reforms had provided a fairer, more transparent pay system, but trusts were not yet reaping the full benefits.
Deputy director Alistair Henderson said he would welcome a systematic, long-term audit.
Unison argued that the scheme had not overspent but was underfunded by the government.