Speaking at a conference for non-executive directors, Mr Nicholson said performance management would take place at a local level.
He said: 'We want to performance-manage organisations around making sure trusts and PCTs deliver what they say they are going to deliver.
'Boards increasingly must know what their population and patients need and stretch their ambition for patients and the organisation.
'It is important to get traction in the system because boards have been looking upwards to the centre. It is right that boards are accountable for the money they spend, but they need to be accountable for the things they say they are going to do. That is the next challenge.'
NHS Confederation PCT network chair David Stout said the idea was right but getting the mechanisms in place would be difficult.
He said: 'I generally agree with the idea but what is the mechanism for delivering that? We must look at the role of regulators and the role of strategic health authorities.
'Local service plans could be one of the main ways of providing local accountability.'
NHS Alliance chair Dr Michael Dixon said: 'Hopefully in the future PCTs could be performance- managed by the local population through public and patient involvement, which has not been a great success during the last few years.'
Mr Nicholson, who was speaking at a Health Care Conferences event, said the NHS was at a pivotal stage, with the phase of reform and capacity-building coming to an end.
'We haven't got our heads round how to use the reforms for the benefit of patients,' he said. 'We need to look at how we empower the frontline people and give them the levers, which I would argue are already there, to make those changes.
'I am trying to get the department to a place where it lets people get on with it. I have to create an environment, but no-one has ever been fired for improving services for patients. There is an element of being surprised that if you push the system you will be able to do more.'
Mr Nicholson was also critical about the lack of staff development in the NHS.
'We don't talent spot, we don't nurture, we don't bring people on,' he said. 'It is difficult in terms of recruitment of chief executives. In some cases there is only one person on a list who could be appointed and in some cases none.
'The average length of post is about 700 days for managers, who will be working alongside clinicians with 30 years' experience.
'We don't have a large number of clinicians in management positions. The more we get capable clinicians in senior positions, the more likely we are [to succeed] in making the changes we need to.'