Richard Granger launched the attack at the first hearing of a Commons health select committee inquiry into the controversial project.
'People think they can assess the programme after researching it on Google,' he complained. 'A number of people are whipping up anxiety - privacy fascists - who want to dictate that no-one has a right to information anywhere.'
But he admitted: 'No system is ever going to be totally secure.'
Mr Granger compared concerns over patient confidentiality to an 1834 editorial in The Times, which had argued against the adoption of the stethoscope.
The project will be largely implemented by 2010, although some parts of it are running roughly two years behind schedule, he told the committee. This is due to the specifications changing and a lack of consensus from doctors as to how the project should work, he said.
He vigorously denied a suggestion by Labour MP Jim Dowd that this meant the original plan had been flawed. 'It would be a fantasy to imagine that halfway through a 10-year programme we would only be doing the same things we set out to do five years ago.
'Now we are doing e-mail services, GP payment systems, and bowel cancer screening services, which were never on the original list.'
He rejected calls for an independent technical review into the project, which he said had already come under a significant amount of scrutiny.
The care records service is being developed in two stages - a summary record of key information derived from GP records, and in the longer term, a more detailed record containing full notes of consultations with GPs and consultants.
Committee chair Kevin Barron MP questioned whether this was a 'political compromise'.
Mr Granger responded that the DoH had always wanted two types of record because some health professionals did not always need to access highly detailed information.
The committee discussed the pros and cons of patients opting into the system, as opposed to Mr Granger's preference of opting out.
Only 0.1 per cent of the population had opted out of pilot schemes around the country, he said.