The study, led by Martin Roland, professor of health services research at the University of Cambridge, looked at the results of the latest GP Patient Survey, which was used to allocate £68m of GPs’ pay this year as part of the NHS quality and outcomes framework.

Results from the survey, conducted for the Department of Health by Ipsos Mori using postal questionnaires in January-April this year, have contributed to a fall in payment for some practices.

The survey questions that are linked to pay relate to the availability of urgent appointments and the ability to book appointments in advance. Professor Roland and colleagues looked at the representativeness of respondents, the effect of sample sizes on reliability of data and the relationship between response rates at different practices and the scores they received.

The research found that the characteristics of a practice’s patients (such as age, sex, ethnicity and levels of deprivation) did affect response rates, and that practices with lower response rates tended to get lower scores. However, the researchers said there was little evidence that high or low response rates led to any “systematic disadvantage” for practices.

Although the survey met recognised reliability standards, margins of error for some practices were as high as plus or minus 7%, and changes to the payment formula this year have increased the effect that this random sampling error could have on payments.

The researchers said that for performance-based incentives to be effective, they must be seen to be reliable, valid and have “a strong correspondence between actual performance and compensation”.

In an accompanying editorial piece in the BMJ, Chris Salisbury, professor of primary healthcare at the University of Bristol, said larger samples should be used to prevent large year-on-year fluctuations in payments. The current system means that an individual practice might receive anything between 15% and 85% of the maximum payment “simply because of random sampling error”, he wrote.

Salisbury also called for further research to pin down whether the lower scores given by certain sectors of the population are attributable to worse care or simply different perceptions.

“Surveys are a good way to measure subjective views,” he wrote, “but if the aim is to measure system performance objectively, other methods may be more reliable. For example, waiting times can be measured using data extracted from electronic appointment systems and appointment availability assessed using simulated patients or an independent audit.”