Reg, I was struck by what you said near the end about how researchers appear to be more concerned than clients about data quality. It's amazing to me how the business world is so insistent on hard evidence and "proof points" for ROI and other essential metrics yet often seems to accept survey data at face value, with no apparent assessment of the underlying validity and reliability. My observation is admittedly anecdotal but is based on countless cases where research findings are reported/relayed -- in social media, elsewhere online, and on dead trees -- with little to no methodological disclosure. I must say this applies even to two stories atop the research-live.com home page as I write. And, again anecdotally, it seems to me this is particularly a problem with publicly reported consumer/market research; political and other public affairs polling has undergone more scrutiny in recent years as major media organizations become more attentive to data quality and with the advent of blogs like Pollster.com. Specific survey percentages seem to be shiny objects that distract many otherwise sophisticated consumers from thinking critically about whether the estimates really are representative of the population being studied. This baffles me. (My own disclosure: Reg and I serve on the Executive Council of the American Association for Public Opinion Research, where the issues of methodology disclosure and survey representativeness are and will continue to be hot topics.)

Like Reply Report