The history of the data economy: The birth of customer insight

Data is now the fuel that drives business, identifying potential markets, shaping new products, and targeting would-be consumers. To understand where we may be heading next, Impact has partnered with Significance, the magazine of the Royal Statistical Society, to jointly publish a series exploring the past, present and future of the data economy. The first part tells the story of the birth of customer insight. By Timandra Harkness

Today’s data-driven economy relies heavily on mathematics, statistics and computer science, but its roots owe as much to pragmatic trial and error as to pioneers of social statistics such as Adolphe Quetelet.

While theoreticians wrangled over how humans varied, and how to quantify this, those who saw the value of data in the nascent mass society were already collecting and using it.

The first people to treat public opinion as a form of data were newspaper publishers in 19th-century America, who used ‘straw polls’ of readers to anticipate election results. The Harrisburg Pennsylvanian’s 1824 presidential election poll is often cited as the first political poll. It accurately predicted that Andrew Jackson would win the popular vote, though John Quincy Adams was ultimately elected president.

This straw-poll approach continued in use until the 1930s. Although newspapers actively went out to survey different groups in the population, they relied more on very large numbers of responses than on any statistical theory to accurately reflect the mood of the nation.

Meanwhile, America’s flourishing mail-order market made customer information so valuable that letter-brokers bought and sold customer letters. Those letters, originally solicited by newspaper adverts or leaflets, might include not only names and addresses, but also useful details such as medical histories. In 1910, Louen Atkins, of Chicago, accused his former business partner James Rainey of taking data from his mailing list to poach a customer. The dispute culminated in Rainey shooting Atkins dead.

The research profession

In the early 20th century, this kind of ad hoc research began to take a more coherent form with the professionalisation of marketing. Market research pioneer Archibald M Crossley reports applying for a job with a Philadelphia advertising firm in 1918. “My prospective employer asked how I would like to set up a research department. I said: ‘I would. What is it?’ And his answer was: ‘I don’t know either.’”

Crossley did some research into research and found that many other advertising agencies had research departments already. The most influential among them was probably J Walter Thompson (JWT), founded in 1878, and trading today as Wunderman Thompson, part of marketing group WPP. Stanley Resor, who took over JWT in 1916, believed that human behaviour, taken en masse, could only be understood through statistical and scientific study.

Some magazine publishers also had research departments to help them attract lucrative advertising and make it more effective. Charles Coolidge Parlin, widely regarded as the world’s first professional market researcher, was hired by the Curtis Publishing Company in 1911. His extensive research into entire sectors, first agriculture and then automobiles, produced volumes of data and analysis.

Duly informed, Crossley set up a research department for his new employer. After a stint in the research department of the Literary Digest magazine, he established his own research company in 1926. By Crossley’s account, this kind of quantitative research started out partly as a sales technique for advertising firms, to help them compete for clients, but came into its own with the growth of mass media. Advertising spend in the US increased tenfold between 1900 and 1930, and clients wanted to know whom their radio – and, later, television – adverts were reaching, and to what effect.

Coming at the same question from another direction, Arthur C Nielsen set up a business to test the quality of conveyor belts and turbines in 1923, before applying similar methods to market research. As an engineer, Nielsen applied rigorous statistical techniques of probability sampling to new problems – such as calculating brand market share – and, later, to measuring broadcast audience habits.

A revolution begins

George Gallup revolutionised quantitative market research by bringing together statistics, journalism and psychology. The method he outlined in his doctoral psychology dissertation, ‘A new technique for objective methods for measuring reader interest in newspapers’, transferred to human attitudes the method used by inspectors of wheat or water – testing a number of small samples to assess the whole.

While working as director of research for New York advertising agency Young & Rubicam, Gallup began to widen his focus beyond studying consumer responses to journalism and advertising. This sampling approach could equally be applied, he thought, to public opinion on politics and social issues.

In 1932, his research helped his mother-in-law, Ola Babcock Miller, to run as Iowa’s secretary of state. In 1934, Gallup’s predictions came within one percentage point of the congressional election results.

The final overthrow of the newspaper straw poll by more statistically robust methods came in 1936. Now running the American Institute of Public Opinion from a small office in Princeton, Gallup used the results of his surveys to produce a regular syndicated column, America Speaks.

Gallup challenged news magazine the Literary Digest to beat his methods with its straw-poll forecast of the presidential election results. He judged, correctly, that the Digest’s straw poll over-represented people with telephones and cars, who were unlikely to vote for Franklin Delano Roosevelt, the Democratic candidate. Gallup’s surveys used a quota system to match the electorate demographically, and he correctly predicted a Roosevelt victory. The Literary Digest closed down not long afterwards.

The following year, Gallup polling arrived in the UK. Harry Field, a Briton who had worked with Gallup at Young & Rubicam, was despatched to the London School of Economics (LSE) to find a suitable leader for a British Institute of Public Opinion (BIPO) to mirror its American cousin. Field convinced research student Henry Durant to take on the job.

The role must have appealed both to Durant’s political leanings and his lack of private means. The son of a warehouseman, he had won a scholarship to Christ’s Hospital school and then worked as an insurance clerk before studying sociology at LSE. The £150 per year salary from the BIPO would help support him and his wife while they pursued their academic research careers, and the prospect of giving the public a voice on social and political issues chimed with his left-wing views.1

With Durant in post, Field returned to the US to establish the People’s Research Corporation, and then initiate the American Association for Public Opinion Research and the World Association for Public Opinion Research. Tragically, he was killed in an air crash in Paris before either was established.

The population of inter-war Britain was studied by a number of government bodies – not only as citizens, but also as consumers. The Empire Marketing Board, set up to promote the consumption of goods produced within the British Empire, segmented its audience according to social class and sex, placing adverts in the relevant papers, and distributing targeted posters and pamphlets to schools and Women’s Institutes. It enlisted advisers from the advertising industry, notably William Crawford, whose 1938 report The People’s Food ruffled government feathers with its finding that millions of British citizens could not afford to eat properly.

During World War II, the distinction between commercial and political polling became almost meaningless. Governments on both sides of the Atlantic took control of information, mindful of the need to know the level of public support for wartime policies and of the potential power of information delivered to the right audience at the right time.

In the UK, the government brought several research groups together as the Social Survey Unit, run by Louis Moss, who had been managing the BIPO. The unit directly employed researchers and social scientists to supplement official data, but also farmed out survey research to commercial agencies such as JWT’s London branch and Britain’s largest advertising agency, the London Press Exchange.

Random sampling methods, stratified for occupation, age, sex, and so on, produced tabulated data on vital issues, including bicycle use, attitudes to fuel rationing, demand for brooms, and cake consumption in private homes. From 1940, long-running surveys emerged from the Social Survey Unit’s work. The National Food Survey, for example, ran for 60 years, until 2000, when it was merged into the Expenditure and Food Survey.

What people say and do

After the war, both flavours of public opinion – the commercial and the political – continued to be valuable. The Market Research Society was formed in 1946, with members from public and private organisations. Initially, it was a couple of dozen people meeting over lunch in Soho, but within 10 years it had hundreds of members and held its first conference in Brighton in 1957. Henry Durant was its first president.

Durant established his reputation, as Gallup had done, by correctly predicting the outcome of an election; his polling anticipated Labour’s 1945 victory under Clement Attlee. Durant’s background in social science and actuarial work proved a good foundation for innovation in polling techniques. He adopted Gallup’s ‘quintamensional design’ – five questions designed to find out a respondent’s knowledge about an issue, their level of interest in it, their attitude to it, reasons for the attitude, and strength of opinion.

As Durant’s experience grew, he refined other aspects of data collection by survey – but he saw problems, too.

In a frank article for The Incorporated Statistician in 1954, Durant discusses practical issues in data gathering.2 How, for example, would you discover drinking habits and consumption though a field survey? You could visit people at home, but unless you weight the responses, people who do not go out much will be over-represented. If you do weight the responses, the ‘only home one night a week’ group will be represented by the smallest sample, giving the least reliable results.

Home interviews could also elicit less accurate answers. “Husbands may not want their wives to know the truth about the amount they drink,” says Durant. But if interviewers were, instead, to be stationed outside pubs, might “there not also be the danger that interviewers will tend to avoid the rough-and-ready types, who, in fact, do consume more than their due proportion?” asks Durant, reasonably.

Some of these problems were mitigated, to an extent, by the advance of technology. As more households acquired their own telephones, phone interviews began to take over from face-to-face surveys. Because each telephone number was linked to a specific household, randomised sampling became more practical and cheaper to execute, because the interviewer did not need to travel. Later still, the internet and smartphones provided easier, lower-cost ways to contact potential interviewees.

By that time, however, survey data collected directly by asking questions had competition. Data generated as a by-product of our everyday activities could, potentially, reveal far more about us than we would willingly reveal in words. All that was needed were the right techniques to analyse this data.

You can download the complete four-part ‘History of the data economy’ here.

Reference:

1 Roodhouse, M ( 2013 ) “Fish-and-chip intelligence”: Henry Durant and the British Institute of Public Opinion, 1936–63. Twentieth Century British History, 24( 2 ), 224–248.
2 Durant, H ( 1954 ) The Gallup poll and some of its problems. The Incorporated Statistician, 5( 2 ), 101–112.

We hope you enjoyed this article.
Research Live is published by MRS.

The Market Research Society (MRS) exists to promote and protect the research sector, showcasing how research delivers impact for businesses and government.

Members of MRS enjoy many benefits including tailoured policy guidance, discounts on training and conferences, and access to member-only content.

For example, there's an archive of winning case studies from over a decade of MRS Awards.

Find out more about the benefits of joining MRS here.

0 Comments

Display name

Email

Join the discussion

Newsletter
Stay connected with the latest insights and trends...
Sign Up
Latest From MRS

Our latest training courses

Our new 2025 training programme is now launched as part of the development offered within the MRS Global Insight Academy

See all training

Specialist conferences

Our one-day conferences cover topics including CX and UX, Semiotics, B2B, Finance, AI and Leaders' Forums.

See all conferences

MRS reports on AI

MRS has published a three-part series on how generative AI is impacting the research sector, including synthetic respondents and challenges to adoption.

See the reports

Progress faster...
with MRS 
membership

Mentoring

CPD/recognition

Webinars

Codeline

Discounts