Humans vs AI: Researchers take on ChatGPT

At a conference that had artificial intelligence (AI) as a core theme on the agenda, it seemed appropriate to end day one of our recent international Association for Survey Computing (ASC) conference with a fun ChatGPT challenge in the bar of the Student Union at Royal Holloway University.

Six teams of conference attendees agreed to take the ChatGPT challenge and pooled their creative talents over a late evening pint or two. They had 45 minutes to complete four tasks head-to-head against answers already generated through ChatGPT. 

The tasks were:

1. Writing a poem entitled ‘data’;
2. Describing the future of market research;
3. Creating a joke about market research
4. Deciding on one item a market research technologist would take to a desert island.

The teams got to work intensely debating and creating. Some worked in silence, delegating tasks across the team, others were more vocal brainstorming their masterpieces. But the dominating sound was laughter, which only diminished when the two-minute warning was announced.

Our three expert and impartial judges were ASC directors Rebecca Cole and Tim Macer, and ASC alumni Pat Molloy. Entries were shared with them anonymously, so they had no idea which were human, and which were AI generated. The tension mounted as the scores were collated, and the event host took to the mic to announce the results.

With a score of 26 out of 30, ChatGPT was the winner of the ‘data poem’ challenge, perhaps the most difficult and time-consuming of them all. This impressive verse was created in just three seconds:

‘In bytes and bits, the stories flow,
A silent dance, the numbers know.
From depths of code, insights arise,
A digital truth, behind the guise.
Patterns whisper, secrets unfold,
In data’s weave, the future is told.
Infinite streams, a realm to explore,
Data’s tale, forevermore.’

But our market research ‘humans’ fought back and showed that there is no substitute for context and a little humour when it comes to creativity. When asked to create a market research joke, ChatGPT scored only 16 points with: ‘Why did the market researcher bring a ladder to the office? Because they wanted to take their data analysis to the next level.’

But with an incredible 26 points out of 30, our top ‘human’ team crushed ChatGPT with: ‘What is a clown’s favourite statistical chart? A clustered pie.’ (Editor: It made me laugh)

And there was no stopping our humans when it came to what a market research technologist would take to a desert island. ChatGPT struggled and scored a lowly 17 points with: ‘Their data analytics software because even on a desert island, they’d want to survey the local coconuts.’

But our humans didn’t disappoint, with several teams earning 24 points out of 30, with perhaps the pick of the answers being: ‘Quantum on a dongle, because we are geeks’ (Editor: the scary thing is that they probably would) and ‘Rebecca Cole from Cobalt Sky, because she would bring a boat!’ (Editor: Which got a 10 from one of our judges).

In the end, five out of our six teams beat ChatGPT across all four challenges, retaining the honour for human beings, and they all took home the trophies and medals to prove it.

Was this a rigid scientific experiment? Perhaps not. But it was an awful lot of fun and rounded off the day, with an experience that was talked about until the end of the conference. Oh, and by the way, this write-up was written by a human.

AJ Johnson is chief operating officer at Mediaprobe and executive committee member at the ASC