The Observatory×Dialogue
May 15, 2026
World Cup Study · Panel Quality Side Project

We asked for Japan.
Here's what we got.

A comparison of two panel providers — Prolific and GMO — for recruiting Japanese respondents in a qualitative study about the World Cup.

Part 1
The story in 7 slides
1 / 7
Part 2
Head to head
Prolific
English-proficient panel
GMO
Japan-domestic panel
26
participants fielded
Sample size
32
participants fielded
19–23%
5–6 wrong-country respondents
Out-of-market
3%
1 Canadian. That's it.
12%
3 dead transcripts
Dead / void
16%
5 dead + 1 test account
54–58%
14–15 usable
Usable sample
63–69%
20–22 usable
19%
5 of 26 female
Female rep
25%
8 of 32 female
25–34 heavy
Narrow age band
Age range
25 to 65+
Wide distribution
High
Clean English, ready to quote
Verbatim clarity
Medium
Translation artifacts
Moderate
English limits depth
Cultural depth
High
Agony of Doha, J.League, Stojković
GMO wins on the thing that matters most: it delivers Japanese people talking about being Japanese. The cost is more dead transcripts and translation noise. Those are fixable. A recruitment bias that fills your Japan sample with expats is not.
Part 3
The memes
The drake
😐
Recruiting “Japanese respondents” through an English-proficiency panel and getting 5 Egyptians, Indians, and Brits
😎
Recruiting through a domestic panel and getting someone who wakes up at 3 AM for the Spain match and references the Agony of Doha
The distracted boyfriend
😍
GMO panel
“actual Japanese people”
🫤
Research team
looking at the data
😭
Prolific Japan
“but my English is so clean”
Expanding brain
Levels of Japan panel quality
🧠
Running the screener in English and hoping for the best
🧐
Noticing 19% of your “Japan” sample is from India and Egypt
💫
Using a domestic panel and getting the Agony of Doha guy
✨
Fielding both panels and writing a 2,000-word comparison document
☀️
The test account that made it into production data (GMO moment)
The chat log
Moderator
What does the World Cup mean to you as a Japanese fan?
Prolific P03
When I was in India there was no friend with whom I can watch
Prolific P16
I'm Egyptian. I'm living in Japan now, but I'm Egyptian.
Prolific P04
True. SNS. USA. Mine.
GMO P05
That victory against Germany is a memory I'll never forget. We finally compete on the world stage.
Screener
All participants passed. ✔️
Stages of grief (panel edition)
😶
Denial“The screener will catch wrong-market respondents”
😠
Anger“Why is P20 talking about Egypt in our Japan file”
🙏
Bargaining“Maybe expats count as in-market if they live there?”
😢
Depression“14 usable out of 26. And 3 of those are one-word answers.”
🤗
Acceptance“We're running GMO. Also we built a whole skill about this now.”
Part 4
Who's actually in your sample?

Each square = one participant. Hover to see who.

Prolificn = 26
EN
EN
EN
✕
✕
✕
✕
✕
!
—
—
—
54% in-market19% wrong country
GMOn = 32
✕
!
!
!
—
—
—
—
—
TST
69% in-market3% wrong country
In-market Japanese
EN-native / heritage
Out-of-market
Low signal / unusable
Dead / void
Test account
Prolific: 14 green squares out of 26. GMO: 22 green squares out of 32.
The green squares are your actual study. Everything else is waste.