LLMs make reliable synthetic consumers, but not if you ask for a score

Most teams using LLMs for market research make the same mistake. They ask the model to rate a product on a scale of 1 to 5. The model gives them 3. Every time, or close to it.

Not because the model lacks judgment. Because you asked the wrong question.

A paper published in October 2025 by researchers at PyMC Labs and Colgate-Palmolive identifies this problem precisely and shows a way around it. Their method, Semantic Similarity Rating (SSR), produced synthetic consumer panels that matched real human purchase intent at 90-92% of human test-retest reliability. Validated against 57 actual corporate surveys and 9,300 US respondents. Zero-shot. No fine-tuning.

That figure beats a trained machine learning model by 25 percentage points. It beats two more obvious approaches to LLM survey research by a similar margin. And it works by never asking the model to name a number at all.

What Colgate was actually testing

Colgate-Palmolive runs concept tests before committing to product launches. The process: show a product idea to a panel of 200-400 consumers. Ask how likely they are to buy it, on a 1-5 scale. Aggregate the results across demographic groups. Use the distribution of scores to predict which concepts are worth developing.

This works. It is also slow. A single concept study takes weeks and costs enough that only the most promising product ideas get tested. If you have 100 concepts and budget for 10 human studies, you are already making bets before the research starts.

The research team asked whether LLMs could serve as a first screen. Not replace the human panel permanently. Narrow the concept pool before committing to the full study.

They had 57 surveys to test against: real corporate concept tests covering personal care products, toothpaste, whitening strips, that product category. 9,300 US participants total, 150 to 400 per survey. Demographics available for most surveys: age, gender, location; income and ethnicity for a subset.

The human data showed a mean purchase intent of 4.0 out of 5 across all surveys. Real consumers, when shown a product and asked if they would buy it, skew positive. The data was not flat across the 1-5 scale. It had a shape, and that shape mattered.

The question: could LLMs reproduce that shape, not just the average but the full distribution of how humans spread across 1 to 5, when given the same product concepts and demographic context?

Why asking for a number fails

The researchers tested three approaches to eliciting ratings from LLMs.

The first: Direct Likert Rating. Show the model a product concept. Ask it to rate purchase likelihood on a 1-5 scale. This is the default approach. It produced correlation attainment around 80%, which sounds acceptable. The distributional similarity was another story: 0.26 for GPT-4o, 0.39 for Gemini-2.0-flash.

The Kolmogorov-Smirnov metric used here measures how similar two distributions look. A score of 1.0 means identical shapes. A score of 0.26 means the synthetic distribution and the human distribution barely resemble each other.

What happened in practice: both models clustered almost every response around 3. They ventured to 2 or 4 occasionally but rarely reached 1 or 5. The result was a narrow, central distribution that bore no resemblance to the positively-skewed human data. A survey where every respondent rates 3/5 is useless. You cannot compare two concepts if the model treats them identically.

This is not a design flaw specific to these models. LLMs trained with reinforcement learning from human feedback are shaped by what their raters reward. A rater reviewing "I would rate this product 5/5" may flag it as overconfident. A rater reviewing "I would rate this 3/5" sees a measured response. Over millions of training examples, the model learns that the middle is safe. That instinct, useful in conversation, destroys variance in survey elicitation.

The second approach, Follow-up Likert Rating, improved things. The researchers asked the model to first write a few sentences describing its reaction to the product, then give a score. Correlation attainment improved to 85-90% for both models. Distributional similarity improved to 0.59-0.72. Better. But the model was still anchoring to a number at the end of its process, which pulled responses back toward the center.

The write-then-score approach was on the right track. SSR pushed the logic one step further.

How SSR works

Semantic Similarity Rating sidesteps number assignment entirely. The model never chooses a score. The score is inferred from words.

The prompt gives the model three things: a demographic persona, a product concept description, and an instruction to write its honest reaction as that persona. Age range, income level, region, gender. Something like: "You are a 45-54 year old woman living in the Midwest with a household income of $40,000-$60,000. Here is a description of a new whitening toothpaste. Write your genuine reaction to this product: what appeals to you, what concerns you, whether you would buy it."

The model writes a few sentences. The tone shifts with the persona. A lower-income persona mentions value and price sensitivity. A middle-aged persona weighs brand trust. A younger persona focuses on efficacy claims. The model, freed from number-assignment, produces varied prose that reflects the demographic context it was given.

That free-text response is converted to a numerical embedding using OpenAI's text-embedding-3-small model. The embedding represents the semantic content of the response as a vector in high-dimensional space.

Next, five anchor statements, one per Likert value, are also embedded. These are short, generic, domain-independent expressions of purchase intent: phrases ranging from strong rejection at 1 to strong intent at 5. The cosine similarity between the persona response embedding and each anchor embedding produces a similarity score for each Likert value. Those five scores are normalized into a probability distribution.

That distribution is the synthetic rating. The model never committed to 3. The mapping from words to numbers happened geometrically, in embedding space, based on how close the model's prose sat to each anchor phrase.

The effect: varied, persona-appropriate prose produces genuine distributional spread. The same product concept, fed to personas across the age and income spectrum, generates responses that cluster differently on the Likert scale, just as real human panels do.

The numbers

SSR with GPT-4o achieved correlation attainment of 90%. Gemini-2.0-flash achieved 92%.

Understanding what this metric means matters. The original tweet describing the paper stated "90% accuracy". The author later corrected it. The actual metric is correlation attainment: the ratio of the synthetic-to-human correlation against the theoretical maximum set by human test-retest reliability.

Human test-retest reliability is the ceiling. If you give the same survey to the same people twice, they will not give identical answers. Mood, day of the week, and attention all introduce noise. That natural inconsistency sets an upper bound: no synthetic instrument can exceed the consistency of the humans it is trying to simulate.

Correlation attainment measures how much of that ceiling SSR reaches. The researchers split each survey's participant pool into test and control cohorts across 2,000 simulations, computed the retest correlation for real data, then measured how well SSR correlated with real data relative to that benchmark.

SSR reached 90% of the ceiling. The gap between the synthetic panel and the real panel was largely explained by natural human inconsistency, not model error.

For reference across methods:

• Direct Likert Rating: 80% correlation attainment, 0.26-0.39 distributional similarity

• Follow-up Likert Rating: 85-90% correlation attainment, 0.59-0.72 distributional similarity

• SSR: 90-92% correlation attainment, 0.80-0.88 distributional similarity

• LightGBM trained on demographics and product features: 65% correlation attainment, 0.80 distributional similarity

That last row is striking. A supervised ML model, trained on this type of consumer research data specifically, achieved 65% correlation attainment. Zero-shot SSR with GPT-4o achieved 90%. The LLM outperforms the trained model not because it was fine-tuned on market research, but because it has internalized enough about human behavior to generalize in ways a narrow supervised model cannot.

What transfers across demographics

Demographic grounding is not optional. When the researchers removed persona information from the prompt entirely, distributional similarity stayed at a reasonable 0.91, but correlation attainment dropped to 50%. The demographic context is what makes synthetic ratings useful for segmented analysis. Without it, you get plausible-looking outputs that cannot tell you anything about how different groups respond differently.

But demographic signal transfers unevenly.

Age works. The human data showed a concave pattern: middle-aged respondents rated products higher than younger or older cohorts. GPT-4o reproduced this pattern reliably. The model understands, at some level, that middle-aged consumers in this product category represent the core purchasing demographic.

Income works. Lower-income personas rated products lower, a consistent pattern in the human data and one both models replicated. Budget consciousness translates into synthetic purchase intent in the way you would expect.

Product category differences transferred. The researchers' four product categories had distinct mean purchase intent levels in the human data; SSR reproduced the ranking. Price tier preferences transferred too: human respondents favored mid-range tiers over the lowest, a common phenomenon where the cheapest option reads as lower quality. The synthetic panel matched this.

Gender did not transfer reliably. Regional variation within the US did not transfer either. The likely explanation: LLM training data does not encode the specific pattern of gender-based purchase intent variation at the product category level in the same way it encodes age and income patterns. The signal exists somewhere in the training corpus, but the model does not surface it consistently through persona elicitation.

For products with strong gender or regional variation, this gap is consequential. Calibrate against existing human data before trusting SSR outputs on those dimensions.

The qualitative bonus

The paper notes a secondary finding that product teams should not overlook.

Human survey respondents skew positive. Mean purchase intent across 57 surveys was 4.0 out of 5. They also write short, vague open-text responses. "Looks good." "I'd try it." "Nice packaging."

LLM synthetic consumers were more critical and more specific. One example from the paper: "Ease of use and the promise of no sensitivity are appealing. Plus, it's from a trusted brand." That is more useful than most open-text responses from real panels.

The model has no social pressure to be polite about a product it is evaluating as a persona. It just reacts. And because it is generating coherent prose rather than filling out a form under time pressure, it produces more structured thinking about what appeals and what does not.

The researchers call this less positivity bias, and frame it as an advantage: synthetic consumers provide more discriminative signals than human respondents, who tend toward charitable ratings. A concept that scores 3.2 from a synthetic panel warrants scrutiny in a way a 4.1 from a human panel might not.

The practical implication: even teams not yet ready to use SSR scores as a primary signal can use the free-text responses from synthetic personas as a cheap pressure test on product positioning. Give the model five demographic personas and ask them to react to your product copy. Read what they say. The criticism is often more useful than the score.

Where it breaks

Three constraints matter before anyone applies this.

Domain coverage is the binding constraint. SSR works because LLMs have absorbed enough about personal care products, household goods, and general consumer behavior to produce credible reactions. Ask them to evaluate a niche industrial software product, a novel medical device with no existing category, or a B2B tool used in a specific professional workflow, and the method cannot work the same way. The model will generate plausible-sounding reactions backed by nothing grounded. The 90% correlation attainment figure applies to categories where LLMs have strong training signal. It does not automatically transfer to markets outside that coverage.

Anchor statement design affects the mapping. The researchers designed six sets of anchor statements and manually optimized them for their 57 surveys. They are explicit in the paper that uncertainty remains about how well these transfer to other domains. The mapping from free-text to Likert value depends on where the anchor phrases sit in embedding space relative to the concept responses. Different anchors shift the distribution. Domain-specific calibration requires existing human data to validate against.

Demographic gaps are real. Gender and regional patterns do not transfer reliably. If your product has meaningful variation on those dimensions, such as gendered positioning or strong regional demand differences, SSR will miss it. Plan for human research to cover those segments specifically.

What to do with this

The paper proposes a specific workflow: use SSR to screen a large concept pool, then commission human panels on the shortlist.

If you have 100 concepts and budget for human research on 10, run SSR on all 100 first. Correlation attainment of 90% means weak concepts will identify themselves with high reliability. Genuinely strong concepts will cluster at the top. The human panel confirms or reorders the shortlist. It does not start cold from 100 options.

A few practical rules the research supports:

Always give demographic context. The difference between a prompt with and without persona information is 50 percentage points of correlation attainment. That gap represents the full value of the method.

Do not use Direct Likert Rating. The distributional similarity gap between DLR (0.26) and SSR (0.88) means DLR cannot produce meaningful rankings. If every concept clusters around 3, the research is noise.

Use the free-text responses, not just the scores. The LLM reactions from varied demographic personas are often more actionable than the mapped scores, especially for surfacing pricing objections, positioning confusion, and brand trust issues before spending on human panels.

Treat gender and regional SSR outputs as directional until validated. If those dimensions drive variation in your category, run a calibration study against existing human data before relying on SSR for segmented conclusions.

And keep the workflow staged. SSR is a first screen. Human panels remain the ground truth, particularly for final go/no-go decisions on significant product investments.

A precise reading of what this shows

The original tweet describing this paper said "90% accuracy." The author corrected it: the metric is correlation attainment, not accuracy. That correction matters and is worth holding onto.

90% correlation attainment does not mean the LLM correctly predicted purchase decisions 90% of the time. It means the synthetic panel's scores correlated with human scores at 90% of the rate that real humans correlated with themselves on a retest. The comparison is not synthetic vs actual behavior. It is synthetic vs human survey consistency.

It also measures survey purchase intent specifically, not actual purchases. Whether stated intent predicts real sales is a separate question, and one where the research literature is mixed even for human surveys. SSR replicates the survey instrument; it cannot fix the gap between what people say they would buy and what they actually buy.

And it applies to categories with strong LLM training coverage. The researchers are clear that generalizability to other domains is an open question. Personal care products, packaged consumer goods, established product categories: solid ground. Genuinely novel technology products, niche B2B tools, markets with thin training signal: proceed with calibration first.

That said, 90% correlation attainment, zero-shot and outperforming a trained ML model, is a meaningful result for any team running concept testing in established consumer categories. The method is not a research department replacement. It is a screening tool that makes human research more targeted and less expensive to run.

The full paper is by Benjamin F. Maier and colleagues at PyMC Labs and Colgate-Palmolive, published October 2025. It is available at arxiv.org/abs/2510.08338.