Ask a language model to play a Slovak pensioner. Then four hundred of them, with names and ages and jobs, and put a real survey in front of the result. People are doing this now, in production, and it half works.
In a large preregistered evaluation across 70 survey experiments, treatment effects estimated from LLM-simulated samples correlated with the real effects about as strongly as pooled human forecasters did — and the correlation held up for studies published after the model’s training cutoff (Ashokkumar et al., 2026). That is a genuinely surprising result. The same evaluation also found that the simulated effects were systematically too large. Separate audits find synthetic respondents breaking down on exactly the subgroup structure that most inference depends on (Bisbee et al., 2024).
Strong headline correlation, systematic distributional bias. That combination is not a mystery — it is the signature of a method whose statistical foundations have never been written down. Correlation with held-out outcomes tells you a synthetic population carries signal. It is silent on the four questions a survey statistician asks first:
- What population quantity is the engine actually estimating?
- What should we compare its error against, given that a finite run of a perfect engine still misses any finite target?
- When one engine has to reproduce many surveys at once and no configuration fits them all, what does that failure mean?
- When a prediction is wrong, is the fault in who the synthetic people are, or in how they answer?
Our paper answers those four. This post is the tour — the same argument with fewer symbols and more moving parts. The proofs, the estimator theory, and the worked derivations live in the PDF.
A person is a distribution, not an answer
Start with the thing everyone gets wrong, including the tooling.
A survey respondent is not a fixed answer waiting to be read out. Ask the same person the same question on Tuesday and Thursday and you will often get different answers — mood, framing, genuine ambivalence. So the honest object attached to a person is not “4 out of 5” but a distribution over the answer options. One per question.
A population, in turn, is not a list of people. It is a distribution over people. And what a survey publishes is neither of those things: it is the average of everyone’s response distribution, one number per answer option. Statisticians call it the marginal. It is a shadow of the population, and it is the only thing you ever get to see.
A population, and the single shadow it casts
Table view
| Answer option | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Population marginal | 11.5% | 21.9% | 33.2% | 21.9% | 11.5% |
Each person on the left is a response law — a spread over the five options, driven by where they sit on a latent trait. Draw three more and watch them change completely. Now watch the right-hand panel, which is what the survey prints: it barely moves. That stability is what makes surveys useful, and it is also the entire problem. Enormously different populations cast almost the same shadow.
Building a synthetic population is therefore an inverse problem: reconstruct a distribution over people from a handful of its projections. That is the same mathematical situation as tomography or deconvolution, and it comes with the same warning label — the reconstruction is not unique, and it is not stable. Every useful thing in the rest of this post follows from taking that sentence seriously.
The formal version
A population is a probability measure over latent individual states. Each individual carries a family of parametric response kernels — one per question — whose parameters are driven by a low-dimensional latent trait, with answers conditionally independent given that trait. The population observable is the kernel integrated over the measure. This is item response theory (Lord & Novick, 1968; Rasch, 1960; Reckase, 2009) embedded in a de Finetti mixture (de Finetti, 1937): exchangeability forces the mixture structure, so the modelling choices are the trait dimension and the kernel family, not whether to mix at all. See Sections 2–4 of the paper.
There is a pleasing wrinkle when the engine is a language model: the formalism is not a metaphor for the machinery, it is a description of it. A transformer emits logits that are a linear readout of its internal state, passed through a softmax with a temperature — which is exactly a parametric response law whose parameters are linear in a latent state, with temperature governing how much the same “person” scatters when asked twice. A persona queried at temperature zero is a degenerate response law: a person who is a fixed answer. Hold that thought; it comes back with a vengeance.
Calibration is fitting the shadow
If the survey shows you marginals, then fitting a synthetic population means turning whatever dials the engine has until its marginals line up with the real ones. Formally that is a minimum-distance estimator, and when the discrepancy is quadratic it is exactly the generalized method of moments (Hansen, 1982; Wolfowitz, 1957) — well-understood machinery.
Two choices inside it look like implementation details and are not.
The weights. Questions do not deserve equal say. The efficient weighting is the inverse variance of each moment residual; uniform weights quietly over-weight high-entropy questions.
The discrepancy. The standard goodness-of-fit family (Read & Cressie, 1988) treats answer options as an unordered set. For a Likert item that is wrong: predicting “4” when the truth is “5” should cost less than predicting “1”. Ordered and numeric questions want a transport metric — a Wasserstein distance (Villani, 2009) or a kernel MMD (Gretton et al., 2012) — that respects the distance between categories.
And one warning, which the rest of the post keeps earning: matching the data you fitted is not prediction. Because the inverse problem is ill-posed, many different synthetic populations hit the same marginals and disagree about everything else. Credit is only earned on instruments held out of the fit.
Even a perfect engine gets a nonzero score
Here is the result we most wish were common knowledge.
Suppose your synthetic engine is flawless — its true population marginal is exactly the real one. You run 1,000 synthetic respondents. You compare against a survey topline from 800 real respondents. The two will not agree. Both are finite samples, both are noisy, and the discrepancy between them has a computable, strictly positive expected value.
We call it the sampling floor. It is not a fudge factor or a tolerance someone chose. It is the expected score of perfection.
What perfection scores
The curve is the distribution of scores a perfect engine posts, over repeated runs. Zero is off to the left, and it is unreachable. Drag either sample size: the whole distribution slides, shrinking like one over the square root of the sample size, but it never arrives at zero. The dashed teal line is the mistake almost everyone makes — computing the floor from your own run size alone, as though the survey topline were exact. At these sizes that understates the bar by roughly half, which makes a correct engine look deficient.
The two-sample point matters more than it sounds. If you treat the published topline as ground truth rather than as an estimate with its own noise, you set a bar nobody can clear and then penalise your engine for failing to out-predict the survey’s own sampling error.
The formal version
Under the null that the engine’s population marginal equals the truth, the run’s empirical marginal is multinomial around it. Applying the multinomial CLT and the folded-normal mean gives the floor in closed form; comparing two independent samples of sizes N and n replaces the variance factor with the sum of their reciprocals. Ordinal and scale channels give the same statement on CDF increments and on the standard error of the mean. Verified against a 2 × 105-replicate Monte Carlo to four decimals. Section 6, Proposition 6.1.
A score that flatters, and a test that does not
Once you have a floor and a zero-information reference — what you would score by guessing uniformly — you can normalise. Skill of 1 means “at the floor, indistinguishable from perfect”. Skill of 0 means “no better than knowing nothing”. It is a nice, readable number.
It is also too forgiving, and here is the demonstration.
A solid-looking score that fails the test
Drag the bias. At full bias the engine’s answer distribution is visibly wrong — it over-claims the bottom two options and under-claims the middle — yet it still posts a skill score of about 0.7, which reads like a solid B. The verdict axis tells the truth: repeated runs land inside the band a perfect engine would occupy only about one time in six. Same engine, same data, two very different conclusions.
The fix is to stop reading skill as a grade and start using the floor’s sampling distribution rather than just its mean. Draw many replicate runs under the assumption that the engine is perfect, take the middle 95% of the resulting scores, and you have a null band. An engine is “indistinguishable from perfect” exactly when its score lands inside that band. This is an ordinary parametric-bootstrap goodness-of-fit test (Read & Cressie, 1988) — nothing exotic — and it is strictly more informative than the scalar.
Rule: gate acceptance on band membership, not on a skill threshold.
One population, many surveys
A population is one object. It gets probed by many surveys, each with its own questions, houses, modes, and dates. A single calibrated engine has to answer to all of them.
There are two honest ways to combine those objectives, and choosing between them is not cosmetic — it fixes what your calibration loop is allowed to do.
You can pool: minimise a weighted average of per-study error. Smooth, easy to optimise, and perfectly happy to wreck one study to buy a bigger gain on another. Or you can constrain: require every study to be matched to within its own sampling floor. Nothing gets sacrificed, and the tolerance is principled rather than chosen — it is the bar from Figure 2.
The constrained version can fail to have any solution at all. That is the interesting part.
When no population explains all your surveys
Each region is the set of population configurations that match one survey to within its own noise. Toggle to surveys disagree: the regions come apart, there is no configuration that satisfies all three, and the pooled compromise — the thing an averaged objective would hand you — sits outside every one of them. It is not a numerical failure to be weighted away. It is the data telling you about house effects (Jackman, 2005), mode or wording effects, temporal drift, or a broken independence assumption (Holland & Wainer, 1993).
The same machinery scores genuine prediction. Fit on all but one instrument, then ask whether the held-out study lands inside its null band. That is the formal version of “credit is earned on data you didn’t fit”. Note how much stricter it is than reporting a correlation between predicted and observed effects across studies: the correlation lives on the study axis, while the band test asks for distributional agreement, study by study.
Two populations you cannot tell apart
Why is any of this hard? Because marginals genuinely do not determine a population, and the failure is not subtle.
Identical on everything a survey can see
Table view
| Observable | Population A | Population B |
|---|---|---|
| 1 item = yes | 0.3333 | 0.3333 |
| 2 items, both yes | 0.1667 | 0.1667 |
| 3 items, all yes | 0.1000 | 0.0833 |
Population A is a smooth spread of opinion. Population B has just two kinds of people: two-thirds who are coin-flippers and one-third who never say yes. These are not similar populations. Yet every one-item and every two-item observable they produce is exactly the same number — no survey of that shape can distinguish them. They part company only at the third item, and by 17%. Which one you pick is a modelling act, not an inference.
Three consequences follow, and they are the reason the field’s current practice feels unsteady.
The data confine the population to a set, not a point — the honest report is a set-valued one (Imbens & Manski, 2004; Manski, 2003). Regularization is therefore mandatory rather than optional: a low-dimensional trait, a prior, and the conditional independence assumption are what make the reconstruction tractable at all, and they are assumptions, not free lunches. (Applied population synthesis has always known this — iterative proportional fitting (Deming & Stephan, 1940) and MRP (Gelman & Little, 1997; Park et al., 2004) are exactly regularized reconstructions from margins.) And, most practically: calibration can succeed while the joint structure fails. You can match every marginal and still have the cross-question structure badly wrong, which is precisely where two calibration-equivalent engines will disagree about anything you did not measure.
Where the error actually lives
When a synthetic prediction misses, there are exactly two places the error can be: the composition of the population — who is in it — or the response mechanism — how they answer. Nowhere else. That is not a heuristic; it is the law of total variance, which is also the reliability decomposition of classical test theory (Cronbach et al., 1972).
The two require completely different fixes, and confusing them is expensive.
Two dials, and the effect sizes they distort
Turn the temperature to zero and the top strip collapses: every person becomes a fixed answer. Total variance shrinks, and since standardized effects divide by it, the reported effect size inflates — with a true contrast held fixed the whole time. Narrow the spread of people instead and you get the same inflation by a different route. This is a live diagnosis for the systematic effect-size overestimation reported in current evaluations (Ashokkumar et al., 2026): the decomposition, computed question by question, says which mechanism is operating.
An LLM persona queried at temperature zero is exactly the pathology on the left. It looks like a tidy engineering choice — deterministic, reproducible, easy to cache — and it silently breaks every standardized quantity downstream.
Does it survive contact with a real system?
The formalism would be cheap if nothing implemented it. Mantle is our production engine for building, interrogating, calibrating, and auditing synthetic populations, and it is organised component-for-component around the objects above: the population is a stored, inspectable roster rather than a side effect of prompt wording; channels are sampled at positive temperature on purpose; a survey enters the system as toplines plus its sample size, because the floor needs it; calibration is a loop whose accept rule is the constrained reading of Figure 4 in miniature — a move must improve the fitted questions without regressing the untouched ones.
We ran it end to end against a 2025 Ipsos study of Slovak attitudes to artificial intelligence — twelve binary toplines, 1,000 respondents, a population whose headline finding is ambivalence: high adoption and high worry at once. The synthetic population was 400 Slovak personas.
The uncalibrated engine was not merely wrong, it was coherently wrong. It over-claimed worry by 42 points on concern, 39 on privacy, 34 on danger, while matching the adoption questions almost exactly. It was playing an alarmed European where the survey shows an ambivalent one — a response-layer miscalibration, which is a diagnosis, not a shrug.
Before and after, against the only two anchors that matter
Table view
| Quantity | Mean divergence |
|---|---|
| Two-sample sampling floor (N = 400, n = 1,000) | 0.024 |
| Uncalibrated engine | 0.181 |
| Zero-information anchor | 0.100 |
| After calibration, fresh regeneration | 0.106 |
Five conditioning moves were scored, two accepted, three reverted by the collateral guard. The accepted configuration was then used to regenerate all 400 individuals from scratch — a fresh run the loop never saw — and mean divergence fell from 0.181 to 0.106, a 41% reduction, with seven of twelve questions within 0.08 of the topline. Note where 0.106 sits: still about four times the floor. Attuned, not indistinguishable from perfect.
Two honest observations. The gap between the in-loop estimate (0.066 on the held-aside half) and the fresh run (0.106) is the generalization cost of the accepted moves — real, and worth reporting. And the pooled skill of the fresh run came out at about −0.07, which sounds like failure and is not: five of the twelve targets sit within ten points of a coin flip, one of them exactly at 50/50, where the zero-information anchor is zero and the ratio degenerates. The per-question divergences against per-question floors are the informative report. That the pooled anchor can be nearly degenerate is itself a finding about the anchor.
Then the diagnostic that the whole theory says to go looking for.
Four hundred people, fifty-one opinions
The calibrated population’s marginals are near their targets. Its joint is not: the 400 individuals occupy only 51 distinct answer profiles. This is the non-identification result made empirical — calibration constrained the projections, and the directions nobody was looking at drifted toward degeneracy. It is also the temperature-zero pathology in softened form. (Cluster shapes are illustrative; the two counts are measured.)
The architectural point is that the system surfaces this. Profile multiplicity and variance components are standing diagnostics, so the collapse is visible before anyone makes a downstream claim from the run — rather than being discovered post hoc by an auditor (Bisbee et al., 2024).
By our own standard, that study is calibration, not prediction. No held-out instrument was scored, so no predictive credit is claimed. Running the held-out protocol at scale and in public is the empirical program this formalism exists to score, and it is what we are building toward.
Five rules we would like the field to adopt
- Report skill against the two-sample floor. A finite target cannot be matched better than its own noise. Floors computed from your run size alone understate the bar and make correct engines look broken.
- Gate on the null band, not a skill threshold. A meaningfully biased engine posted about 0.7 skill and was still rejected roughly five times out of six.
- Calibrate to many instruments with per-study constraints, and treat infeasibility as a result. It is telling you about house effects, mode effects, drift, or a broken assumption.
- Evaluate distributionally on held-out instruments. Calibration-equivalent engines are guaranteed to exist and to disagree wherever you did not look.
- Keep the response channels stochastic. Temperature zero collapses within-person variance and inflates every standardized contrast downstream.
What we are not claiming
Three idealizations bound all of this. Real surveys are stratified, clustered and weighted rather than i.i.d.; the design-effect correction handles the floor, but we do not attempt informative nonresponse. Conditional independence given a low-dimensional trait is a regularizing fiction — we turn its failure into a detectable signal, but we do not yet estimate richer dependence (testlets, context and question-order effects) inside the calibration itself. And we position the estimator within minimum-distance and GMM theory without deriving consistency and asymptotic distribution under partial identification; set-valued reporting with valid coverage remains to be worked out for this problem class.
The closing point is the one we think the field most needs stated plainly. A synthetic population is not validated by fitting the data it was built from. It is validated by matching instruments it never saw, to within their own sampling noise, distribution by distribution.
Credit is earned on held-out instruments. Everything else is curve fitting with extra steps.