Repeated AI model observations you can report
Models change between versions, answer differently each time they are asked, and give different answers to the same question in a different format. Studying any of that means asking the same thing, the same way, many times, and keeping everything. Mind Polling does that, without you running the infrastructure.
What the literature already shows
- Answers drift with versions. A 36-week study of political answers found a small drift that often coincided with new version releases (Nogueira et al., "Changing Tunes", ICML 2026).
- Format changes the answer. The best systems answered news questions correctly over 90% of the time as multiple choice, and lost 11 to 13% under free-response evaluation, in the paper's own terms (Suzgun et al.).
- Single runs are noise. Measurement of AI answers "must be reported with uncertainty estimates" (Sielinski).
These motivate the design. None of them validates it; that is what your study is for.
What you get
- Wording and time kept apart. Several wordings per question, and a new version whenever the poll changes, so phrasing effects and change over time are separate comparisons.
- The model that answered, beside the one you asked for, on every call.
- Every call, as CSV or JSON: prompt and hash, option order, settings, raw reply, outcome, tokens and charge. The charts are summaries; the export is the data.
- Outcomes counted, not dropped: refusals, unreadable replies, cut-offs and errors, so your denominator is honest.
What to plan before you scale
Decide what difference would matter, how you will treat a refusal, and which comparisons you will report. Samples from one model are not independent people, and the built-in ranges do not correct for dependence or for many comparisons at once. A pilot run tells you a model's format compliance, cost and refusal rate before you commit a budget.
If you study synthetic respondents
Mind Polling does not simulate people. But studies that ask whether models can stand in for survey respondents need exactly what it records: repeated samples, fixed settings and version boundaries.