What twenty answers can and cannot tell you

This example is made up to show how to read a result. No poll produced these numbers.

The question

A city has money for one transport project. Which should it fund?

• Improve bus frequency
• Expand cycling lanes
• Repair existing roads
• There isn't enough information to recommend one

Three models are asked, in two wordings, ten times per wording. Each call sees the options in a fresh order, and the order is kept with the reply.

Reading one model's result

Suppose one model picks "improve bus frequency" in 14 of its 20 readable replies. That is 70%, and the 95% range runs from 48% to 85%. In plain words: a model that "really" picks buses anywhere from about half the time to most of the time could have produced this result. Twenty answers tell you a lot less than "70%" sounds like.

With 200 answers, the same 70% would run from 63% to 76%. More answers buy precision, at a price you can see before you run.

It also tells you nothing about what residents want. It describes what this model said, to this question, through its API, that day.

Three questions to ask of any result

1. What was left out? If six more calls were refused or unreadable, the 70% is 14 of 26 attempts, and the refusals are part of the story.
2. Does the wording matter? Compare the two wordings before trusting the combined figure.
3. Did it change, or did the question? Compare the same version across runs. An edited poll starts a new series on purpose.

What stays with the result

The prompt each model received, the order the options were shown in, the settings, the model that answered and its raw reply, for every one of the 60 calls, ready to download. A reader who doubts the 70% can count the answers.

Build your own poll · Read the methodology · Read a result like an editor