Altamens

Altamens Investigación · Reproducible methods note

Same Accuracy, Different Certainty: Why Trial Count Matters

An 80% result can mean 8 correct answers out of 10 or 80 out of 100. The percentage is identical; the statistical precision is not.

Altamens9 min de lectura

Resumen

Percent correct is easy to read, but the denominator carries essential information. Eight correct answers out of ten and eighty out of one hundred both equal 80%. Under a simple binomial model, their 95% Wilson score intervals are approximately 49.0–94.3% and 71.1–86.7%, respectively.

This reproducible methods note explains that calculation, why the familiar normal or Wald interval is unreliable for small counts and extreme percentages, and why an interval around one session is not automatically a test of change between sessions. The examples are deterministic calculations, not observations of people, Altamens accounts or game telemetry.

Hallazgos clave

  • Always display the number correct and total eligible trials beside a percentage.
  • Equal percentages can have very different uncertainty: 8/10 and 80/100 are both 80%, but their model-based intervals differ substantially.
  • A one-session interval describes uncertainty under stated assumptions; it does not by itself prove improvement or make two sessions comparable.

1The denominator is part of the result

A percentage compresses two counts into one number. That is convenient, but it hides how many opportunities produced the result. A task with ten scored trials can only report accuracy in ten-percentage-point steps. A task with one hundred trials has a finer scale and, under the same statistical model, generally gives a more precise estimate of its underlying success probability.

The observed percentage is still the correct descriptive answer for that run. The uncertainty question is different: if eligible trials were independent attempts with a common probability of a correct response, which probabilities are reasonably compatible with the observed count? A confidence interval is one model-based answer.

2Worked example: four ways to obtain 80%

The figure uses the two-sided 95% Wilson score interval without a continuity correction. All four rows have 80% observed accuracy. Only the number of eligible trials changes.

Figura 1The same 80% point estimate at four trial counts
8 correct / 1080.00% (49.00–94.30)
16 correct / 2080.00% (58.40–91.90)
40 correct / 5080.00% (67.00–88.80)
80 correct / 100primary80.00% (71.10–86.70)
0%26%52%78%104%

More trials narrow the model-based interval when the observed percentage is held constant. Width is not a quality score and does not repair violations of the model.

Deterministic Wilson score intervals under a binomial model. These are teaching calculations, not observed players or thresholds for meaningful change.

For 8/10, the interval is about 49.0–94.3%, a width of 45.3 percentage points. For 80/100 it is about 71.1–86.7%, a width of 15.5 points. The longer run is not automatically fairer or more valid; it is more precise only for the probability defined by the model and scoring rule.

Download the worked examples CSV, standard-library calculator and methods and claim sheet. The files contain no participant or customer data.

3Why the familiar plus-or-minus formula is a poor default

A common Wald interval takes the observed proportion and adds or subtracts 1.96 estimated standard errors. Wilson's 1927 score interval instead inverts a score test and remains inside the meaningful 0–100% range.1

Newcombe compared seven interval methods for a single proportion across many parameter and sample-size combinations. The paper found poor coverage for the Wald method and supported score-based alternatives.2

Brown, Cai and DasGupta later documented that the Wald interval's coverage can remain erratic even at sample sizes people may assume are adequate. Their evaluation recommended the Wilson or Jeffreys interval for many practical uses rather than the standard Wald interval.3

Agresti and Coull studied a related adjusted-Wald approach that behaves like adding two successes and two failures for a 95% interval. That simple rule is useful pedagogically, but it is a different method from the Wilson interval used in our downloadable calculation.4

4A perfect percentage can still be imprecise

Five correct responses out of five and fifty out of fifty are both 100%. The 95% Wilson intervals are approximately 56.6–100% and 92.9–100%. The first result records a flawless short run; it does not establish that errors are nearly impossible.

The same point applies at zero correct. A zero or one hundred per cent result is not a reason to collapse uncertainty to zero. Score displays should avoid turning an endpoint percentage into certainty about future trials.

5The binomial model is a useful simplification, not a task audit

The calculation treats trials as independent and governed by one unchanged success probability. Mental tasks can violate both assumptions. Items may differ in difficulty; adaptive tasks deliberately change difficulty; learning and fatigue can create trends; responses from the same session can be serially related; omissions may be scored differently from errors.

  • Define an eligible trial, a correct response, an error, an omission and an exclusion before inspecting the result.
  • Keep the task version, difficulty policy, time limit, instructions and response device comparable.
  • Do not pool sessions automatically when the underlying probability may have changed.
  • Treat a narrow interval as precision under the model, not proof that the task measures the intended ability.

For a fixed-duration task, the number attempted is itself affected by speed. Accuracy conditional on attempted items and the total number correct answer different questions. Report both rather than letting a single percentage hide the speed–accuracy trade-off.

6One interval is not a test of improvement

Two session intervals overlapping does not prove equality, and non-overlap is not the general definition of a statistically supported difference. A comparison must reflect the design: whether the same items were repeated, whether responses can be paired, whether difficulty changed and which estimand matters.

For personal progress, start with a sequence plot of counts and percentages from comparable sessions. Mark task or device changes. Use a prespecified model for repeated observations if an inferential claim is needed; do not choose a method after seeing which one declares improvement.

7A seven-field record for an accuracy score

  1. Correct count.
  2. Eligible-trial count.
  3. Observed percentage.
  4. Errors, omissions and exclusions, each defined.
  5. Task/version, difficulty policy and time or opportunity window.
  6. If shown, confidence level, interval method and model assumptions.
  7. A comparison rule chosen before inspecting whether the score improved.

8Limitations, provenance and review status

This is a targeted narrative methods note, not a systematic review or a validation study. The worked rows are deterministic arithmetic selected to show how trial count changes interval width. They are not a sample, simulation of a population, minimum-trial recommendation or estimate of any Altamens game.

No Altamens user records, accounts, analytics or game telemetry were accessed. Literature search, drafting, calculations, source checks and publication preparation used an OpenAI automated assistant. The calculator and expected outputs received automated consistency checks, but the paper has no independent human methods review and is not journal peer reviewed.

Referencias

  1. 1.Wilson EB Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22(158), 209–212, 1927. Introduced the score interval used in the worked examples. DOI: 10.1080/01621459.1927.10502953.. https://www.tandfonline.com/doi/abs/10.1080/01621459.1927.10502953
  2. 2.Newcombe RG Two-sided confidence intervals for the single proportion: comparison of seven methods. Statistics in Medicine 17(8), 857–872, 1998. Comparative evaluation of seven methods. DOI: 10.1002/(SICI)1097-0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E.. https://pubmed.ncbi.nlm.nih.gov/9595616/
  3. 3.Brown LD, Cai TT, DasGupta A Interval Estimation for a Binomial Proportion. Statistical Science 16(2), 101–133, 2001. Coverage evaluation and recommendations. DOI: 10.1214/ss/1009213286.. https://projecteuclid.org/journals/statistical-science/volume-16/issue-2/Interval-Estimation-for-a-Binomial-Proportion/10.1214/ss/1009213286.full
  4. 4.Agresti A, Coull BA Approximate Is Better than ‘Exact’ for Interval Estimation of Binomial Proportions. The American Statistician 52(2), 119–126, 1998. Adjusted-Wald or plus-four method. DOI: 10.1080/00031305.1998.10480550.. https://www.tandfonline.com/doi/abs/10.1080/00031305.1998.10480550