Altamens

Altamens Investigación · Reproducible simulation · methods note

Why Personal Bests Rise Even When Performance Is Stable

A record is the most favourable observation so far. This simulation separates that selection effect from practice, learning and genuine change.

Altamens9 min de lectura

Resumen

A personal best can be motivating, but it is not a neutral summary of typical performance. It is selected from every attempt seen so far, so the expected record increases as the number of opportunities grows—even when the process generating each score does not change.

We simulated independent, standardised scores for synthetic profiles under three deliberately simple stable distributions. With normally distributed run-to-run variation, the mean best score after 20 attempts was 1.87 within-profile standard deviations above the unchanged centre; the mean of all 20 attempts stayed near zero. Tail shape changed the size of the record effect. These are model outputs, not observations of Altamens players or estimates of a training benefit.

Hallazgos clave

  • More attempts create more chances for an unusually favourable score, so records from different attempt counts are not directly comparable.
  • In the normal model, the expected best rose from 0.57 standard deviations after two attempts to 1.87 after 20 and 2.25 after 50, despite no underlying change.
  • Practice effects and learning can also change repeated scores, but this simulation excludes them; a rising record alone cannot distinguish these mechanisms.

1The question: what does a record measure?

A personal best answers a precise question: what is the most favourable recorded result among the attempts observed so far? It does not answer what a typical run looks like, whether the underlying performance level changed, or why the record occurred. Those questions require the full sequence and an appropriate comparison rule.

Selecting an extreme observation creates familiar interpretation problems. Regression to the mean concerns the tendency of an extreme measurement to be followed by a less extreme one when measurements contain variation; Barnett and colleagues explain why selecting on an extreme can create apparent change without an intervention.1

Personal bests can still be meaningful goals. A large observational study of online chess ratings found that records acted as reference points associated with effort and stopping decisions. That behavioural finding does not make a record an unbiased estimate of typical performance.2

2A deliberately stable model

For each synthetic profile, every attempt was an independent draw from a distribution with mean zero and standard deviation one. There was no learning, fatigue, changing difficulty, device effect or trend. The unit is therefore a within-profile standard deviation, not points, milliseconds or correct answers.

We generated 250,000 synthetic profiles for each of three distributions: standard normal; a bounded uniform distribution; and a heavier-tailed Student t distribution with five degrees of freedom, rescaled to standard deviation one. For each profile we retained the running maximum at 1, 2, 3, 5, 10, 20 and 50 attempts. A fixed seed makes the supplied code reproducible.

3The expected record rises with opportunity

Figura 1Mean personal best by number of attempts
NormalBounded uniformHeavy-tailed t(5)
-0.30.81.82.911121304050Normal · 1 Attempts observed: 0.004 SDNormal · 2 Attempts observed: 0.566 SDNormal · 3 Attempts observed: 0.847 SDNormal · 5 Attempts observed: 1.164 SDNormal · 10 Attempts observed: 1.539 SDNormal · 20 Attempts observed: 1.867 SDNormal · 50 Attempts observed: 2.25 SDBounded uniform · 1 Attempts observed: 0.002 SDBounded uniform · 2 Attempts observed: 0.58 SDBounded uniform · 3 Attempts observed: 0.866 SDBounded uniform · 5 Attempts observed: 1.155 SDBounded uniform · 10 Attempts observed: 1.416 SDBounded uniform · 20 Attempts observed: 1.567 SDBounded uniform · 50 Attempts observed: 1.664 SDHeavy-tailed t(5) · 1 Attempts observed: -0.001 SDHeavy-tailed t(5) · 2 Attempts observed: 0.536 SDHeavy-tailed t(5) · 3 Attempts observed: 0.804 SDHeavy-tailed t(5) · 5 Attempts observed: 1.123 SDHeavy-tailed t(5) · 10 Attempts observed: 1.55 SDHeavy-tailed t(5) · 20 Attempts observed: 1.992 SDHeavy-tailed t(5) · 50 Attempts observed: 2.62 SDAttempts observed
Mean personal best (within-profile SD)

Model output only. Lines connect the prespecified attempt counts for readability; they are not fitted growth curves.

Monte Carlo mean of the running maximum. Every generating distribution has an unchanged mean of 0 and standard deviation of 1. Differences between curves show why the score distribution matters.

Under the normal model, the mean personal best was 0.57 standard deviations above the stable centre after two attempts, 1.16 after five, 1.54 after ten, 1.87 after 20 and 2.25 after 50. The ordinary mean across all attempts remained within 0.004 standard deviations of zero at every checkpoint.

The bounded distribution approaches its ceiling, while the heavier-tailed distribution produces increasingly extreme records. That sensitivity is a feature, not an inconvenience: without describing the task's score range and run-to-run distribution, there is no universal conversion from number of attempts to expected record inflation.

4How to translate the standardised result

Suppose—only as a worked example—that comparable sessions for one task have a stable within-person standard deviation of two score points and are reasonably approximated by the normal model. The expected best after 20 attempts would then be about 1.87 × 2 = 3.74 points above that stable centre. This arithmetic is not a threshold for genuine change and should not be applied to bounded, skewed or changing tasks without checking the assumptions.

The required standard deviation is within-person variation under comparable conditions, not the spread between different people. Estimating it requires repeated ordinary sessions, a fixed task/version and a rule decided before the results are inspected. A best-ever value cannot supply its own uncertainty.

5Selection is not the same as practice

Real repeated testing can include familiarisation, strategy changes, fatigue and learning. Collie and colleagues tested 113 adults four times in one day and reported practice effects concentrated mainly between the first and second administrations of their computerized battery.3

White and colleagues tested 20 male undergraduates six times across three days and found that shifts and test–retest reliability differed by task and metric; some reaction-time measures were relatively stable, while other scores had poor reliability.4

A newer study of 149 healthy Swedish adults retested after one month likewise found that reliability and practice effects varied across digital test scores, with practice effects for tests having a major speed component but not for every measured domain.5

Those studies show why this simulation should not be used to subtract a generic record bonus from real data. An observed record can reflect selection, genuine task learning, changed conditions, measurement variation or several mechanisms together. The stable model isolates only the first.

6Best-of and mean-of answer different questions

An empirical motor-skill study of 24 football players compared first-attempt, best-of and mean-of scoring across ten juggling trials. Scores and rankings changed with the scoring rule, while mean-of summaries were more stable across trial counts than best-of summaries in that small task-specific sample.6

A best-of rule may be appropriate when the target is peak demonstrated performance under a fixed number of allowed attempts. A mean or median may be more appropriate when the target is typical performance. Neither is universally fair: the estimand, attempt count, familiarisation and handling of invalid trials should be specified before comparison.

7A practical progress-recording rule

  1. Choose the target first: peak performance, typical performance, consistency or speed–accuracy balance.
  2. Fix the opportunity window: compare best-of-five with best-of-five, not best-of-five with best-of-fifty.
  3. Keep the full run history. A record without the ordinary attempts hides the selection process.
  4. Use the same task version, scoring rule, device/input and summary where possible; label changed conditions.
  5. Decide familiarisation and exclusion rules before seeing which attempts are favourable.
  6. For typical performance, report a prespecified mean or median over a fixed recent window, alongside the number of runs.
  7. Describe an observed change before explaining it. A record alone does not identify practice, training, device effects or stable improvement.

Download the simulation code, complete results CSV and methods and claim sheet. The files contain no participant or customer data.

8Limitations, provenance and review status

Independent draws are a simplifying assumption. Real sequences can be autocorrelated, bounded, rounded, heteroscedastic, affected by learning or fatigue, and changed by question mix or device conditions. The simulated distributions are examples rather than fitted models. Monte Carlo estimates contain small sampling error; the CSV reports it. No minimum useful session count or reliable-change threshold is derived.

Altamens publishes this note and displays personal scores in its commercial product. No Altamens user records, accounts, analytics or game telemetry were accessed. Research, code, drafting, source checks and publication preparation used an OpenAI automated assistant. The code and outputs received automated consistency checks, but no independent human methods review or journal peer review.

Referencias

  1. 1.Barnett AG, van der Pols JC, Dobson AJ Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology 34(1), 215–220, 2005. DOI: 10.1093/ije/dyh299.. https://pubmed.ncbi.nlm.nih.gov/15333621/
  2. 2.Anderson A, Green E Personal bests as reference points. Proceedings of the National Academy of Sciences 115(8), 1772–1776, 2018. Observational chess-rating study; not a cognitive-training study.. https://pmc.ncbi.nlm.nih.gov/articles/PMC5828569/
  3. 3.Collie A, Maruff P, Darby DG, McStephen M The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test-retest intervals. Journal of the International Neuropsychological Society 9(3), 419–428, 2003. https://pubmed.ncbi.nlm.nih.gov/12666766/
  4. 4.White N, Forsyth B, Lee A, Machado L Repeated computerized cognitive testing: Performance shifts and test-retest reliability in healthy young adults. Psychological Assessment 30(4), 539–549, 2018. https://pubmed.ncbi.nlm.nih.gov/28557476/
  5. 5.Andersson M, et al. Test-retest reliability, practice effects and estimates of change: A study on the Mindmore digital cognitive assessment tool. Scandinavian Journal of Psychology, 2024. 149 healthy Swedish adults; author affiliations include Mindmore AB.. https://pubmed.ncbi.nlm.nih.gov/39072723/
  6. 6.Helgerud H, Ingvaldsen RP, Kaarstad K, et al. When Is a Test Score Fair for the Individual Who Is Being Tested? Effects of Different Scoring Procedures across Multiple Attempts When Testing a Motor Skill Task. Frontiers in Psychology 8:619, 2017. Small task-specific study of 24 football players; not evidence about Altamens games.. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2017.00619/full