How Much Can the Question Mix Change a 30-Second Mental-Math Score?

1The measurement question

A 30-second test has no room to average over hundreds of items. A handful of unusually slow questions can consume a large share of the available time. That means a correct-count score can reflect both a person's performance and the particular questions selected for that attempt.

The central question is narrow and answerable: in a fixed 30-second arithmetic sprint, how much can item composition alone change a person's correct-count score — and can difficulty-balanced forms reduce that form-to-form variation?

Answering it honestly requires separating four ideas that a single reported score collapses together:

1. The person's performance — their speed and accuracy on that occasion.
2. The questions shown — operation, operand size, truth status, answer plausibility and calibrated item difficulty.
3. The test conditions — time limit, order, interface, device, latency, distractions and practice.
4. The reported score — a correct count and accuracy generated by all three factors above.

2Why arithmetic questions differ in difficulty

The best-replicated reason is the problem-size effect: larger arithmetic problems reliably take longer to answer and produce more errors than smaller ones, a finding studied and modelled for over three decades.³ Our reanalysis reproduced the source paper's result exactly to rounding: for addition, the median participant's Kendall tau between problem size and response time was 0.288, positive for all 59 participants (group t(58) = 18.80); for subtraction, the median tau was 0.190, positive for 58 of 59 (group t(58) = 14.51).¹

• Operation — subtraction items were slower than addition items in this dataset (1.423 s vs 1.237 s on primary trials); the two are not interchangeable in speed or error profile.
• True/false decision demands — verifying a displayed answer engages different processes than producing one; here, false equations were slower than true equations (1.392 s vs 1.267 s) and slightly less accurate (96.92% vs 97.48%).⁴
• Speed–accuracy tradeoff — a person can convert accuracy into attempted items and back; correct count alone hides which strategy they used.
• Order, practice and fatigue — where an item lands in a session affects the response it gets.

Because items differ this much, a short form containing more subtraction items or larger operands is measurably slower than another form of equal duration — which sets up the practical question this study answers: did the person's performance change, or did the form change?

3An independent open dataset, audited

We used the arithmetic-verification task from Bye, Harsch and Varma (2022), an independently collected, peer-reviewed dataset released openly under CC BY 4.0 on OSF.^1,2 Participants judged whether single-digit addition and subtraction equations were true or false, responding as quickly and accurately as possible. False answers generally differed from the correct answer by 2, with specific exceptions to avoid negative results or trivially rejectable values.

The raw archive was audited before any modelling. Sixty per-participant files were present, each with 256 rows; participant identifiers 27 and 39 were absent, consistent with the two pilot-repeat exclusions already removed from the public archive by the source authors.

Privacy handling was fixed before ingestion: the participant date field was dropped immediately; survey, ACT/SAT, health, admissions and demographic fields were never used; and raw files stayed in a restricted analysis workspace. Published outputs are aggregate tables and derived per-participant summaries keyed to the source study's already-public identifiers — no raw trial rows are republished.

4The frozen hypothesis

Primary hypothesis (fixed before analysis): naively sampled forms containing more large-operand, subtraction, or otherwise slower and more error-prone items will produce fewer correct responses and greater within-person form-to-form score variation than forms balanced using item difficulty estimated only from training participants.

The protocol committed to publishing a null or contrary result, and listed in advance the ways balancing could fail: unstable individual difficulty rankings, a too-small item pool, person-level inconsistency dominating item mix, missing difficulty variables, or an insensitive 30-second simulation. As section 6 shows, the variance-reduction part of the hypothesis was supported at roughly 10–13%; the mean-score part was not meaningfully supported — random and balanced forms produced nearly identical average scores.

5Simulating thousands of 30-second forms

The simulation replays real observed behaviour into a time budget. In plain English:

1. Participants were split using five repeated five-fold cross-validation runs, and item difficulty was estimated only from training participants — a held-out participant's own data never shaped the difficulty used to build their forms.
2. The difficulty score combined participant-centred log response time with a smoothed item error rate, and items were ranked into difficulty quartiles within operation × displayed truth status.
3. Each held-out participant received 1,000 simulated forms per condition: fully random, operation/truth quota-balanced, and fully difficulty-balanced (locally balanced blocks spanning operation, truth status and difficulty strata — the stratified-assignment logic studied in randomly-equivalent-forms research).
4. Observed response times were accumulated until the 30-second limit: an item counts only if its response finishes in time; correct and incorrect responses both consume time; only correct responses score.
5. Confidence intervals resampled participants 10,000 times (nonparametric bootstrap, preserving participant clustering).

The quota-balanced comparator is the stronger test: it preserves broad source-task proportions (operation, truth status) without balancing calibrated difficulty, so the comparison isolates whether balancing predicted difficulty adds value beyond simple content quotas.⁵ This remains a simulation using independently collected true/false arithmetic-verification trials — not a direct reliability study of the Altamens four-choice mobile test.

6Results: balancing cut form-to-form variance by about 10–13%

Primary result: difficulty-balanced forms reduced the mean within-person variance of simulated 30-second correct-count scores by 13.14% compared with fully random forms (participant-bootstrap 95% CI 11.08% to 15.10%). The RMS within-person standard deviation fell from 1.99 to 1.85 correct answers. Because variance is the square of standard deviation, the headline variance reduction corresponds to a smaller 6.8% reduction in SD.

Simple operation/truth quotas achieved only a 4.60% variance reduction (RMS SD 1.94); balancing calibrated difficulty added 8.95 percentage points beyond quotas. Mean scores were essentially unchanged — 22.76 correct under random forms, 22.75 under quotas, 22.66 under difficulty balancing — and the small −0.11-point primary shift shrank to −0.01 when all finite response times were retained, indicating a cutoff artefact rather than a scoring penalty.

What a participant would actually notice: balancing helped, but substantial short-test variability remained. Across pairs of independently simulated 30-second forms, the median participant's 5th-to-95th percentile score width stayed at 6 points in both conditions.

7Safeguards that held the analysis honest

• Item difficulty was two-dimensional — the calibration combined participant-centred log response time and a smoothed item error rate, rather than collapsing to a single composite blind to errors. An RT-only sensitivity variant is reported separately (17.51%) rather than being folded into the primary claim.
• Cross-validation split by participant, never by trial — five repeated five-fold participant-level runs; a held-out participant's own performance never shaped the difficulty used to build their forms.
• Prespecified exclusions — the primary analysis kept finite positive response times at or below 4 seconds (correct and incorrect, because both consume sprint time); the all-finite sensitivity analysis repeated everything without the cutoff and is quoted alongside the primary result.
• Uncertainty — participant-level nonparametric bootstrap with 10,000 resamples, preserving participant clustering, with 95% intervals reported for every headline estimate.
• Conservative language — the term “reliability improvement” is not used, because no recognised reliability coefficient was estimated; the claim is a variance reduction in simulated forms.

8What we do and do not claim

The claims boundaries were fixed before results existed, in line with good testing practice.⁶ With the numbers now attached, the supported claims are:

• In this independent dataset and simulation, the mix of arithmetic questions materially changed short correct-count scores.
• The tested balancing procedure reduced form-to-form variance by about 10–13% in the most defensible specifications.
• Balancing improved comparability but did not make short scores perfectly stable.
• Larger/slower arithmetic items and operation/truth composition are relevant form-design variables.

9What this means for a 30-second Altamens score

The practical lesson is not that a 30-second score is meaningless — it is that it should be read as an informal performance snapshot with a real form-to-form component. Even under difficulty balancing, two simulated forms differed by 2+ points more than half the time. Repeating the test under similar conditions, and comparing trends rather than single attempts, remains the honest way to use a short score. The source task differs from Altamens, so these numerical estimates are not transferred as Altamens norms or error bands.

10Limitations

1. The sample contains only 59 US undergraduates aged 18–23.
2. The source task is laboratory-based true/false verification with keyboard responses; Altamens uses four answer choices on different devices.
3. Source response times do not include measured Altamens device, rendering, touch or transition latency.
4. The 30-second forms are simulations, not prospectively administered alternate forms.
5. Each simulated form reuses one observed response per participant-item, so it cannot model day-to-day state, repeated-exposure learning, fresh response noise, mobile latency or four-choice guessing.
6. The 4-second cutoff and other preprocessing choices affect estimates — which is why the all-finite specification (9.98%) is quoted next to the primary result (13.14%).
7. Item difficulty can vary between people; a group-calibrated “hard” item is not equally hard for everyone.
8. The balancing algorithm is transparent but not necessarily optimal; better algorithms could do more, or the ceiling may be low.
9. The item pool is limited to the source study's addition/subtraction design.
10. The analysis cannot produce national, age, education, clinical or IQ norms, and does not validate the Altamens test as a medical, educational or psychometric instrument.
11. The study was protocol-guided, with the design frozen before analysis, but it was not formally preregistered in a public registry before execution.

References

1. 1.Bye JK, Harsch RM, Varma S Decoding Fact Fluency and Strategy Flexibility in Solving One-Step Algebra Problems: An Individual Differences Analysis. Journal of Numerical Cognition, 8(2), 281–294, 2022. Source paper for the arithmetic-verification dataset. https://doi.org/10.5964/jnc.7093
2. 2.Bye JK, Harsch RM, Varma S Open Science Framework project: data, codebooks and analysis files. OSF, 2022. CC BY 4.0; raw files at osf.io/j79a6, codebooks at osf.io/kgd4c, analysis/data component at osf.io/zmn8a. https://osf.io/vx2rh/
3. 3.Ashcraft MH, Guillaume MM Mathematical Cognition and the Problem Size Effect. Psychology of Learning and Motivation, Vol. 51, 121–151, 2009. https://doi.org/10.1016/S0079-7421(09)51004-3
4. 4.Faulkenberry TJ A Single-Boundary Accumulator Model of Response Times in an Addition Verification Task. Frontiers in Psychology 8:1225, 2017. https://doi.org/10.3389/fpsyg.2017.01225
5. 5.Liao C-W, Livingston SA Examining an Alternative to Score Equating: A Randomly Equivalent Forms Approach. ETS Research Report RR-08-14, 2008. Stratified random assignment on content and predicted difficulty. https://doi.org/10.1002/j.2333-8504.2008.tb02100.x
6. 6.American Educational Research Association, American Psychological Association, National Council on Measurement in Education Standards for Educational and Psychological Testing. AERA, 2014. https://www.testingstandards.net/