Artul.ai Research LibraryStudy No. 90Flagship StudyUpdated 2026-09-17
Thinking Is Expensive, Deciding Is Not: 500 Questions per Earnings Call in One Second
By Artul.ai Research Group · n = 174,989 earnings calls · First published 2026-09-17
The study in 30 seconds: the stopwatch race, the bill, the calibration curve.Square MP4 or the 16:9 version.Abstract
Every earnings call on this site is scored by the same yes/no questions; the scorer was a reasoning LLM at about four minutes per 250 questions. We replaced it with a decision-only model (TypeSafe Jev 1.13) that returns a probability per question and no text: 500 questions in one call in 0.4–1.7 s, calibrated against the stored answers, about $0.001 per transcript, which let us build a 2,600-question battery in a day. Nothing here shows the questions predict returns: the strict out-of-sample test is at chance.
Key findings (all measured in-house, September 2026)
150–500× per call vs the run-log GLM figure (238 s) and union-alpha (217–320 s); 60–250× vs a fresh GLM call at low reasoning effort (89–108 s). Jev: 0.4–1.7 s for all 500 questions in one call.
7× more questions per call: ~1,750 with a 120k-character transcript, 2,800 with none, against 250.
1,579 calls/min in bursts (32 simultaneous in 1.2 s, ~0.8M questions/min) vs 4/min for the free model. Sustained: ~175 calls/min per key, an account limiter.
87.0% agreement with the stored GLM answers (108 transcripts × 500 questions, p ≥ 0.5); 94.1% when confident (67% of answers); 98.0% where GLM's own retest stability is ≥ 0.99.
Calibrated: p < 0.2 → GLM said yes 5.3%; 0.2–0.4 → 27.8%; 0.4–0.6 → 54.7%; 0.6–0.8 → 76.3%; p ≥ 0.8 → 93.3%.
$620 vs $8,700 (GLM, list price) for 2,600 questions × 175,000 transcripts; GPT-5 mini $45,000, Claude Haiku 4.5 $127,000, GPT-5 $228,000, Claude Sonnet 5 $254,000.
2,600 questions in one day: 3,431 → 2,692 (wording dedupe) → 2,341 (validated on 300 transcripts, 900 calls, 259 s) → 2,600 (fill round).
Benchmark, 3 transcripts: Claude Fable 5.1, Sonnet 5, Haiku 4.5, GPT-5, GPT-5 mini, GLM-5.3-Flash, same 250 questions. Jev fastest and cheapest by two to three orders of magnitude; agreement with each 76–85%, while they agree 84–92% with each other (Table 1, Figures 8–11). Spend $7.14.
Not proven to predict returns. On 5,820 labelled calls, 447 of 2,600 questions beat the shuffled-label null maximum in-sample; out of sample (top 30 chosen on 2017–2020, tested 2021–2025) the composite spreads −4.5% vs −5.6% for 30 random questions.
1Speed
Method. Jev timed at 100/250/500 questions, across 40 transcripts, and in bursts of 8/16/32 calls; the per-call ceiling is where the API returns 400 max_tokens_exceeded. union-alpha timed at concurrency 4–32. The GLM 238 s is active calls ÷ completions per second from the original run's log (667 steady-state minutes, IQR 226–254 s), not a stopwatch.
Figure 1. Five measured speed axes, log scale: per call, per transcript, questions per call, calls per minute in bursts, questions per minute. The GLM run had no burst test; its throughput came from ~1,200 concurrent calls.Figure 2. Seconds per scoring call, log scale; whiskers are the measured range. The free model's provider cuts every call at 317 s.Figure 3. Questions accepted per Jev call against transcript length: a ~65k input-token budget shared by transcript and questions. The corpus run sends two calls of 1,300.Figure 4. The live 2,600-question corpus run, calls per minute from its own log. This is the account limiter, not the model: bursts of 32 finish in 1.2 s.
2Cost
Method. Jev: measured input tokens × $0.042 per million, output free. Others: the measured GLM token profile (~15–18k in, ~9.5k out per 250-question call, ~8,000 of it hidden reasoning) scaled to 2,600 questions (11 calls) at each model's OpenRouter list price. Section 4 replaces these with billed usage.
Figure 5. Cost of asking 2,600 questions of every transcript in the corpus, log scale. The original 500-question GLM run cost on the order of $1,600; the same 500 on Jev cost about $170.
3Does it agree with the old scorer?
Method. Jev vs the stored GLM answers on the same transcripts and questions: 40 × 250 (85.8%) and 108 × 500 (87.0%), p ≥ 0.5; yes-rates 49.5% vs 49.8%. Calibration is the GLM yes-rate inside each Jev bucket. GLM's own retest agreement (91% at 250 per call) is the noise floor.
Figure 6. Jev probability bucket against the share of stored GLM yeses. Two-thirds of answers sit in the outer buckets, where agreement is 94.1%.Figure 7. Agreement with GLM grouped by GLM's own retest stability. Disagreement lives where the old answers were noise. 13 of 500 questions disagree by interpretation, e.g. "Does any analyst identify themselves with a two-part introduction (name and firm)?": Jev yes 94%, GLM 23%.
4Benchmark against frontier models
Method. Three real transcripts of ~60k characters (ZPTA (2024-05-15, 58,690 chars), HLLY (2024-05-12, 60,222 chars), MLYS (2024-05-11, 58,535 chars)) with stored GLM answers. Each model got the exact 250-question prompt of our earlier free-model test, temperature 0.3, max_tokens 16,000, one call per (model, transcript), via OpenRouter on 2026-09-17 (anthropic/claude-fable-5.1 is listed there and was used directly); Jev got the same 250 questions via TypeSafe. Cost is billed usage; agreement is identical yes/no answers, Jev at p ≥ 0.5. Calls that spent the cap on hidden reasoning and returned nothing were rerun once at 32,000; GLM was also rerun at reasoning effort: low, the corpus run's setting, which reproduces its ~8,000-token reasoning profile. n = 3 transcripts; agreement between models is not ground truth.
Table 1. Benchmark on 3 transcripts × 250 questions, max_tokens 16,000. Latency is mean (min–max) of the three calls; tokens are mean prompt / completion / reasoning per call; cost is mean billed cost per 250-question call. Agreement uses the 32k rerun where the 16k call returned no answers.
Model
Latency per call
Tokens in / out / reasoning
Cost per call
Answers parsed (of 250)
Agrees with Jev
Agrees with stored GLM
Jev 1.13
0.9 s (0.7–1.2)
19,030 / 4,960 / 0
$0.0008
250/250/250
—
83.9%
GLM-5.3-Flash (reasoning: low)
98.6 s (88.5–107.9)
17,134 / 9,269 / 8,242
$0.0068
250/250/250
83.9%
91.5%
GLM-5.3-Flash (default reasoning)
183.9 s (167.4–206.6)
17,134 / 16,000 / 14,934
$0.0063
0/0/0 (32k: 0/0/0, 415 s)
n/a
n/a
GPT-5 mini
175.7 s (153.2–194.4)
17,016 / 15,425 / 14,506
$0.0351
250/166/239 (32k: 250, 186 s)
78.2%
80.2%
Claude Haiku 4.5
5.5 s (5.3–5.6)
18,548 / 1,003 / 0
$0.0236
250/250/250
76.1%
71.2%
GPT-5
186.0 s (165.9–215.9)
17,016 / 15,732 / 15,061
$0.1786
232/0/250 (32k: 250/250, 197 s)
83.5%
86.3%
Claude Sonnet 5
151.5 s (148.8–153.7)
25,512 / 16,000 / 15,999
$0.2110
0/0/0 (32k: 250/0/250, 278 s)
80.4%
88.8%
Claude Fable 5.1
172.0 s (160.2–180.2)
25,514 / 15,323 / 14,489
$1.0213
250/0/250 (32k: 250, 212 s)
85.2%
91.5%
Figure 8. Seconds per call for the same 250 questions on the same 3 transcripts, log scale; whiskers are the range across transcripts.Figure 9. Seconds per 1,000 questions from a single call, i.e. sequential time per transcript.Figure 10. Measured cost per transcript at 500 questions, and the projection to 175,000 transcripts × 2,600 questions at the same per-call cost.Figure 11. Pairwise agreement on the 750 answers. The stored GLM row is the answer written during the original corpus run; the GLM-5.3-Flash row is a fresh call. Neither is ground truth.
52,600 questions in a day
Method. 90 personas (forensic linguist, bankruptcy trustee, air-traffic controller, jazz musician, beekeeper…) × 40 LLM-written questions, minus ~1,200 ever tried: 3,431. Wording dedupe (TF-IDF word + character n-grams, cosine ≥ 0.62): −739. The 2,692 plus the old 500 scored on 300 random transcripts with Jev (900 calls, 259 s): −372 outside 3–97% prevalence, −479 behavioural twins (|corr| ≥ 0.92, e.g. "Does the CEO answer the majority of analyst questions personally?" vs "…more questions than any other executive?", 0.94) → 2,341 (420 old + 1,921 new). Fill round, 40 fresh lenses: +259.
Figure 12. The build. Prevalence of the final set: 409 questions under 10%, 894 at 10–30%, 887 at 30–70%, 410 above 70%. Rare questions survive because a probability of 0.08 is still information.
6Early signal, and the test that failed
Method. First 5,820 labelled transcripts (12-month excess return over real SPY; cap ≥ $1B, ADV ≥ $3M; 41 months): per-question within-month Spearman IC vs the same with labels shuffled; era splits; partial IC after 14 price factors; strict test: top 30 by IC on 2017–2020, spread on 2021–2025.
Figure 13. |IC| across the 2,600 questions, real labels against shuffled. Real 95th/99th/max 0.075/0.110/0.136; null 0.028/0.037/0.048; 447 questions beyond the null maximum; 147 keep their sign in all three eras. In-sample.
Table 3. Strongest questions by |IC| on 5,820 labelled calls. Descriptive and in-sample; not a recommendation.
Question
IC
Detail
Does the word "strong" modify a financial metric at least three times?
+0.136
2017–20 +0.124 · 2021–22 +0.164 · 2023–25 +0.130
Is a buyback or dividend amount cited?
+0.133
—
Did a metric reach a record or all-time high?
+0.132
partial IC after 14 price factors +0.100
Is a "pivotal trial" mentioned?
−0.119
—
Is a trial described as "on track"?
−0.116
—
Out of sample, the top-30 composite spread was −4.5%. Thirty random questions gave −5.6%. At this sample size the selected questions are indistinguishable from chance.
7Limitations
The questions are not shown to predict returns. The in-sample tail is real; the out-of-sample composite (−4.5% vs −5.6% random) is at chance. The test is repeated at 50,000+ labelled calls with the rule fixed in advance. Our best earlier pattern passed every in-sample control and lost 12.4 points to SPY live: Your AI Backtest Is Lying to You.
Accuracy is measured against other models, none checked against human labels at scale; GLM disagrees with itself 7–9% of the time.
The 238 s GLM figure is implied, not timed, and it was measured under ~1,200 concurrent calls. A fresh single call at low reasoning effort takes 89–108 s (Table 1); at OpenRouter's default reasoning the same prompt spent the whole 16,000- and 32,000-token caps thinking and returned no answers. Speedups quoted against GLM depend on which of those you take.
The benchmark is 3 transcripts and one day's prices; reasoning latency varies with load and with how much the model chooses to think; projections assume no batch discounts.
Sustained throughput is an account limit, not the model: 4–6 concurrent per key; one transient HTTP 402 after 1,149 calls.
Scripts, for the record: typesafe_test.py, ts_stab.py, union_parallel.py, union_compact.py, gen2600.py, validate2600.py, reselect2600.py, gen_fill.py, corpus2600.py, early2600.py, early2600b.py, bench.py. Every figure above is also available as PNG in light and dark variants under /research-media/deciding-is-not/. Prices are TypeSafe and OpenRouter list prices on 2026-09-17 and will change.
Cite this study
Artul.ai Research Group (2026). “Thinking Is Expensive, Deciding Is Not: 500 Questions per Earnings Call in One Second.” Artul.ai Earnings-Call Research Library, Study No. 90. https://artul.ai/research/thinking-is-expensive-deciding-is-not