Tools chosen: kalshi_direct claude_news claude_news kalshi_related polymarket_related wikipedia

Sub-questions (Fermi decomposition)

Planner reasoning

This is about AI benchmark progress on Humanity's Last Exam (HLE) by end of 2027. Need to anchor on Kalshi price, find current SOTA on HLE, and check related AI benchmark markets for cross-signal.

# Event Will any LLM score at least 70% on Humanity's Last Exam before Dec 31, 2027? # Outcomes to forecast - **Yes**: At least one LLM achieves ≥70% on HLE before Dec 31, 2027 - **No**: No LLM reaches 70% by that date # Kalshi market anchor **Current YES price: 46.00%** — sharp recent decline (-23% over 7 days, -22% over 30 days), suggesting a significant negative update (likely prompted by new benchmark data or model release results falling short of expectations). Price range over 8 days: 41–69%. Average daily volume: 346 contracts. Market is actively traded with meaningful liquidity. # Sub-question answers 1. **Current Kalshi price?** — 46.00% YES, recently dropped ~23 pp in a week. [Kalshi direct] 2. **Current SOTA on HLE?** — Claude Mythos Preview leads at **64.7% with tools** (April 2026 release, restricted access due to cybersecurity concerns); best publicly-available models score ~57–58%. [BenchLM.ai, LLM Stats, NxCode] 3. **Rate of progress?** — Explosive: <10% (Jan 2025) → ~44–50% (Jul 2025, Grok 4) → ~58–65% (Apr–Jun 2026). ~55 pp gain in ~17 months. However, pace appears to be slowing as the benchmark hardens near the human-expert ceiling. [xAI, NxCode, Artificial Analysis] 4. **Related prediction markets?** — No Polymarket markets found. Only the same KXLASTEXAM-27DEC31-T70 Kalshi market found. [Polymarket scan, Kalshi related] 5. **Models expected before 2027 and scaling trends?** — Multiple frontier model generations expected (GPT-6+, Claude Mythos GA or successor, Gemini 4+, Grok 5+) with ~18 months remaining. Tool-use and multi-agent pipelines drive gains. Contamination/memorization concerns raised for Mythos. [NxCode, EmergentMind] # Key facts (high-confidence, factual) 1. [Wikipedia/CAIS] HLE = 2,500 questions, created by CAIS and Scale AI; designed to resist saturation 2. [xAI] Grok 4 Heavy first crossed 50% (50.7%) in July 2025 — text-only with tools 3. [NxCode/BenchLM.ai] Claude Mythos Preview scored 64.7% with tools as of ~April–June 2026; not publicly available 4. [LLM Stats] Human experts achieve ~65% accuracy on HLE (one source says ~98% — discrepancy exists) 5. [Wikipedia] ~30% of HLE text-only chemistry/biology answers may be incorrect per FutureHouse (July 2025); benchmark integrity is disputed 6. [IntuitionLabs] HLE-Rolling launched Oct 2025 to refresh dataset dynamically as models saturate it 7. [LLM Stats blog] Anthropic flagged possible memorization in Mythos Preview's HLE performance # Cross-market signals - **Kalshi related**: Only the same ticker found; no sister bucket markets available for arbitrage comparison - **Polymarket**: No matching markets found - **Sportsbook implied**: N/A # Analyst opinions and speculation - HLE organizers (CAIS) noted benchmarks "quickly saturate" and expected >50% by end of 2025 — prediction roughly correct [agi.safe.ai] - The ~23 pp Kalshi price drop in one week suggests market participants received a strong negative signal — possibly Mythos' restricted access (no public verification) or signs of plateau - Trajectory extrapolation: +55 pp in 17 months → another ~6 pp needed in 18 months seems achievable, but gains are compressing near human ceiling # Directional lean per outcome - **Yes (≥70%)**: Strong historical pace; SOTA at 64.7% already; 18 months of further model releases; tool-use/agentic pipelines continue to drive gains; multiple frontier labs competing - **No (<70%)**: Sharp market price drop signals negative update; benchmark near human-expert ceiling (~65%); contamination/memorization concerns may inflate apparent progress; Mythos not publicly verifiable; benchmark may be refreshed (HLE-Rolling) resetting scores; diminishing returns on hardest questions; human experts ~65% suggests 70% may be a genuine ceiling # Gaps / unknowns - Why did Kalshi price drop ~23 pp in one week? (No specific news event identified) - Whether "with tools" scores qualify for market resolution (rules silent) - Whether HLE-Rolling replaces original benchmark for resolution purposes - Actual public availability of ≥70% scoring model before deadline - Human expert ceiling discrepancy (65% vs 98% across sources) # Calibration anchors - **Kalshi current YES price: 46%** (anchor — recently fell sharply from ~69%) - SOTA is 64.7% as of ~June 2026 with 18 months remaining; needs ~5.3 pp more - Pace of improvement has been ~3–4 pp/month historically, but appears to be slowing - Benchmark integrity and access restrictions add downward pressure on confidence

1. Both forecasters underweight the benchmark integrity and resolution ambiguity issues. The evidence brief explicitly flags that "with tools" scores may not qualify for resolution (rules are silent), and the 64.7% SOTA is tool-assisted and from a restricted-access model. If the market resolves only on publicly verifiable, tool-free scores, the effective SOTA drops to ~57–58%, requiring ~12 pp more — a much harder target. 2. Neither forecaster adequately addresses the human-expert ceiling discrepancy (65% vs. 98%). If the true human ceiling is ~65%, then 70% represents superhuman performance on one of the hardest benchmarks ever designed, which is a qualitatively different milestone than "5.3 pp more progress." If human experts score ~65%, achieving 70% may require capabilities that don't follow the same extrapolation curve used by both forecasters. 3. Both forecasters treat the ~23 pp Kalshi price drop as a vague "negative signal" without sufficiently updating on it. A market drop of that magnitude in one week on an actively traded contract with meaningful liquidity represents a very strong Bayesian update, yet both forecasters end up 4–5 pp above the post-drop Kalshi price without identifying a specific reason to override the market's information. 4. The HLE-Rolling replacement risk is underweighted by both forecasters. The brief notes HLE-Rolling launched in October 2025 specifically to refresh the dataset as models saturate it. If the market resolves on a dynamic/rolling version of HLE rather than the static original, current scores of 64.7% are irrelevant, and the question effectively resets — a scenario that sharply favors "No" but receives no probability mass from either forecaster. 5. Both forecasters use a linear extrapolation of "3–4 pp/month" without accounting for the well-documented phenomenon of benchmark saturation accelerating at the tail. The brief itself notes "pace appears to be slowing as the benchmark hardens near the human-expert ceiling" and cites FutureHouse's finding that ~30% of chemistry/biology answers may be incorrect, meaning the effective ceiling for a well-calibrated model may be below 70% regardless of capability improvements.

42% (38% confidence)

Yes: 42% No: 58%

The critique highlights real downward pressures both forecasters underweighted: tool-use/resolution ambiguity, the ~65% human-expert ceiling making 70% qualitatively harder, HLE-Rolling refresh risk, and the magnitude of the recent Kalshi drop. I anchor near the post-drop market price but pull slightly below it to reflect these unaddressed tail risks, while still giving meaningful weight to the strong trajectory and 18 months of frontier model releases remaining.

Total pipeline time: 113.2s