forecasting
agent
← back to metaculus scans

[PRACTICE] What will the average "new forecasters per day" be on this question before this question closes?

post 45516·question 45707·discrete·fall-futureeval-2026·2026-09-07T18:14:25.313755·view on metaculus ↗
6.45
Submitted p50
weighted_average
Fermi logic
tech
Domain

Plan 24.97s

Establish the open date (denominator) from the Metaculus page, then combine it with the expected final forecaster count (numerator anchored on 156 from Summer 2026, adjusted for season growth) to produce a distribution over forecasters/day, with sq1 governing scale and sq2/sq3 governing the numerator's center and upper tail.

Sub-questions (3)

sq1
Will this question be open for fewer than 10 days in total (open date to 2026-09-28 close)?
The denominator (days open) is the dominant driver: ~150 forecasters over 7 days gives ~21/day, while over 30 days gives ~5/day. FutureEval warm-up questions typically have short open windows, so pinning the open date is critical.
w=0.40
sq2
Will the forecaster count displayed on the Metaculus UI at close be at least 150?
The referenced Summer 2026 practice question drew 156 forecasters; the number of bots registered for the Fall 2026 FutureEval season sets the numerator scale.
w=0.35
sq3
Will the forecaster count at close exceed 250 (i.e., substantially more bot participation than the Summer 2026 season)?
Bot participation in the benchmark has grown season over season; a large jump in registered bots would push the per-day average well above the naive extrapolation from 156.
w=0.25

Tool requests (3)

web_search sq1, sq2, sq3 {"queries": ["Metaculus FutureEval Fall 2026 practice question new forecasters per day", "Metaculus AI Forecasting Benchmark Fall 2026 warm-up questions open date number of bots", "site:metaculus.com \"[PRACTICE]\" \"new forecasters per day\""]}
claude_news sq1, sq2, sq3 {"brief": "Find the Metaculus FutureEval (AI Forecasting Benchmark) Fall 2026 tournament practice/warm-up questions. Determine: (1) the exact open date of the \u0027[PRACTICE] What will the average new forecasters per day be on this question\u0027 question and thus how many days it is open before the 2026-09-28 close; (2) the current number of forecasters shown on that question and on sibling practice questions; (3) how many bots are registered for the Fall 2026 season compared to Summer 2026 (where a practice question got 156 forecasters).", "max_searches": 4, "question_title": "[PRACTICE] What will the average \"new forecasters per day\" be on this question before this question closes?"}
code_execution sq1, sq2, sq3 {"code": "import numpy as np\nrng=np.random.default_rng(0)\nN=50000\n# days open scenarios: short window (5-10 days) most likely for warm-up questions, longer tail\ndays=np.where(rng.random(N)\u003c0.65, rng.integers(5,11,N), rng.integers(11,35,N))\n# forecasters: lognormal centered ~170 based on 156 last season with growth\nforecasters=np.clip(rng.lognormal(np.log(170),0.35,N),30,600)\navg=forecasters/days\nprint(\u0027mean\u0027,avg.mean().round(1),\u0027median\u0027,np.median(avg).round(1))\nfor q in [5,10,25,50,75,90,95]:\n print(q,\u0027th pct\u0027,np.percentile(avg,q).round(1))\nfor d in [5,7,10,14,21,30]:\n print(\u0027days\u0027,d,\u0027-\u003e avg at 156 forecasters:\u0027,round(156/d,1),\u0027 at 250:\u0027,round(250/d,1))", "description": "Monte Carlo over open-window length and forecaster count to get a distribution of forecasters-per-day, plus deterministic scenario table."}

Research 34.54s · 3 tool(s) · 0 error(s)

web_search ok 20 results 34.54s
claude_news ok 7 citations 21.33s
code_execution ok Monte Carlo over open-window length and forecaster count to 0.12s
Cross-market signals
## Cross-Market Signals

### No signal found

Evidence Synthesis 29.24s · 10 items

Direction mix: UP 2 DOWN 0 NEUTRAL 8
sq2 question_description STRONG cred 92 NEUTRAL DATED
The Summer 2026 FutureEval practice question (Metaculus #43339) received 156 forecasters, per the question text itself.
sq1 question_description MODERATE cred 85 NEUTRAL RECENT
Close date is fixed at 2026-09-28; the open date was not found by any research tool, leaving the denominator unknown.
sq1 web_search MODERATE cred 75 NEUTRAL DATED
Spring 2026 AIB began Jan 5 with fewer questions in the first 1-2 weeks; practice questions served as a warm-up period for new bots.
sq1 question_description WEAK cred 60 NEUTRAL RECENT
Question states it exists to make bots aware of open/close dates, implying the open window is a deliberate design variable.
sq3 web_search MODERATE cred 78 NEUTRAL DATED
AIB/FutureEval runs three seasons per year with $50k prize pools and 300-500 questions each, a stable format across seasons.
sq3 web_search WEAK cred 70 UP DATED
FutureEval launched publicly Feb 2026 with press coverage, and Metaculus adds ~100 Pro Forecaster-covered questions per tournament.
sq3 web_search WEAK cred 55 NEUTRAL VERY_RECENT
No Fall 2026 AIB announcement, registration count, or bot participation figure was found in any search result.
sq1 code_execution WEAK cred 40 NEUTRAL VERY_RECENT
Monte Carlo over window length and forecaster count gives median 17.6 and mean 19.6 forecasters/day, with 5th-95th range 4.7-42.7.
sq2 code_execution MODERATE cred 80 NEUTRAL VERY_RECENT
At 156 forecasters, average is 31.2/day for a 5-day window, 15.6 for 10 days, 7.4 for 21 days, 5.2 for 30 days.
sq3 code_execution MODERATE cred 80 UP VERY_RECENT
At 250 forecasters, average is 50.0/day for 5 days, 25.0 for 10 days, 11.9 for 21 days, 8.3 for 30 days.

Information gaps

Key uncertainties

Forecast 34.27s · opus, secondary

Rationale

(a) The question closes 2026-09-28; the open date is unknown, but the designer's chosen range of [0.95, 10.05] strongly implies they anticipated a value of a few forecasters per day, which pins the open window at roughly 2-6 weeks rather than a few days. (b) Status quo: the Summer 2026 analogue drew 156 forecasters; over a ~3-4 week window that yields ~5-7/day. (c) Trend: bot participation has grown season over season and practice questions attract many new bots, so 150-250 forecasters is plausible, but many established bots may skip warm-ups, so the count could also fall below 156. (d) No market or community signal exists; the base rate anchor is 156 forecasters and the designer-implied range centered near ~5. (e) Low scenario (~2.5): question opened ~6 weeks early (e.g., mid-August) and only ~100 bots forecast it, giving ~2.4/day. (f) High scenario (~9-10+): a ~2-week window with 180-250 forecasters, or a surge of new bots registering for Fall 2026, pushing 9-12/day. I center near 5 with a wide interval reflecting the unknown denominator and the multiplicative uncertainty, allowing some mass beyond the displayed range.
Evidence brief shown to forecaster
1. [sq2 | question_description | STRONG cred 92 | NEUTRAL | DATED] The Summer 2026 FutureEval practice question (Metaculus #43339) received 156 forecasters, per the question text itself.
2. [sq1 | question_description | MODERATE cred 85 | NEUTRAL | RECENT] Close date is fixed at 2026-09-28; the open date was not found by any research tool, leaving the denominator unknown.
3. [sq1 | web_search | MODERATE cred 75 | NEUTRAL | DATED] Spring 2026 AIB began Jan 5 with fewer questions in the first 1-2 weeks; practice questions served as a warm-up period for new bots.
4. [sq1 | question_description | WEAK cred 60 | NEUTRAL | RECENT] Question states it exists to make bots aware of open/close dates, implying the open window is a deliberate design variable.
5. [sq3 | web_search | MODERATE cred 78 | NEUTRAL | DATED] AIB/FutureEval runs three seasons per year with $50k prize pools and 300-500 questions each, a stable format across seasons.
6. [sq3 | web_search | WEAK cred 70 | UP | DATED] FutureEval launched publicly Feb 2026 with press coverage, and Metaculus adds ~100 Pro Forecaster-covered questions per tournament.
7. [sq3 | web_search | WEAK cred 55 | NEUTRAL | VERY_RECENT] No Fall 2026 AIB announcement, registration count, or bot participation figure was found in any search result.
8. [sq1 | code_execution | WEAK cred 40 | NEUTRAL | VERY_RECENT] Monte Carlo over window length and forecaster count gives median 17.6 and mean 19.6 forecasters/day, with 5th-95th range 4.7-42.7.
9. [sq2 | code_execution | MODERATE cred 80 | NEUTRAL | VERY_RECENT] At 156 forecasters, average is 31.2/day for a 5-day window, 15.6 for 10 days, 7.4 for 21 days, 5.2 for 30 days.
10. [sq3 | code_execution | MODERATE cred 80 | UP | VERY_RECENT] At 250 forecasters, average is 50.0/day for 5 days, 25.0 for 10 days, 11.9 for 21 days, 8.3 for 30 days.

## Cross-Market Signals

### No signal found

Information gaps:
  - Actual open date of this question (denominator) not retrieved from Metaculus page
  - Open-window length of the Summer 2026 practice question (#43339) for base rate
  - Fall 2026 bot registration count or season growth trend vs Summer
  - Whether UI forecaster count includes human forecasters or only bots

Key uncertainties:
  - Open window length: 5 vs 30 days changes result ~6x
  - Season-over-season growth or decline in bot participation
  - Late-joining bots inflating count right before close
  - Possible discrepancy between UI-displayed count and API count
Raw forecast prompt
You are an elite superforecaster. Produce a probability distribution over the answer to this Metaculus numeric question.

## Question
[PRACTICE] What will the average "new forecasters per day" be on this question before this question closes?

## Description / Resolution Criteria
## Description
Each season of FutureEval, bots forecast the warm-up questions in the tournament. In Summer 2026, one of these [practice questions](https://www.metaculus.com/questions/43339/major-state-exits-eu-before-2027/) received 156 forecasters. This question helps encourage bots to be aware of the open/close date of a question. Bots should assume a question has not yet been resolved by the time a question opens.

## Resolution Criteria
This question resolves to the number of forecasters listed in the Metaculus UI on the close date for this question divided by the number of days this question is open. The question will round to the nearest tenth.

## Range
The answer must be a number in [0.95, 10.05] (units: forecasters).

## Sub-question decomposition (planner)
- (w=0.40) Will this question be open for fewer than 10 days in total (open date to 2026-09-28 close)?  — The denominator (days open) is the dominant driver: ~150 forecasters over 7 days gives ~21/day, while over 30 days gives
- (w=0.35) Will the forecaster count displayed on the Metaculus UI at close be at least 150?  — The referenced Summer 2026 practice question drew 156 forecasters; the number of bots registered for the Fall 2026 Futur
- (w=0.25) Will the forecaster count at close exceed 250 (i.e., substantially more bot participation than the Summer 2026 season)?  — Bot participation in the benchmark has grown season over season; a large jump in registered bots would push the per-day 

## Synthesized evidence
1. [sq2 | question_description | STRONG cred 92 | NEUTRAL | DATED] The Summer 2026 FutureEval practice question (Metaculus #43339) received 156 forecasters, per the question text itself.
2. [sq1 | question_description | MODERATE cred 85 | NEUTRAL | RECENT] Close date is fixed at 2026-09-28; the open date was not found by any research tool, leaving the denominator unknown.
3. [sq1 | web_search | MODERATE cred 75 | NEUTRAL | DATED] Spring 2026 AIB began Jan 5 with fewer questions in the first 1-2 weeks; practice questions served as a warm-up period for new bots.
4. [sq1 | question_description | WEAK cred 60 | NEUTRAL | RECENT] Question states it exists to make bots aware of open/close dates, implying the open window is a deliberate design variable.
5. [sq3 | web_search | MODERATE cred 78 | NEUTRAL | DATED] AIB/FutureEval runs three seasons per year with $50k prize pools and 300-500 questions each, a stable format across seasons.
6. [sq3 | web_search | WEAK cred 70 | UP | DATED] FutureEval launched publicly Feb 2026 with press coverage, and Metaculus adds ~100 Pro Forecaster-covered questions per tournament.
7. [sq3 | web_search | WEAK cred 55 | NEUTRAL | VERY_RECENT] No Fall 2026 AIB announcement, registration count, or bot participation figure was found in any search result.
8. [sq1 | code_execution | WEAK cred 40 | NEUTRAL | VERY_RECENT] Monte Carlo over window length and forecaster count gives median 17.6 and mean 19.6 forecasters/day, with 5th-95th range 4.7-42.7.
9. [sq2 | code_execution | MODERATE cred 80 | NEUTRAL | VERY_RECENT] At 156 forecasters, average is 31.2/day for a 5-day window, 15.6 for 10 days, 7.4 for 21 days, 5.2 for 30 days.
10. [sq3 | code_execution | MODERATE cred 80 | UP | VERY_RECENT] At 250 forecasters, average is 50.0/day for 5 days, 25.0 for 10 days, 11.9 for 21 days, 8.3 for 30 days.

## Cross-Market Signals

### No signal found

Information gaps:
  - Actual open date of this question (denominator) not retrieved from Metaculus page
  - Open-window length of the Summer 2026 practice question (#43339) for base rate
  - Fall 2026 bot registration count or season growth trend vs Summer
  - Whether UI forecaster count includes human forecasters or only bots

Key uncertainties:
  - Open window length: 5 vs 30 days changes result ~6x
  - Season-over-season growth or decline in bot participation
  - Late-joining bots inflating count right before close
  - Possible discrepancy between UI-displayed count and API count

## Required pre-forecast walkthrough

Before giving percentiles, address these explicitly in your rationale:
  (a) The time left until the question resolves.
  (b) The outcome if NOTHING changes from today (the status quo value).
  (c) The outcome if the CURRENT TREND continues.
  (d) The expectations of experts / markets / base rates.
  (e) A plausible scenario that produces a LOW outcome (near p10).
  (f) A plausible scenario that produces a HIGH outcome (near p90).

## Calibration guidance

- **Be humble about tails.** Good forecasters set WIDE 90/10 intervals to account for unknown unknowns. Narrow tails get punished by the log score far more than slightly-biased medians.
- **Status quo anchoring.** The p50 should be close to the status quo value unless you have strong evidence of a trend.
- Don't pile mass at one value — if you're tempted, widen the spread by 20-50%.
- **Anchor on markets/experts.** If liquid market prices, analyst forecasts, or community percentiles appear in the evidence, center your distribution on them and widen — don't override a liquid market without specific evidence it lacks.
- **Relative-return / spread questions ("how much will X's return exceed Y's").** A near-zero median is usually right, but size the TAILS to the more VOLATILE leg, not to a generic 2-3pp spread. Two broad equity indices (e.g. Nasdaq-100 vs S&P 500) do stay within roughly ±2-3pp over a two-week window. But when one leg is a commodity (crude oil, gold) or a single high-beta stock (e.g. Nvidia), the two-week realized spread regularly reaches ±10pp or more — crude-vs-equity spreads have hit ±18pp. In those cases set p10/p90 to at least ±10pp (±15pp or wider for oil) while keeping the median near the status-quo spread; the 2-3pp intuition applies only to index-vs-index.
- **Heavy-tailed one-off magnitudes (opening-weekend box office, product-launch or debut numbers).** Pre-event tracking for these routinely misses by ±40% or more, and results land both far above and far below the consensus. Do NOT treat the displayed [min, max] range as the plausible range or clamp p10/p90 to its edges: the outcome can fall outside the range in either direction (recent opening weekends resolved ~7% above a displayed ceiling in one case and roughly half the displayed floor in another). Set p10/p90 wide enough that a 2x-or-0.5x surprise versus your central estimate still lands inside the interval, and when concrete current tracking disagrees with historical/franchise base rates, weight the tracking.

## Output

Return ONLY valid JSON, no markdown fences:

{
  "rationale": "<address (a) through (f) above, then state your distribution — 6-10 sentences total>",
  "p10": <number>,
  "p25": <number>,
  "p50": <number>,
  "p75": <number>,
  "p90": <number>
}

Percentiles MUST be in non-decreasing order (p10 <= p25 <= p50 <= p75 <= p90).