methodology/ · eval-first discipline

Don't ship what hasn't passed a measurable gate.

One engineering principle. Applied to LLM outputs and to systematic trading strategies with the same gate stack. The method is what scales; the tools change.

11 min readlast updatedhow verified ↗

The gate stack

G1–G31. Six families. One contract.

Every candidate — LLM output, systematic strategy, content artifact — passes through the same gate stack. Gates are mechanical (exit-0 contract), not opinionated. A pass means the candidate is allowed to surface; a fail means it isn't.

G1–G5

Mechanical validity

Schema conformance, exit-0 contract, structured output parse, JSON Schema strict-mode, byte-length budget.

G6–G10

Statistical nulls

Block-bootstrap CIs, random-timing nulls, regime-shuffle, parameter-prior sensitivity, Monte-Carlo coverage.

G11–G15

Walk-forward & OOS

Locked OOS windows, 5-era stability, expanding/rolling windows, frozen-spec evaluation, embargoed test set.

G16–G20

Multiple testing

Deflated Sharpe Ratio (DSR), CSCV-based Probability of Backtest Overfit (PBO), Minimum Backtest Length (MinBTL), Bonferroni–Holm, BH-FDR.

G21–G25

Look-ahead discipline

Point-in-time dataset verification, survivorship-bias check, rebalance-vs-signal timing, frozen-spec, no-leakage to scoring.

G26–G31

Economic & cost realism

Spread/slippage/latency per asset class, capacity-constraint report, funding-carry, borrow cost, vol-targeted sizing, regime overlay.

Model routing

Tiered dispatch, not single-agent.

A multi-agent system runs at ~15× the token cost of a single agent. The cost is real — the answer is to route by what the step actually needs to compute, not by default. Every dispatch has a per-step budget.

Opus
judgment / hardest autonomous decisions
~1 dispatch per analysis chain
Sonnet
assembly / structured generation
~3–5 dispatches per chain
Haiku
mechanical / fast deterministic steps
~10–30 dispatches per chain
Fable
long-horizon autonomous research
one off, overnight

Interactive · G16

Try the deflated Sharpe calculator.

See how the same in-sample Sharpe can pass or fail G16 depending on how many strategies you actually tried. This is the gate that turns "1.14 looks great" into "DSR 0.70 — fail."

deflated Sharpe · interactive

DSR calculator

Drag the sliders. DSR tells you whether your in-sample Sharpe survives the multiple-testing penalty for the number of strategies you tried.

11001000
−0.52.04.0
0.5y10y20y
computing…
DSR
denominator expected SR γ₃ γ₄

DSR uses the normal-approximation of the expected maximum Sharpe under N i.i.d. trials, then deflates by √(1 − γ₃·SR + (γ₄·SR² − 1)/4).

Why 0.95? — reading the z-score and the numerator

The number the calculator prints is a z-score on the null H₀: "your IS Sharpe is just the best of N i.i.d. N(0,1) draws." Pass at DSR ≥ 1.645 = one-sided 95% test (the standard frequentist threshold; equivalent to 0.05 significance). Anything below is inconclusive or fail — not because 0.05 is sacred, but because 1.645σ is the conventional bar for "I tried many things and this one still looks unlikely to be noise."

Read it as a ratio:numerator = SR̂ − E[max SR under N trials] (how much your Sharpe beats the expected best of N random strategies);denominator = σ_SR̂ · √(1 − γ₃·SR + (γ₄−1)·SR²/4) (how noisy the Sharpe estimate is, with the higher-moment correction for non-normal returns). A DSR of 0.70 means "the gap is 0.70 noise-units wide" — not a 70% pass rate.

Ref: Bailey & López de Prado, "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality" (2014), JPM 40(5).

Agent charter

Six required fields per agent.

A charter is a contract, not a prompt. Every agent has the same six fields, with the same semantics. No exceptions.

role
A specific function, never generic "AI assistant"
inputs
Typed input contract (JSON Schema, fail-closed)
outputs
Typed output contract with strict validation
handoffs
What this agent must NOT do (anti-charter)
gates
Which of G1–G31 apply, with exit codes
evidence
What the agent must attach as proof

What I will not do

Negative-space as design.

  • I will not claim current employment at any NDA-protected firm. The 19V Capital engagement is a closed past contract (03/2026 – 06/2026).
  • I will not reference proprietary data sources, fund renames, or per-strategy t-stats from any NDA-protected desk.
  • I will not share a "Sharpe = X.X" number without showing the gate stack that produced it.
  • I will not ship an agent without an exit-0 mechanical-validator contract.
  • I will not present a backtest without capacity, slippage, and a locked OOS window.

What I'd build first

If you hired me onto a quant desk.

These are the first 90 days, ordered by leverage. A new QR on a 2–5 person pod doesn't start with alpha — they start by making the existing book defensible to a risk manager and a PM. The discipline compounds; the alpha is downstream.

  1. WK 1–2

    Backtest-engine audit + look-ahead scan

    Run the one-bar shift test (project 09) on every strategy in the book. Quantify the phantom-Sharpe gap. This is the gate that catches the most expensive error class first.

    G21–G25 · look-ahead discipline
  2. WK 3–4

    Cost-model calibration + break-even turnaround

    Per-strategy slippage / impact / spread estimate. For every strategy, ship the break-even turnaround cost (project 08). Strategies that win gross and lose net come off the book.

    G26–G31 · economic & cost realism
  3. WK 5–8

    Multiple-testing correction across the book

    Re-rank every strategy by DSR (project 01), not raw Sharpe. Re-rank by CSCV-based PBO (prob. of backtest overfit). The book's "best strategy" is rarely the strategy with the highest raw Sharpe.

    G16–G20 · multiple testing
  4. WK 9–12

    Walk-forward OOS window standardization

    Locked, embargoed OOS windows (project 06 pattern) across the book. Frozen-spec evaluation. Embargoed test sets. The book becomes reproducible from cold start.

    G11–G15 · walk-forward & OOS
The pattern: defenses before alpha. A QR who can defend the book is hireable; a QR who can find alpha is hireable-and-defensible. The first 90 days establish the defense.

Method under-spec? Send the JD.

If your role has a written spec, send it. I'll respond with which gates from G1–G31 already apply to your stack, and which I'd build.