Order-Flow Signal Atlas — L2 & L3¶
One row per signal: what it is betting on, and where it comes from. Brain markers: ● = paper wiki'd in the second brain · ◐ = method page exists but paper not ingested · ○ = not in the brain.
L2 — signals computable from top-10 snapshots¶
| # | Signal | Rationale | Source paper(s) |
|---|---|---|---|
| 1 | Best-level OFI | Over short windows, Δmid is linear in net signed flow at the best quotes, with slope ∝ 1/depth. A market-sell and a bid-cancel of equal size are the same event: both drain the bid queue. | Cont, Kukanov, Stoikov (2014), J. Fin. Econometrics, arXiv 1011.6402 ● |
| 2 | Multi-level OFI (10-level vector) | Deeper levels carry real signal; CKS's "best level suffices" was an OLS multicollinearity artefact. Under Ridge every level to depth 10 is significant; the gain is largest for large-tick names — i.e. most of this market. | Xu, Gould, Howison (2019), arXiv 1907.06230 ● |
| 3 | Integrated OFI (PCA) | The 10-level OFI vector is highly collinear; its first PC captures >89% of variance. ℓ1-normalised PC weights collapse it to one scalar in OFI units — R² 71%→87% vs best-level on Nasdaq. Best default aggregator. | Cont, Cucuringu, Zhang (2023), Quant. Finance, arXiv 2112.13213 ● |
| 4 | Lagged cross-asset OFI (LASSO) | Contemporaneous cross-impact is redundant once levels are integrated, but lagged cross-asset OFI adds forecasting power at short horizons, decaying fast. Use sparse selection, prediction only. | Cont, Cucuringu, Zhang (2023) ● |
| 5 | L1 queue imbalance (+ logistic map) | The heavier best queue predicts the direction of the next mid move; the relation saturates near ±1. Predictive lift is strongest on large-tick stocks — the dominant regime on this market. | Lipton, Pesavento, Sotiropoulos (2013), arXiv 1312.0514 ● · Gould, Bonart (2015), arXiv 1512.03492 ● |
| 6 | Depth-weighted multi-level imbalance | Same bet as L1 imbalance, extended with exponential decay across levels; the decay rate trades off L1 purity (large-tick) against deep-book information (small-tick). | Extension of Gould–Bonart ● / Xu–Gould–Howison ● |
| 7 | Order-count imbalance | Same side pressure read through number of orders rather than volume (the exchange feed provides both): many small orders ≠ one large order. Cross-check against volume imbalance to flag spoofed size. | Practitioner-standard; enabled by the exchange feed's per-level order counts (no canonical paper) |
| 8 | Microprice (basic + Markov g(I,s)) | Fair value inside the spread, biased toward the heavy queue; formally the fixed point of expected future mid adjustments conditional on (imbalance, spread). Leads the mid; most informative in large-tick books. | Stoikov (2018), Quant. Finance ◐ (method page only — paper is SSRN/QF, not arXiv) |
| 9 | VOI / OIR | Best-quote size deltas with CKS-style price conditioning (VOI ≈ best-level OFI) plus the L1 ratio (OIR = I1); the practitioner packaging that combined book and trade terms into one tradeable signal. | Shen (2015), Oxford MSc thesis ○ |
| 10 | Trade-flow imbalance (signed volume) | Sign executed volume instead of queue changes — the second, orthogonal "OFI" convention. On crypto it explained contemporaneous returns better than event-OFI; keep both as separate features. | Silantyev (2019), Digital Finance ○ · trade-based convention covered in brain via Anantha–Jain (2024) ● |
| 11 | Book slope / shape asymmetry | How fast depth accumulates away from the mid proxies elasticity: a flat (shallow-sloped) side is cheap to push through. On the meso-scale, LOB shape beats plain imbalance. | Order Flows and LOB Resiliency on the Meso-Scale, arXiv 1708.02715 ○ · Næs, Skjeltorp (2006), J. Fin. Markets ○ |
| 12 | Cancellation-rate proxy + spread resiliency (AR half-life) | High cancel intensity = fragile displayed liquidity; slow spread reversion after widening = weak replenishment. At L2 both are proxies (size drops at unchanged prices); superseded by exact L3 versions. | Large (2007), J. Fin. Markets ○ · arXiv 1708.02715 ○ |
| 13 | VPIN (toxicity gate) | Volume-clocked |buy−sell| imbalance estimates flow toxicity — the probability passive quotes face informed flow. Contested as a crash predictor; use to throttle, never to direct. | Easley, López de Prado, O'Hara (2012), RFS ○ · critique: Andersen, Bondarenko (2014), J. Fin. Markets ○ (both non-arXiv — skipped by pipeline) |
| 14 | Propagator drift | Price = past signed flow convolved with a slowly decaying kernel; the transient component predicts residual drift/reversion. Execution-scheduling signal more than tick-level alpha. | Bouchaud, Farmer, Lillo (2009), arXiv 0809.0822 ● · Brokmann et al. (2015), arXiv 1407.3390 ● |
| 15 | Handcrafted ML feature stack | Feed models stationary flow features, not raw book levels — flows beat levels at every horizon; the LOB→price map is largely stock-agnostic, so pooling across names works. | Kolm, Turiel, Westray (2023), Math. Finance, arXiv 2102.08811 ○ · Zhang, Zohren, Roberts DeepLOB (2019), arXiv 1808.03668 ○ · Sirignano, Cont (2019) ● · Briola et al. (2024) ● |
L3 — signals requiring order-by-order data¶
| # | Signal | Rationale | Source paper(s) |
|---|---|---|---|
| 1 | Event-type-decomposed OFI (MO/LO/CX) | Market orders, limit orders and cancellations have different impact per share and different persistence; on large-tick names LO/CX impact rivals MO impact. Let the regression weight the six components instead of forcing equality. | Eisler, Bouchaud, Kockelkoren (2012), Quant. Finance, arXiv 0904.0900 ● · EBK (2011), Models for the impact of all order book events ● |
| 2 | Sweep detection / signed sweep volume | One aggressive order walking ≥2 levels pays for immediacy — the classic informed-flow fingerprint. Exact aggressor sides and event clustering make sweeps observable only at L3. | Rooted in Easley, O'Hara (1987), JFE (trade size ↔ information) ○; sweep construction is practitioner-standard |
| 3 | Size-bucketed TFI (large vs small clips) | Information per share is not monotone in clip size — stealth traders slice into medium clips. Bucket the signed flow and fit weights per bucket rather than pooling. | Barclay, Warner (1993), JFE (stealth trading) ○ · Easley, O'Hara (1987) ○ |
| 4 | Queue-reactive intensity race | Arrival/cancel/execution intensities at the best are functions of queue state; estimate them exactly from the event tape and race the two best queues to first depletion. Subsumes static imbalance (I1 = zeroth-order case). | Huang, Lehalle, Rosenbaum (2015), JASA, arXiv 1312.0563 ◐ (method page only) |
| 5 | Order-age-weighted imbalance | Aged resting orders are patient capital; sub-second orders are noise. Weight depth by order age (exact at L3 via ts_add) before computing imbalance — cleans both QI and microprice. | Hasbrouck, Saar (2009), J. Fin. Markets ○ |
| 6 | Fleeting-liquidity ratio | The share of added volume cancelled within τ (~500 ms) measures how much displayed depth is real. A fleeting-heavy side gives way; the mid moves through it. | Hasbrouck, Saar (2009, 2013), J. Fin. Markets ○ |
| 7 | Depth concentration / largest-order fragility | A level held by one order vaporises with one cancel; the same size across 30 orders does not. max-order-share of the level is the fragility the L2 order-count imbalance only proxied. | Bouchaud, Bonart, Donier, Gould, Trades, Quotes and Prices (CUP 2018) ○; practitioner-standard |
| 8 | Cancel-flow imbalance / withdrawal cascades | Liquidity withdrawal leads price: cancel intensity spiking on the side being hit means makers are stepping away — continuation. Exact CX flow (incl. partial reductions) exists only at L3. | Eisler, Bouchaud, Kockelkoren (2012) ● (cancels carry standalone impact) |
| 9 | Pull / spoof-pattern fade | Large size posted near the touch and cancelled unexecuted within a short lifetime was pressure that was never real — fade the advertised direction, and gate imbalance signals while the pattern is live. | Cartea, Jaimungal, Wang (2020), Applied Math. Finance ○ · Lee, Eom, Park (2013), J. Fin. Markets (KRX spoofing empirics) ○ |
| 10 | Reload (synthetic iceberg) detection + exhaustion | The exchange has no native icebergs, so every reload pattern (execution → same-price re-add, similar clip, repeated) is an algo revealing hidden interest: support while active, breakout fuel when it stops refilling. | Christensen, Woodmansey (2013) ○ · Esser, Mönch (2007) ○ (native-iceberg detection, methodology transfers) |
| 11 | Metaorder / child-slicing detection | Institutional metaorders execute as regular clips; while active, drift continues in their direction (square-root regime) and transient impact partially reverts on completion. Detect regularity in inter-arrival times and clip sizes. | Tóth et al. (2011), arXiv 1105.1694 ○ · Brokmann et al. (2015) ● (post-metaorder decay) |
| 12 | Hawkes intensity imbalance | Order flow is self- and cross-exciting; fitted conditional intensities per event type give a de-noised, forward-looking version of raw flow imbalances, updated O(1) per event. | Bacry, Mastromatteo, Muzy (2015), arXiv 1502.04592 ◐ (method page only) · Large (2007) ○ |
| 13 | Participant-ID queue composition (venue-dependent) | Where a venue disseminates participant identity at the touch, it is information — anonymity regimes change market quality, so its presence is exploitable: institutional-heavy, low-churn queues are real levels; retail-heavy, high-churn queues are not. | Foucault, Moinas, Theissen (2007), RFS (value of anonymity) ○; practitioner territory |
| 14 | Message-rate / OTR burst gate | Bursts of adds+cancels (quote stuffing, HFT skirmishes) degrade signal quality and widen effective spreads; use as a toxicity regime gate scaling down all directional signals. | Hasbrouck, Saar (2013), J. Fin. Markets ○ · Egginton, Van Ness, Van Ness (2016), Financial Management ○ |
| 15 | Event-time resiliency asymmetry | After a level is wiped, the time to rebuild — measured exactly per depletion event — reveals which side's makers are committed; slow-rebuilding side is fragile. | Large (2007), J. Fin. Markets ○ · Obizhaeva, Wang (2013), J. Fin. Markets (resilience in execution) ○ |
Cross-cutting conditioners (not signals — interaction terms and gates)¶
| Conditioner | Rationale | Source |
|---|---|---|
| Tick regime (rtick = tick/mid) | The single strongest moderator: large-tick names → L1 imbalance, microprice, queue race dominate; small-tick names → multi-level/integrated OFI dominates. Convergent evidence across equities and crypto; re-fit everything across the venue's tick-reform history. | Gould, Bonart (2015) ● · Xu, Gould, Howison (2019) ● · Briola et al. (2024) ● · Bieganowski, Ślepaczuk (2026) ● · Dayri, Rosenbaum (2015) ○ |
| Session / auction hygiene | The opening and closing call auctions have no continuous matching — continuous-trading coefficients are mis-specified there; reset all signal state at each continuous-session open (the book itself persists across the midday break), on feed gaps, and around volatility-halt cooling-offs. | exchange trading-mechanism rules; no paper |
| Toxicity gating | Scale exposure down, not direction, when VPIN / cancel intensity / message-rate z-scores spike. | Easley, López de Prado, O'Hara (2012) ○ |
---¶
Implementation Specification (self-contained, agent-ready)¶
This section is written so that an implementation agent can build the full library without reading any paper or any other document. Every part states what it does, why, the exact inputs, every parameter with a default, the algorithm in Python-style pseudo code, the outputs with their sign convention, the edge cases, and — where possible — a numeric fixture the implementation must reproduce. Atlas rows L2-n / L3-n map 1:1 to §6.n / §7.n.
The three inputs (our own recording, not a vendor normalization):
- L3 file — order events recorded from the exchange: ADD, MODIFY, CANCEL only. There are no execution messages: fills appear as CANCELs (full fill) or MODIFY-downs (partial fill) of resting orders, and trades must be inferred (§4.3).
- L2 file — our own rebuilt top-10 snapshot of the book, one row per book change, carrying the exchSeq of the L3 event that produced it. Used (a) as the direct input to every §6 signal, (b) as the prevailing book for trade-side inference, (c) as a cross-check on the §4 book builder.
- Trade file — executions (price, quantity, timestamp; exchSeq and/or trade id if available; no aggressor flag assumed). The join of this file against the L3 stream is what turns CANCEL/MODIFY events into labelled executions.
Reading order for the agent: §1–§5 are shared infrastructure and must be built first; §6 (L2 signals) runs directly on the L2 file; §7 (L3 signals) runs on the canonical tape produced by §4; §8–§10 define combination, calibration and acceptance.
§1. Global conventions (read first — every later section assumes these)¶
1.1 Prices are integers. All prices are stored and compared as 64-bit integers in units of one thousandth of the quote currency ("milli"). The smallest tick on the venue is 0.001, so every legal price is exactly representable. Reason: floating-point price equality is the single most common bug class in order-book code; the branch logic in §6.1 and the trade-inference join in §4.3 depend on exact == comparisons. A static check must prove no float price comparison exists anywhere. Convert once, at file ingest.
1.2 Sizes are integer shares. On the main board all resting size is in multiples of the per-stock board lot. Odd lots trade through a separate mechanism and are not in these files.
1.3 Ordering is exchSeq. Events are processed strictly in exchange-sequence order; ties (if the recording ever duplicates a seq) broken by file order. Never sort by timestamp — recording timestamps carry capture jitter; exchSeq is the exchange's own total order and it is what makes the L2 file and the L3 file line up row-for-row. Any "prevailing book at trade time" lookup keys on exchSeq (greatest snapshot seq strictly below the trade's seq), never on ts; the timestamp path (§4.3 mode B) exists only for the case where the trade file has no seq.
1.4 Sides. BID = +1, ASK = -1. For inferred executions (MO events on the tape) side is the aggressor side — the opposite of the resting order that was consumed.
1.5 Sign convention. Every directional output is oriented so positive = buy pressure = expected upward mid move. If a formula naturally produces the opposite sign, negate it at the source. One automated sign-audit test covers the whole library (§10, T-12).
1.6 Missing-level sentinel. An absent level is px = 0, sz = 0. On the bid side integer comparisons still behave correctly (0 is "worse" than any real bid). On the ask side they do NOT (0 would read as "better", since lower = better) — every ask-side price comparison carries explicit empty-state branches (§6.1 shows the pattern; reuse it everywhere).
1.7 Emissions. Each signal emits {name: float} at bucket close. NaN = "no value this bucket"; the runner drops NaNs and never forward-fills (forward-filled signals fake autocorrelation and leak stale state into evaluation).
1.8 Determinism. Same three input files ⇒ bit-identical outputs. No wall-clock reads, no unordered-container iteration in output-affecting paths, fixed seeds for anything stochastic.
§2. Market environment and the tick regime¶
2.1 Spread (tick) table. The venue's tick is price-banded, milli-units, bands (low, high]:
# KEEP IN CONFIG, NOT CODE — versioned as tick_table_version. Populate from the venue's
# published spread table. Any announced band change must flip the config, which must
# invalidate every calibration artefact (artefacts are keyed by tick_table_version, §8).
BANDS = [(band_high_milli, tick_milli), ...] # shape only — the real values live in config
def tick(px_milli: int) -> int:
for band_high, t in BANDS:
if px_milli <= band_high:
return t
raise error("price out of range")
2.2 Tick-regime classification — measured, not assumed. rtick = tick(prev_close)/prev_close is only the input. The operative definition is behavioral: a stock is large-tick when the tick binds — when the quoted spread is pinned at its minimum. Whether a given rtick binds depends on volatility, so the regime is measured nightly per symbol from the L2 file:
# nightly per sym, continuous sessions only, time-weighted
rtick = tick(close) / close
sbar = time_weighted_mean(spread / tick) # mean spread in ticks
p1 = time_weighted_fraction(spread == tick) # how often the tick binds
eta = n_continuations / (2 * n_alternations) # over successive mid-price changes:
# low eta = bounce-dominated = large tick
classify:
LARGE if p1 >= 0.85 # equivalently sbar <= ~1.2
SMALL if sbar >= 2.5 # p1 <= ~0.5, eta drifting toward >= ~0.5
INTERMEDIATE otherwise
# hysteresis : reclassify only after 5 consecutive sessions on the other side
# band edges : flag EDGE when |price - nearest band boundary| / price < 2% —
# rtick is discontinuous there (a name just below a boundary: ~4bp below, ~10bp above it);
# gate reclassification and force a re-fit when a boundary is actually crossed
# reforms : the artefact is keyed by tick_table_version; a tick-table change re-measures everything
sbar or p1 as interaction terms — because the regime is a dial, not a switch (§2.6).
Venue finding and its two consequences. Measured on this board, the classifier returns LARGE for essentially every liquid main-board name. That is structural, not a bug: the spread table's relative-tick floor is ~4bp at every band top, while a liquid large-cap's economic spread is ~1–3bp — the tick binds almost everywhere (which is precisely the motivation for the venue's own tick-reduction reform program). Consequences:
1. ILLIQUID pre-label, applied before the LARGE/SMALL logic. On this venue a floating spread has two causes: a genuinely non-binding tick, or nobody quoting. Rule: ILLIQUID if ADV20 < adv_floor (default: 20th percentile of the universe) or fraction of continuous-session time with both sides quoted < quote_floor (default 0.95). ILLIQUID names are excluded from the directional stack entirely — never routed to the deep-OFI/slope family, which has no depth to read there. SMALL must mean "fine grid", never "empty book".
2. Stratify within LARGE. With one label covering the board, pooling by label is one giant bucket, but real variation survives inside it — a 100bp-tick name pinned 99.9% is a different animal from a 4bp name pinned 88% with regular 2-tick excursions. Pooling key for every per-band calibration: (rtick band {≥50bp, 10–50bp, <10bp} × sbar decile). The §2.6 small-tick column does not disappear: it applies to floating episodes (sessions where a normally pinned name's sbar drifts above ~1.5) and to whatever population a future tick-table change migrates.
2.3 Sessions — message-driven where possible. The trading day: an opening call auction; a continuous AM session; a midday break; a continuous PM session; a closing call auction with a random close. Half-days: AM only, then the closing auction. session_state ∈ {PRE, OPEN_AUCTION, AM, BREAK, PM, CLOSE_AUCTION, CLOSED}. If the capture includes trading-session-status and security-status messages, drive session_state from them and use the clock only as fallback — the feed carries a session-control flag that flips to manual precisely when the normal schedule is invalidated (unscheduled closures, exchange intervention), and per-security halts arrive only via the security-status stream. Signals accumulate only in AM and PM. The auctions have no continuous matching, so every relation calibrated on continuous trading is mis-specified there; auction-window rows in all three files go to a quarantine stream (kept for research, never fed to signals). Additionally, per the feed spec, full order-book messages are not disseminated at all during the auctions — see 2.4 for the consequence at the continuous open.
2.4 Reset policy — signal state vs book state. All mutable signal state (accumulators, prev pointers, EW estimators, ring buffers, Hawkes states, pattern detectors) resets on: (1) each continuous-session open (AM and PM); (2) the first event after any exchSeq gap in the L3 file; (3) both edges of a volatility-halt cooling-off; (4) trading-halt resume; (5) any sequence-reset message in the capture. Calibration artefacts (read-only, per-date) persist. The first transition after any reset produces zero flow; a time bucket spanning a reset boundary is discarded, not truncated.
The order book itself is NOT signal state: resting orders persist across the midday break, so the §4 book must not be cleared at the PM open — only on sequence reset / start of day. Because the order-by-order feed sends no book data during auctions, the standing book (post-auction residual orders) arrives at the AM open as a replay burst of ADD messages: this burst is book establishment, not order-flow information. Rule: suppress all flow accumulation (LO/CX/MO flows, §6.1/§7.1 etc.) for warmup_s (default 5 s) after the AM session open or until the first automatch trade, whichever comes first; the book builder of course processes the burst normally. No such burst exists at the PM open (the book carried over).
2.5 Volatility halts. Some names are subject to a dynamic price-band volatility control mechanism with a cooling-off period on trigger. If the capture includes the trigger messages, take the cooling-off window (start, end) and price band directly from them; otherwise supply intervals externally. Directional signals must not run inside a cooling-off; both edges are reset boundaries.
2.6 Tick-regime playbook — what to expect at each pole, and where this knowledge comes from.
The mechanism. Beneath the quoted book there is an efficient price diffusing with some volatility, and an economic spread — what makers would quote if they could quote anything: adverse-selection plus inventory cost per round trip, which scales with volatility per trade. The entire classification is one ratio: tick vs economic spread. When the tick is several times the economic spread, the quoted spread cannot compress to fair value: it pins at one tick, quoting at the touch earns rent, rent attracts size, queues grow long, and — since nobody can undercut by less than a tick — competition migrates from price to time priority. Prices go sticky: the efficient price wanders inside the one-tick cell while the book doesn't move; when it crosses a boundary the mid jumps a full tick and frequently bounces back (that bounce is what eta measures). All the information concentrates in the state of the queues — their relative sizes, depletion rates, composition — because those literally tell you which cell boundary the efficient price will cross first; the microprice is the sub-tick estimate of where inside the cell it currently sits. When the tick is well below the economic spread, everything inverts: the spread floats at multiple ticks, undercutting is nearly free, nobody accumulates queues, depth smears thin across many levels, the mid tracks the efficient price almost continuously, and the information lives in flows and aggregates, not in any single level.
What works where:
| Signal family | Pinned large-tick pole | Diffusive small-tick pole |
|---|---|---|
| L1 queue imbalance + logistic (§6.5) | Primary predictor — the next mid move is the race between two queues | Weak: L1 is thin, fleeting, constantly refreshed |
| Microprice (§6.8) | Maximal value — up to half a tick of adjustment, leads the mid | Collapses onto the mid; adjustment ≈ 0 |
| Queue race / intensities (§7.4) | Is the mechanism itself: price moves iff a best queue dies | Ill-posed: price moves without depletion |
| Best-level OFI (§6.1) | Works; fat depth ⇒ small β — normalize carefully | Works; thin depth ⇒ large β, noisier per event |
| Deep MLOFI / integrated OFI (§6.2–6.3) | Adds a lot: deep queues are persistent, real interest; index-keyed is safe | Aggregation mandatory: single levels are noise, levels shift (price-keyed/banded); PCA carries the signal |
| EBK decomposition (§7.1) | High value: cancels at the touch rival trades, and cancel flow leads | Trade flow dominates; LO/CX terms fade |
| Trade flow / sweeps / size buckets (§6.10, §7.2–7.3) | Trades rarer per message; a 2-level sweep is violent, information-dense | The workhorse family: TFI + propagator carry the P&L accounting |
| Age / fleeting / concentration (§7.5–7.7) | Strong: queue composition decides who survives to the fill | Depth too transient for composition to mean much |
| Pull / spoof fade (§7.9) | Pays: displayed size moves QI and micro, so spoofing pays, so detection pays | Single-order display carries little value; manipulation shifts to cross-level layering |
| Reload / iceberg (§7.10) | Clean to detect (persistent levels) and meaningful (priority valuable) | Interest sliced across prices — catch via clip regularity (§7.11), not reloads |
| Resiliency (§7.15 / §6.12) | Event-time depletion→replenishment race | Clock-time spread/shape recovery |
| Monetization | Maker-side: join/lean/cancel; payoff quantized to 1 tick; queue position is the edge | Taker-feasible when edge > thin spread; impact cost dominates |
Modelling corollary. At the pinned pole, mid changes are rare, discrete, zero-inflated — model next-move direction with logistic / hitting-probability machinery in event time. At the diffusive pole, returns are near-continuous — linear regressions of returns on flows in clock time are well-specified.
Provenance — kept honest in three tiers. Direct empirics: the QI split (Gould–Bonart 2015: strong on large-tick, moderate on small-tick), hitting-probability framing (Lipton–Pesavento–Sotiropoulos), the microprice contrast (Stoikov's own large- vs small-tick example pair), the queue-reactive premise (Huang–Lehalle–Rosenbaum, validated on large-tick CAC names), depth-in-OFI helping large-tick most (Xu–Gould–Howison) while integrated-OFI's lift was measured on a mostly small-tick Nasdaq universe (Cont–Cucuringu–Zhang), the event-type asymmetry split by tick size (Eisler–Bouchaud–Kockelkoren), the bounce/η machinery (Robert–Rosenbaum; Dayri–Rosenbaum), the economic-spread ∝ volatility-per-trade relation this whole section hinges on (Wyart–Bouchaud–Kockelkoren–Potters–Vettorazzo 2008), fleeting liquidity (Hasbrouck–Saar), spoofing-pays-on-large-tick (Lee–Eom–Park, KRX), and tick-change natural experiments (US Tick Size Pilot, JPX 2014, and this venue's own tick reforms). Synthesis: the large-tick chapter of Bouchaud–Bonart–Donier–Gould, Trades, Quotes and Prices. Extrapolation/practitioner folklore: the participant-ID queue reading, reload directionality, sweep rarity-value asymmetry — thin or venue-mismatched academic support. Every magnitude above was measured on Nasdaq, Paris, Korea or crypto; nobody has published this venue's numbers. That is the design rationale for §2.2's nightly measured classification and §9.5's per-band slicing: the library re-derives this table empirically on our own market instead of trusting the transfer.
§3. Data contracts — the three recorded files¶
All three are converted at ingest to the integer conventions of §1 and merged by exchSeq. Field semantics below are aligned with the exchange's binary feed specification (order-by-order tier).
3.1 L3 order-event file (recorded from the exchange; the only stream that mutates the book):
L3Event:
exchSeq: int64 # per-channel message sequence number, strictly
# increasing within the day; a sequence-reset message starts the day at 1.
# A security lives on one channel for the whole day, so per-symbol
# ordering by exchSeq is total.
ts: int64_ns # capture timestamp (jittery; never used for ordering)
msg: ADD | MODIFY | CANCEL # = the feed's add / modify / delete order messages
oid: int64 # OrderId: unique per SECURITY per day, values not consecutive
side: +1 | -1 # mapped from the wire encoding at ingest
px: int64 # ADD only. Wire uses implied decimals == milli-units integers (§1.1)
qty: int64 # ADD: order qty · MODIFY: NEW ABSOLUTE quantity
otype: MARKET | LIMIT | absent # ADD only; informational
3.2 L2 snapshot file (our own rebuild, one row per book change):
L2Snapshot:
exchSeq: int64 # equals the exchSeq of the L3 event that produced this state
ts: int64_ns
bp[10], bs[10] # bid px/sz, index 0 = best, px descending
ap[10], as_[10] # ask px/sz, px ascending
bct[10], act[10] # optional per-level order counts. Note: the exchange-side
# aggregate book message carrying NumberOfOrders exists only on
# the aggregated-book tiers, not the order-by-order tier — so on this recording, counts are
# DERIVED (count of resting oids per level in the §4 book).
bp strictly decreasing, ap strictly increasing, bp[0] < ap[0] (else flag CROSSED and drop), sz > 0 ⟺ px > 0.
Because this file shares exchSeq with the L3 stream, it serves three roles: (a) direct input to all §6 signals — no book replay needed on the L2 path; (b) O(1) "prevailing book at seq s" lookup = row with the greatest exchSeq ≤ s (used by trade inference and §6.10); (c) ground truth for the builder cross-check T-6.
3.3 Trade file (= the feed's trade / trade-cancel stream; no aggressor flag, no order references):
Trade:
exchSeq: int64 # every Trade message carries the channel seq like any other
# message -> mode A (exact) inference applies on this capture
ts: int64_ns # TradeTime, microsecond precision on the wire
trade_id: int64 # TradeID: per security, starts at 1, increments by 1 per trade
# -> gap in TradeID == capture loss (acceptance TR-4)
px: int64_milli
qty: int64
trd_type: enum # venue trade-type code, mapped at ingest to
# AUTOMATCH | AUTOMATCH_INTERNAL | AUCTION |
# OFF_BOOK (the various off-exchange report types) | ODD_LOT
TradeCancel: exchSeq, ts, trade_id # nets against the original trade by TradeID
AUTOMATCH / AUTOMATCH_INTERNAL — these and only these consumed book liquidity → eligible for the book join.
- AUCTION → quarantine with the auction windows; never joined against the continuous book.
- OFF_BOOK (off-exchange / external reports) → route to a separate offbook_trades stream. They never touch the book or the reconciliation, but keep them: block prints are information (a natural input to §7.11's metaorder context).
- ODD_LOT → drop from book context (odd lots are not in the main book).
A TradeCancel emits a negative-qty MO correction on the tape carrying the original cluster id — every accumulator in §6–§7 must accept negative qty.
3.4 Canonical event tape (output of §4; the single derived source of truth for §7):
TapeEvent:
exchSeq, ts, etype: LO | CX | MO, side, px, qty, oid, cluster_id,
label: CONFIRMED | INFERRED_SEQ | INFERRED_TIME | AMBIGUOUS
# LO = limit add · CX = cancel/reduction · MO = inferred execution (side = AGGRESSOR)
# label semantics:
# CX + CONFIRMED : removal with no matching automatch trade quantity — a true cancel
# MO + INFERRED_SEQ : removal matched to a trade record by exchSeq (mode A — the norm here)
# MO + INFERRED_TIME : matched by (px, time-window, qty) only (mode B fallback) — lower confidence
# AMBIGUOUS : removal at the touch that could not be reconciled either way
# Handling rule (conservative, applied by every §7 signal): AMBIGUOUS events are excluded
# from BOTH trade-flow and cancel-flow accumulators and counted in a diagnostics stream.
Not available on this recording (inactive modules, kept stubbed behind flags): participant-ID data — where a venue disseminates it at all, it is typically a conflated side feed, unsynchronized with the streaming book, so even if captured later it must never be seq-joined, only used as the slow conditioner of §7.13; odd-lot order stream; auction order detail (the order-by-order feed sends no book data during the auctions at all).
§4. Book builder + trade inference (L3 + trade file → tape + derived L2)¶
Purpose. (a) Reconstruct the exact book with the exchange's priority semantics; (b) relabel the fills: decide, for every CANCEL/MODIFY-down, whether it was a true cancel (CX) or an execution (MO), by reconciling against the automatch trades; (c) emit the canonical tape and a derived top-10 that must match our L2 file row-for-row.
4.1 State.
orders: dict oid -> Order(
side, px, qty, ts_add, prio_ts,
meta = dict(
dist_at_post, # ticks from same-side touch when posted — §7.9
lvl_qty_at_post, # level quantity at that moment — §7.9
exec_qty = 0)) # cumulative INFERRED executed quantity — §7.9
ladder[side]: sorted map px -> FIFO(oids), cached level_qty & level_count # O(1) top-10
pending: claim buffer — automatch trades awaiting consumption, plus provisional
removals inside their open claim window # §4.3
prio_ts = when the order gained time priority. Because price changes arrive as CANCEL+ADD (fresh oid, §3.1), prio_ts is simply ts_add in the normal flow — the field exists for the defensive branch below and for symmetry with §7.5–§7.6.
4.2 Order-message handlers (removals are emitted as provisional CX until §4.3 rules):
def on_add(m):
orders[m.oid] = Order(m.side, m.px, m.qty, m.ts, prio_ts=m.ts,
meta=dict(dist_at_post=ticks_from_touch(m.side, m.px),
lvl_qty_at_post=ladder[m.side].level_qty(m.px),
exec_qty=0))
ladder[m.side].push_back(m.oid, m.px)
emit(LO, m.side, m.px, m.qty, m.oid)
def on_modify(m): # Feed: quantity only — no price branch exists
o = orders[m.oid]
if m.qty < o.qty:
# Reduction RETAINS time priority. Covers BOTH a voluntary size cut AND a partial
# fill of the front order — §4.3 decides which, via the trade join.
emit_provisional_CX(o.side, o.px, o.qty - m.qty, m.oid, at_front=is_front(o))
o.qty = m.qty # FIFO position unchanged
else:
# Should not occur: priority-losing changes are disseminated as CANCEL+ADD.
# Defensive handling: treat as lose-priority (CX all + LO new at back) and count it;
# a nonzero counter means the capture violates the spec assumption — investigate.
emit(CX, o.side, o.px, o.qty, m.oid, label=CONFIRMED)
emit(LO, o.side, o.px, m.qty, m.oid)
ladder[o.side].move_to_back(m.oid); o.qty = m.qty; o.prio_ts = m.ts
o.meta = recapture(o); ct_defensive_modify_up += 1
def on_cancel(m):
o = orders.pop(m.oid) # px/qty ALWAYS resolved from the map (not on the wire)
emit_provisional_CX(o.side, o.px, o.qty, m.oid, at_front=is_front(o))
ladder[o.side].remove(m.oid)
4.3 Trade inference — the labelling pass. One single exchSeq-ordered sweep over the merged (L3 events + automatch trades) stream; the TrdType routing of §3.3 has already removed everything that never touched the book.
Mode A — the norm on this capture (trades carry exchSeq). The trade print and the removal events it caused are near-adjacent on the same channel sequence — but their relative order is not guaranteed: the removals may precede or follow the print. A trade at seq s therefore claims provisional-CX events at its price within a symmetric window [s − K, s + K] (K = seq_claim_window, default 20 messages):
1. group = trade (px=p, qty=q, seq=s) + unclaimed provisional-CX events at price p
with seq in [s-K, s+K]
2. passive side S = the side whose best price == p in the book state immediately before
the EARLIEST member of the group (print or first candidate removal, whichever is lower).
If BOTH bests == p (transient locked state): S = the side that actually has candidate
removals at p in the group; if both sides do, increment `ambiguous_side_ct`.
3. consume candidates on side S in seq order (= FIFO-front order by construction),
relabelling them MO (side = -S = aggressor, label = INFERRED_SEQ) and adding to each
order's meta.exec_qty, until the cumulative relabelled quantity == q.
A partial fill is a MODIFY-down of the FRONT order whose delta completes q exactly.
4. reconciliation: consumed quantity must equal q exactly. Shortfall or overshoot ->
the involved events get label = AMBIGUOUS and the residual is logged.
# day one: measure the empirical print-vs-removal ordering and tighten K to the observed
# bound — a smaller K shrinks the AMBIGUOUS surface.
(p, q, t) may claim provisional-CX events at price p on the qualifying side within t ± match_window (default 100 ms), matched greedily in seq order under the same FIFO-front and quantity-conservation constraints; relabelled events get INFERRED_TIME. Everything unclaimed at window expiry finalizes as CX, CONFIRMED.
Finalization. A provisional CX finalizes as CX, CONFIRMED once the sweep's current seq passes event.seq + K with no claim (mode A), or when its time window expires (mode B). Sweeps arrive naturally: an aggressive order walking the book produces trade prints at successive prices, each consuming that price's front events — clustering (§4.4) stitches them back into one parent.
Why this is safe where it matters. Mislabelling MO↔CX cannot corrupt the §6 family at all — snapshot-based OFI, imbalance and microprice are label-blind. It also cannot corrupt the queue-race drain (§7.4): λ_MO and λ_CX enter only through their sum. The label-sensitive signals are §7.1 (MO/LO/CX split), §7.2–7.3 (trade flow), §7.6 (fleeting — CONFIRMED cancels only, else a missed fill masquerades as a sub-second cancel), §7.9 (exec_qty), §7.10 (executions at a level) and §6.13 — exactly the set gated by the reconciliation tests below.
4.4 Aggression clustering. Over inferred MO events: same aggressor side and (same exchSeq burst or inter-event gap ≤ 1,000,000 ns) ⇒ one cluster_id. Cluster attrs maintained online: qty, n_px (distinct prices), px_range. Used by §7.2, §7.3, §7.11.
4.5 Derived snapshots and the cross-checks. After every event, if the top-10 changed, emit a derived L2 row keyed by the event's exchSeq.
Acceptance T-6 (strong form). The derived top-10 must equal our recorded L2 file exactly, row-for-row at every shared exchSeq (both derive from the same L3 stream — any mismatch is a defect in one of the two builders; investigate, never tolerate). Exclusions: quarantined auction windows, the opening replay burst, the first row after an exchSeq gap.
Acceptance TR-1. Per session per symbol: Σ inferred-MO quantity == Σ automatch trade quantity (automatch types only), exactly in mode A; within 0.1% in mode B.
Acceptance TR-2. Every inferred MO sits at a price equal to the prevailing best of its passive side. Violations = 0.
Acceptance TR-3. AMBIGUOUS quantity < 0.1% of automatch traded quantity (mode A) / < 1% (mode B); above that, halt and inspect before trusting any label-sensitive §7 signal.
Acceptance TR-4. Per security, observed TradeIDs form 1..N with no gaps (net of TradeCancels). A gap = trade-capture loss → the label-sensitive family is quarantined for the session.
Fixture BB-2 (priority semantics). add oid1 = 500,000 @ 10,000 → MODIFY qty→100,000 ⇒ emit provisional CX 400,000, prio_ts unchanged, FIFO position unchanged. A price change then arrives from the exchange as CANCEL oid1 + ADD oid2 @ 10,010 ⇒ emit CX 100,000 @ 10,000 (CONFIRMED once unclaimed) then LO 100,000 @ 10,010 with oid2's own prio_ts.
Fixture TI-1 (inference). Book: best ask 10,020 with FIFO [oidA 3,000; oidB 5,000]. Trade file: automatch (px 10,020, qty 4,000) at seq s. L3 shows at ~s: CANCEL oidA; MODIFY oidB → 4,000. Expected tape: MO(aggr=BUY, 3,000, oidA, INFERRED_SEQ), MO(aggr=BUY, 1,000, oidB, INFERRED_SEQ); oidB keeps priority with exec_qty = 1,000; reconciliation exact.
§5. Normalizers, seasonality and targets¶
All estimators run in continuous-session time only and reset per §2.4.
5.1 Exponentially-weighted mean (implement once, reuse everywhere):
5.2 Depth and event-size scales.
Q_L1 = ewm of (bs[0] + as_[0]) / 2, halflife = 30 min # best-depth scale
Q_TOP = ewm of sum(top-10 depth both sides), halflife = 30 min # book-depth scale
AES = rolling median of |qty| over LO and MO tape events, 30-min window
5.3 Seasonal z-score zs(x). A z-score where "normal" is defined by time of day, not by the whole day pooled: it answers "how unusual is this value for this time-of-day slot on a typical day", not "versus the overall average".
Mechanics: cut the trading day into 5-minute bins. Nightly, per feature × symbol × bin, compute the mean and std of that feature in that bin over the trailing 20 sessions (one (mean, std) pair per bin per feature per symbol across the two sessions). Online, an observation is scored against its own slot's baseline only:
Why: almost everything in a limit order book has a deterministic intraday shape — message rates and volumes are huge at the open, sag through midday, spike into the close; depth builds through the morning; spreads are widest in the first minutes. Concretely: a name whose cancel volume runs ~200k/min just after the open but ~40k/min mid-afternoon — a mid-afternoon reading of 150k is a violent withdrawal (~4σ for that hour), while the same 150k just after the open is below its slot's average. A whole-day z blurs both toward "roughly normal" and, worse, fires systematically at every open and close purely because of the clock. The seasonal version subtracts the clock first, so what remains is genuine surprise.
Consequences downstream: gates are only well-defined on zs (zs(msg_rate) > 3 is a meaningful burst definition because it is measured against the slot-specific baseline — on raw values it would trip every morning by construction); the combiner's coefficients stop soaking up time-of-day instead of signal; and cross-name pooling (§9.5) becomes legitimate, since every feature is in its own σ units rather than its own depth and volatility scale. Every directional emission is also published in zs form; gates (§7.8, §7.14) consume zs only. Winsorization at ±8 is the single outlier choke point for the heavy-tailed features.
Caveat: the bins carry 20 sessions of memory, so the first days after a regime change (a tick-band crossing, a tick-table change) are scored against old-regime baselines — one more reason those events force re-baselining (§2.2, §9.5) rather than letting the rolling window catch up quietly.
5.4 Evaluation targets (never features). Forward log-mid returns at h ∈ {1, 5, 10, 30, 60, 300} s and, in event time, {next mid change, +10 events}; plus microprice-marked variants (§6.8) for market-making evaluation. Mid/micro built from the L2 file.
§6. L2 signals — run directly on the recorded L2 file¶
These consume L2 rows keyed by exchSeq (no book replay needed) and are immune to trade-inference errors — they never use the MO/CX labels. Every subsection is self-contained: inputs, parameters, algorithm, calibration, output, edge cases, fixture. Every header carries a role tag — directional / gate / conditioner / diagnostic — with its natural horizon; §8.0 is the consolidated map and the runner enforces the routing.
§6.1 Best-level OFI — ofi_l1 (role: diagnostic + impact β contemporaneous · directional (seconds) when lagged)¶
Computes. Net signed change in resting size at the best bid and ask over a bucket. Why it works: short-horizon mid moves are approximately linear in this net flow, with slope ∝ 1/depth; a market sell and a bid cancel of equal size are deliberately treated identically — both drain the bid queue, and at L2 that is all that matters (which is precisely why no trade labels are needed here).
Inputs: consecutive L2 rows. Params: bucket_ms = 1000, norm = Q_L1. State: prev row; bucket sum.
def ofi_contrib(prev, cur) -> int:
# BID side — better = HIGHER price. Sentinel px=0 compares as "worse": raw ints are safe.
if cur.bp[0] > prev.bp[0]: eb = cur.bs[0] # improved (or empty -> real)
elif cur.bp[0] == prev.bp[0]: eb = cur.bs[0] - prev.bs[0] # same price: size delta
else: eb = -prev.bs[0] # retreated (or real -> empty)
# ASK side — better = LOWER price. Sentinel px=0 would wrongly read as "better",
# so the two empty-state transitions get explicit branches:
if prev.ap[0] == 0 and cur.ap[0] > 0: ea = cur.as_[0] # empty -> real: new sell liquidity
elif cur.ap[0] == 0 and prev.ap[0] > 0: ea = -prev.as_[0] # real -> empty: sell liquidity gone
elif cur.ap[0] < prev.ap[0]: ea = cur.as_[0]
elif cur.ap[0] == prev.ap[0]: ea = cur.as_[0] - prev.as_[0]
else: ea = -prev.as_[0]
return eb - ea # positive = buy pressure
# per bucket: ofi = sum(ofi_contrib over valid transitions) / Q_L1
exchSeqs, neither row CROSSED, prev exists.
Output: ofi (unbounded, depth-normalized; + = up). Since our L2 has one row per book change there is no conflation; if a downsampled variant is ever used, the net remains correct for buckets ≥ the sampling interval — only the gross decomposition is lost (recovered by §7.1).
Fixtures (must reproduce exactly): 1. prev (bid 10000×5000, ask 10020×4000) → cur (bid 10000×6000, ask 10020×3500) ⇒ (6000−5000) − (3500−4000) = +1500 2. same prev → cur (bid 9990×2000, ask 10010×1000) ⇒ eb = −5000, ea = +1000 ⇒ −6000 3. prev ask empty → cur ask 10020×4000 ⇒ ea = +4000 ⇒ −4000
§6.2 Multi-level OFI — mlofi (10-vector) (role: feature vector for §6.3 · directional when lagged)¶
Computes. The §6.1 flow at each of the 10 levels. Why: deeper levels carry real signal (the old "best level suffices" result was a regression-collinearity artefact), and the gain is largest for large-tick names — most of this market (§2.6).
Two modes, both required; mode in config:
def mlofi_index(prev, cur): # convention used by the PCA in §6.3 — the literature standard
# apply the §6.1 branch logic INDEPENDENTLY at each index m = 0..9 on each side
return [bid_branch(prev, cur, m) - ask_branch(prev, cur, m) for m in range(10)]
def mlofi_price(prev, cur): # robustness mode — immune to the level-shift artefact
for side in (BID, ASK):
# visible-window overlap bound: exclude prices that merely entered/left the BOTTOM
# of the 10-level window because the whole book shifted
bound = max(prev.p[9], cur.p[9]) if side == BID else min(prev.p[9], cur.p[9])
for px in union(nonzero_prices(prev, side), nonzero_prices(cur, side)):
if inside(px, bound, side): # px >= bound (BID) / px <= bound (ASK)
dsz[px] = cur_sz(side, px) - prev_sz(side, px) # missing -> 0
bins[side] = aggregate dsz by ticks-from-current-best into 10 bins
return bins[BID] - bins[ASK]
v = v_raw / Q_TOP. Output: v[10] per bucket.
Fixture MLOFI-1 (level shift). prev bids (10000×5000),(9990×3000),…,(9910×·); cur bids (10010×2000),(10000×5000),…,(9920×·): index mode ⇒ v[0]=+2000, v[1]=+5000 (definition artefact of the literature convention — assert it); price mode ⇒ banded total +2000 only — the 9910 level fell off the window bottom and is correctly excluded. This is the guard against the classic spurious-deep-OFI bug.
§6.3 Integrated OFI — ofi_int (PCA aggregation) (role: directional · seconds–minutes)¶
Computes. One scalar from the collinear 10-vector, weighted by its first principal component. Why: the per-level OFIs are highly correlated; the first PC captures the common flow factor, and empirically the deeper levels — not the best — carry the stable weights.
Offline job (nightly, pre-open), per symbol:
# 1. collect 1-minute mlofi_index vectors over the trailing 10 sessions, each divided by
# the bucket's Q_TOP (scale BEFORE the PCA, else intraday depth seasonality
# contaminates the weights)
# 2. if fewer than 500 vectors: fall back to equal weights w = [0.1]*10 and flag
# 3. C = covariance(X) # X: N x 10
# 4. w1 = eigenvector of C with the largest eigenvalue
# 5. w1 = w1 * sign(sum(w1)) # orient: positive = buy pressure
# 6. w = w1 / sum(abs(w1)) # L1-normalize: output stays in OFI units
# 7. persist keyed (sym, date, tick_table_version)
ofi_int = dot(w, v) per bucket.
Acceptance CAL-1: out-of-sample contemporaneous 1-min R²(ofi_int) ≥ R²(ofi_l1) on ≥ 70% of the liquid-20 validation universe; if not, the calibration run fails (usual causes: missing depth scaling, too few observations).
§6.4 Lagged cross-asset OFI — xofi (role: directional · 10–300 s)¶
Computes. A forecast overlay from other names' integrated OFI. Why: contemporaneous cross-impact is redundant once levels are integrated, but lagged cross-asset OFI adds forecasting power at short horizons, decaying fast.
Offline (weekly): for each target and horizon h ∈ {10, 60, 300} s, LASSO of forward return on the universe's one-bucket-lagged ofi_int (universe = index constituents). Penalty by 5-fold purged cross-validation with embargo = h (§9.3). Online: sparse dot of stored coefficients on the lagged vector. Contemporaneous cross terms are forbidden live.
§6.5 L1 queue imbalance — i1 + logistic map (role: directional · event-time → seconds)¶
Computes. i1 = (bs[0] − as_[0]) / (bs[0] + as_[0]) ∈ [−1, +1]; NaN if the denominator is 0. Why: the heavier best queue predicts the direction of the next mid move; the relation saturates near ±1; strongest exactly in the pinned large-tick regime that dominates this market (§2.6).
Calibration (nightly): at every mid-change exchSeq, record (i1 from the row immediately before, y = sign of the change). Fit p_up = 1/(1 + exp(-(x0 + x1*i1))) by maximum likelihood, per (sym, session-half); a cell with < 2000 observations pools within its tick band (§2.2). Emit i1, p_up.
Test QI-1: fitted x1 > 0; property test: p_up monotone non-decreasing in i1.
§6.6 Depth-weighted imbalance — i_w (role: directional · event-time → seconds)¶
def qi_w(s, lam): # lam from {0.3, 0.5, 1.0, 2.0}
wb = sum(exp(-lam*m) * s.bs[m] for m in range(10))
wa = sum(exp(-lam*m) * s.as_[m] for m in range(10))
return (wb - wa) / (wb + wa)
lam chosen per tick band by validation IC (§9.1): large lam ≈ pure L1 (large-tick names), small lam uses depth (small-tick names). Emit chosen i_w alongside the i1 reference.
§6.7 Order-count imbalance — i1_ct (role: directional + reliability weight on i1)¶
i1_ct = (bct[0] − act[0]) / (bct[0] + act[0]), zero denominator → NaN. Requires per-level counts in the L2 rebuild; if absent, compute from the §4 book instead (count of resting oids at the best). Also emit spoof_tell = i1 − i1_ct: a side that is volume-heavy but count-light (one big order) is the fragile/spoofable configuration — feeds the combiner as a reliability weight on i1 and cross-checks §7.9.
§6.8 Microprice — micro (role: directional · event-time; also marking/eval infrastructure)¶
Computes. A fair value inside the spread: micro = mid + g(state), g = expected cumulative future mid adjustment given the (imbalance, spread) state. Why: the queue-weighted lean of the book leads the mid; the Markov construction denoises the naive weighted mid. Maximal value on pinned names, ≈ 0 on diffusive ones (§2.6).
State: i = bin of i1 with fixed edges ±{0.8, 0.6, 0.4, 0.2, 0} (10 bins); s = spread in ticks ∈ {1, 2, 3+}.
Calibration (nightly, on the L2 event grid):
# 1. walk L2 rows; at each row note state (i, s)
# 2. from each state occurrence scan forward to the NEXT row where either
# the mid changes -> ABSORPTION : record dmid
# the state changes -> TRANSITION: record (i,s) -> (i',s')
# 3. G1[i,s] = mean(dmid over absorptions one step ahead)
# 4. B = row-normalized transition-count matrix (non-absorbing moves only)
# 5. g = G1 ; repeat 6 times: g = G1 + B @ g # require sup-change < 0.01 * tick at the end
# 6. antisymmetrize: g[i,s] = (g[i,s] - g[mirror(i),s]) / 2
# 7. any cell with < 50 observations: linear interpolation across i within the same s
micro = mid + g[state]; emit micro_adj = g[state]/tick.
Tests MP-1/MP-2: exact antisymmetry after step 6; micro ∈ [bid, ask] on ≥ 99.9% of rows.
Usage: mark inventory and passive/aggressive decisions at micro, not mid; §7.9's gate temporarily zeroes the micro_adj contribution from a side where a pull was just detected.
§6.9 VOI / OIR (Shen conventions) — voi, oir (role: directional · seconds)¶
Practitioner packaging of best-quote deltas; kept because desks compare against it.
# dB: 0 if bid px moved DOWN | size delta if unchanged | +bs[0] if moved UP
# dA: +as_[0] if ask px moved DOWN | size delta if unchanged | 0 if moved UP
# (ask-side sentinel branches exactly as §6.1)
voi = sum(dB - dA) / Q_L1 ; oir = i1
§6.10 Trade-flow imbalance — tfi (role: directional · seconds)¶
Computes. Signed executed volume — the second, orthogonal "OFI" convention (§6.1 signs queue changes; this signs trades). Keep both as separate features.
Primary path (this recording): aggressor signs come from the §4.3 inference — do not classify. Each inferred MO carries its aggressor side; tfi = Σ side·qty / ewm(bucket volume), tfi_cnt = Σ side, computed from tape events with label ≠ AMBIGUOUS.
Fallback path (only for periods where the trade join fails TR-1/TR-3, e.g. a corrupted trade file): classical classification against the L2 file —
def sign_trade(t):
q = L2 row with greatest exchSeq < t.exchSeq # or ts-window lookup in mode B
m = (q.bp[0] + q.ap[0]) / 2
if t.px > m: return +1
elif t.px < m: return -1
else: return tick_rule(t) # vs last different trade px; carry prev sign
degraded flag so the combiner can down-weight them.
§6.11 Book slope, curvature & shape — slope_asym, curv_asym, alpha_asym, shape_skew (role: conditioner — interaction-only)¶
Computes. The geometry of the visible ladder: how liquidity is distributed in price space, per side, at this instant. This is the state complement to the rest of the stack — OFI measures flow, i1 measures the L1 state, slope/shape describe the whole 10-level supply and demand curves.
Why it works — two readings that point the same way. Mechanical: the ask side of the book is literally a supply curve — cumulative depth at distance Δ is how much can be bought within Δ ticks — so its slope is local liquidity density (shares per tick) and its inverse is the instantaneous impact function: a flat side is cheap to push through, so even balanced flow drifts the mid toward the flatter side. In the locally-linear book model, the slope is the empirical liquidity parameter that sits in the prefactor of the square-root impact law. Informational: patient makers shade away from the side where they fear adverse moves — a thin ask side is partly the crowd declaring it doesn't want to be short here. Both readings give: positive slope_asym (bid steeper) ⇒ up, consistent with §1.5.
X = [abs(px[m] - mid) / tick for valid m] # distance from mid in ticks (sentinels excluded)
Y = cumsum(sz over valid m) # cumulative depth
# need >= 5 valid levels per side, else all outputs NaN
# (a) headline slope — Theil–Sen (robust): median of all pairwise slopes.
# 10 levels -> 45 pairs, trivial cost; one spoofed giant level barely moves it (fixture SL-2)
slope[side] = median( (Y[j]-Y[i]) / (X[j]-X[i]) for all i < j )
slope_ols[side] = ols_slope(X, Y) # kept for continuity with the literature
slope_asym = slope[BID] - slope[ASK] # positive = bid steeper = support
# (b) curvature — front-loaded vs back-loaded defense
# near-slope drives the immediate one-tick dynamics; far slope drives multi-tick impact
near[side] = theil_sen(X[:split], Y[:split]) # split = 5 (levels 0..4)
far[side] = theil_sen(X[split:], Y[split:])
curv[side] = far[side] - near[side] # >0 back-loaded (wall sits deep, touch is soft)
curv_asym = curv[ASK] - curv[BID] # ask soft at the touch -> up-drifts short-term
# (c) power-law fit: cumdepth(Δ) ≈ c * Δ^alpha (fit ln Y = ln c + alpha ln X by OLS)
# alpha ≈ 1 linear book · alpha > 1 back-loaded/convex · alpha < 1 front-loaded/concave
alpha[side] = clip(ols_slope(ln X, ln Y), 0.2, 3.0)
alpha_asym = alpha[ASK] - alpha[BID]
# (d) beyond-best lean, deliberately EXCLUDING L1 so it is ~orthogonal to i1:
shape_skew = (sum(bs[1:]) - sum(as_[1:])) / (sum(bs[1:]) + sum(as_[1:]))
slope_b, slope_a, slope_asym, slope_asym_ols, curv_asym, alpha_b, alpha_a, alpha_asym, shape_skew (all also in zs() form).
Fixture SL-1 (baseline OLS). Bid cumdepth over 1..5 ticks = [100, 180, 240, 280, 300] (k-shares) ⇒ OLS slope 50.0 k/tick; ask [40, 70, 90, 105, 115] ⇒ 18.5 k/tick; slope_asym_ols = +31.5 (support).
Fixture SL-2 (spoof robustness). Ask cumdepth [40, 70, 90, 105, 5115] — one fake 5M-share deep level: OLS slope = 1018.5 (inflated ~55×) while Theil–Sen = 27.5 (barely moved). This is why Theil–Sen is the headline.
Edge cases & regime notes.
- Large-tick degeneracy (§2.6): on pinned names the levels are dense and contiguous, distance ≈ level index, and slope collapses toward "average level size" — expect little incremental value there; the signal earns its keep on small-tick names where levels have gaps and spatial density is a real variable. Validate per sbar decile in §9.5.
- The hump: the average book profile peaks a few ticks from the best, not at it, so a straight-line fit averages across the rising branch — fine for the asymmetry, but do not interpret the level of the slope as a clean impact coefficient.
- Deep size is the cheapest thing in the book to fake (far from the touch ⇒ near-zero fill risk): hence Theil–Sen, plus cross-checks with §7.7 concentration and §7.9 — while suppress_qi(side) is raised, the runner zeroes that side's slope/curvature contributions exactly as it zeroes i1.
- Dynamic link: shape is the integral of cancel flow — a §7.8 withdrawal cascade is the flow event, a suddenly flattened side is the state it leaves behind; a one-sided slope collapse is the state-space echo of the cascade and a useful confirmation of it.
Usage. Conditioner / interaction terms, not standalone alpha (§8.2). The canonical interaction: modulate the impact coefficient — the same OFI against a flatter opposing side deserves a larger β.
§6.12 Cancel proxy & spread resiliency — cxprox_imb, spread_hl (role: diagnostic shadow — never routed)¶
Pure-L2 bounds kept for cross-checking the L3 path: a size drop at an unchanged price is cancels + fills; subtracting trade-file quantity at that price in the interval isolates cancels approximately. cxprox_imb = (proxy_ask − proxy_bid)/(sum).
Resiliency: spread sampled at 1 s; rolling 30-min AR(1) spread_t = a + b·spread_{t−1}; spread_hl = −ln(2)/ln(clip(b, 1e−6, 0.999)). Superseded by §7.8 / §7.15 — run in shadow, alert if the two paths diverge (a divergence means the trade join is drifting).
§6.13 VPIN — vpin (role: gate — never directional)¶
Computes. Volume-clocked |buy − sell| imbalance ≈ probability that passive quotes face informed flow. Contested as a predictor; may only throttle (§8.2) — the runner refuses to route it into a directional combiner.
V = ADV20 / 50 # bucket volume
for bar in volume_bars(inferred_MO_events, V): # split at bar boundaries
Vb = signed buy volume in bar # exact via §4.3 — the BVC approximation
Vs = bar.vol - Vb # is unnecessary on this recording
vpin = sma(abs(Vb - Vs), n=50) / V # emitted at bar close (irregular timestamps)
§6.14 Propagator drift — prop_drift (role: directional · 1 s – 10 min)¶
Why: price is past signed flow convolved with a slowly decaying kernel; the transient part predicts drift/reversion — an execution-scheduling signal more than tick alpha.
# grid 1 s ; eps_t = zs(tfi) (fallback zs(ofi))
# LAGS = {1..10, 15, 22, 32, 46, 68, 100, 147, 215, 316, 464, 600} s (log-spaced)
# offline weekly: ridge of r_{t -> t+1} on {eps_{t-l}} ; kernel G[l] = coefficients
prop_drift = sum(G[l] * eps[t - l] for l in LAGS) # online ring buffer of 600 s of eps
§6.15 Feature stack — features[·] (role: infrastructure container)¶
Fixed-order vector; flows, never raw levels (models on stationary order-flow inputs beat models on raw book levels — enforce structurally by excluding un-normalized cumulative sizes):
features = [ relpx_ticks x20, sz/Q x20, i1, i_w, i1_ct, ofi, ofi_int, v[0..9],
spread_ticks, slope_b, slope_a, curv_asym, alpha_asym, spread_hl, rv_1m, cxprox_imb,
trade_intensity, tfi ] # each in zs() form; order is part of the contract
§7. L3 signals — run on the §4 tape¶
Label discipline (applies to every subsection). MO events on this recording are inferred (§4.3). Rules: (1) events with label AMBIGUOUS are excluded from both trade-flow and cancel-flow accumulators; (2) signals marked label-sensitive below may only go live while TR-1..TR-4 pass for the session (T-8..T-10, T-22); (3) §7.4's drain uses λ_MO + λ_CX and is label-robust by construction. Role tags as in §6; consolidated map in §8.0.
§7.1 Event-type-decomposed OFI — ebk (role: directional · seconds · label-sensitive)¶
Computes. §6.1's flow split into its three mechanisms — executions (MO), fresh adds (LO), cancels (CX) — with regression-fitted weights instead of forced equality. Why: the three event types have different impact per share and different persistence; on large-tick names LO/CX flow rivals MO flow, and cancel flow often leads by a few hundred ms (withdrawal precedes the move).
Accumulation, per bucket, band = 2 ticks from the pre-event same-side touch:
f[MO][BUY] += qty of inferred executions with aggressor = buy (in band)
f[MO][SELL] += qty with aggressor = sell
f[LO][BID] += qty of adds on the bid side (in band) ; f[LO][ASK] mirrored
f[CX][BID] += qty of CONFIRMED cancels on the bid side (in band) ; f[CX][ASK] mirrored
x = [ f[MO][BUY] - f[MO][SELL], # signed trade flow
f[LO][BID] - f[LO][ASK], # signed add flow
f[CX][ASK] - f[CX][BID] ] / Q_L1 # signed cancel flow (ask cancels = bullish)
ebk = beta @ x # emit ebk AND the three components separately
beta = Ridge(1-min Δmid_in_ticks ~ x). Implementation check: equal weights must recover §6.1 up to the AMBIGUOUS residual.
Acceptance: contemporaneous R²(ebk) ≥ R²(ofi_l1) on ≥ 70% of the liquid-20 names.
§7.2 Sweep detection — sweep_vol, sweep_cnt (role: directional · seconds · label-sensitive)¶
Why: one aggressive order walking ≥ 2 price levels pays a premium for immediacy — the classic informed-flow fingerprint, visible through §4.4 clustering of inferred MOs.
sweep(cluster) = (cluster.n_px >= 2) or (cluster.qty >= p95(cluster_qty, rolling 5 sessions))
sweep_vol = sum(side * qty over sweep clusters) / AES
sweep_cnt = (#buy sweeps) - (#sell sweeps)
§7.3 Size-bucketed trade-flow — tfi_small/mid/large (role: directional · seconds · label-sensitive)¶
Why: information per share is not monotone in clip size — institutional slicers hide in medium clips (stealth trading). Bucket, and let the combiner fit the weights.
# thresholds: rolling 5-session q50 and q90 of cluster qty, per symbol
bucket(cluster) = SMALL if qty < q50 else LARGE if qty >= q90 else MID
tfi_<bucket> = sum(side * qty over that bucket's clusters) / Q_L1
§7.4 Queue-reactive race — qr (role: directional · event-time · label-robust)¶
Computes. P(best ask dies before best bid), from empirically estimated arrival/cancel/execution intensities as functions of queue size. Why: the dynamic generalisation of §6.5 — static imbalance is the special case of equal intensities; on pinned names this is the price-formation mechanism (§2.6).
State: per side, q = min(ceil(best_size / AES), 30).
# CALIBRATION (weekly, per symbol, continuous-session time only):
# just BEFORE each tape event touching a best queue, note (side, q, type in {ADD, CX, MO});
# between consecutive events add the elapsed time to the occupancy clock of the occupied (side, q).
lam[type][side][q] = (count[type][side][q] + 0.5) / (occupancy_time[side][q] + eps)
# ONLINE:
mu_s = lam[CX][s][q_s] + lam[MO][s][q_s] - lam[ADD][s][q_s] # net drain — NOTE: the
# MO/CX split cancels out of the sum, so mislabelled fills cannot bias the race
r_s = max(mu_s, 0) / q_s
p_bid_first = r_b / (r_b + r_a) # NaN if both r <= 0 (both queues net-growing)
qr = p_ask_first - p_bid_first # positive = ask breaks first = up
§7.5 Order-age-weighted imbalance — awi (role: directional · event-time → seconds · label-robust)¶
Why: aged resting orders are patient capital; sub-second orders are noise. Weighting depth by age (exact via prio_ts from §4) cleans both queue imbalance and the microprice.
# top M = 3 price levels per side; tau = 30 s
w(o) = o.qty * (1 - exp(-(now - o.prio_ts) / tau))
awi = (sum_bid(w) - sum_ask(w)) / (sum_bid(w) + sum_ask(w))
# an all-fresh book gives 0/0 -> NaN (guarded) — correct: no age information exists yet
prio_ts, not ts_add: an order that lost priority (§4.2) is economically new.
§7.6 Fleeting-liquidity ratio — fleet_imb (role: directional · seconds · label-sensitive — CONFIRMED cancels only)¶
Why: the share of added volume cancelled within τ measures how much displayed depth is real; the mid moves through a fleeting-heavy side.
# rolling 5-minute window, per side; tau_f = 500 ms
fleet[side] = qty with label CX-CONFIRMED and (event_ts - prio_ts) < tau_f / qty added
fleet_imb = fleet[ASK] - fleet[BID] # fleeting ask -> up-fragile -> positive
§7.7 Depth concentration — frag_imb (role: directional · event-time → seconds · label-robust)¶
Why: a level held by one order vaporises with one cancel; the same size across thirty orders does not — the exact quantity §6.7 could only proxy.
frag[side] = max(order qty at L1) / L1 level qty # (0, 1]; 1 = one order holds the level
hhi[side] = sum((order_qty / level_qty)^2 at L1)
frag_imb = frag[ASK] - frag[BID] ; hhi_imb = hhi[ASK] - hhi[BID]
§7.8 Cancel-flow imbalance & withdrawal cascades — cx_imb, casc (role: directional · seconds · label-sensitive)¶
Why: liquidity withdrawal leads price. (a) net cancel pressure between sides; (b) cancel intensity spiking on the very side aggressive flow is hitting — makers stepping away, continuation likely.
CX[side] = ewm(CONFIRMED-cancel qty within 2 ticks of touch, halflife = 10 s)
cx_imb = (CX[ASK] - CX[BID]) / (CX[ASK] + CX[BID])
casc_dn = zs(CX[BID]) * indicator(sell-aggressor MO qty over last 5 s > 0.5 * AES)
casc_up = zs(CX[ASK]) * indicator(buy -aggressor MO qty over last 5 s > 0.5 * AES)
casc = casc_up - casc_dn
§7.9 Pull / spoof-pattern fade — pull + gate (role: directional reversal pulse + gate · label-sensitive via exec_qty)¶
Why: large size posted near the touch and cancelled unexecuted within a short lifetime was pressure that was never real. Fade the advertised direction and temporarily distrust the imbalance signals it inflated. (Detection only — generating such patterns is layering/spoofing under market-abuse rules; the naming in the codebase stays honest.)
Uses the order meta captured at post time by §4.1 (dist_at_post, lvl_qty_at_post, exec_qty — the last maintained by the inference pass).
PULL(o) = ( o.meta.exec_qty == 0 # never (inferred-)traded
and lifetime(o) < 2 s
and o.qty >= 0.30 * o.meta.lvl_qty_at_post
and o.meta.dist_at_post <= 2 )
on PULL:
pulse: pull = -o.side, exponential decay halflife 5 s
raise gate suppress_qi(o.side) for 3 s
# while raised, the runner zeroes the i1 / micro_adj contribution of that side (§8.2)
§7.10 Reload (synthetic iceberg) + exhaustion — ice_support, ice_exhaust (role: directional · seconds–minutes · label-sensitive)¶
Why: the exchange has no native iceberg/hidden types, so every reload pattern is an algorithm revealing hidden interest — support while it refills, breakout fuel the moment it stops.
# per (side, px): remember last INFERRED-execution ts and a ring(8) of add clip sizes
RELOAD = ( LO add at px within 1 s of an inferred execution at px
and clip in [prev_clip / 1.5, prev_clip * 1.5] )
ICEBERG = reload_count >= 3
EXHAUST = a fill at px NOT followed by a reload within 3 * median_refill_gap
ice_support = +1 / -1 / 0 (active bid / ask iceberg within 3 ticks of mid / none)
ice_exhaust = pulse in the breakout direction on EXHAUST
hidden_lb = reload_count * median_clip # research output only
§7.11 Metaorder / child-slicing detection — meta, meta_done (role: directional · minutes · label-sensitive)¶
Why: institutional metaorders execute as regular child clips; while active, drift continues in their direction; on completion the transient impact partially reverts.
# rolling 20-minute window per side, over §4.4 clusters on that side
ACTIVE[side] = ( n_clips >= 8
and cv(inter-arrival times) < 0.5 # cv = std / mean
and cv(clip sizes) < 0.3 )
meta = ACTIVE[BUY] - ACTIVE[SELL]
DONE: while active, gap since last clip > 3 * median gap -> pulse meta_done
§7.12 Hawkes intensity imbalance — hwk (role: directional · sub-second → seconds · mildly label-sensitive)¶
Why: order flow is self- and cross-exciting; fitted conditional intensities per event type are a de-noised, forward-looking version of the raw flow imbalances, updated O(1) per event — one state object usable by §7.4 and fill-probability models.
Types {MO_b, MO_a, LO_b, LO_a, CX_b, CX_a}; each source excites through 3 exponential timescales, half-lives {0.1 s, 1 s, 10 s} ⇒ 18 states S.
# ONLINE:
on tape event of type j at time t:
S *= exp(-BETA * (t - t_last)) # elementwise decay of all 18 states
S[the 3 states of source j] += 1.0
t_last = t
lam = clip(MU + A @ S, 0, None) # MU: 6 baselines; A: 6x18 excitation matrix
hwk = gamma @ lam # gamma: weekly Ridge of 1-s dmid on lam
# OFFLINE (weekly, per symbol): maximum likelihood.
# log L = sum_events log(lam_type(i)(t_i)) - sum_types INT_0^T lam_m dt ; with exp kernels
# the integral is closed-form: MU_m*T + sum_k (A_m,state(k)/BETA_state(k))*(1 - exp(-BETA*(T-t_k)))
# box-constrained quasi-Newton, all params >= 0; branching matrix (A_ij/BETA_j summed over
# the 3 timescales) spectral radius < 0.95 — shrink A proportionally if violated.
§7.13 Participant-ID queue composition — INACTIVE on this recording (role: conditioner — interaction-only)¶
Requires a participant-identity feed, which is not in our capture. Keep the module stubbed behind a flag with the §8.2 typing (conditioner, interaction-only) so it can be activated if such a channel is ever recorded; no other section depends on it.
§7.14 Message-rate / order-to-trade gate — gate (role: gate · label-robust)¶
Why: bursts of adds+cancels (quote stuffing, HFT skirmishes) degrade every signal; scale exposure down rather than trading through them.
msg_rate = (n_ADD + n_MODIFY + n_CANCEL) over rolling 10 s # raw L3 message rate — label-free
otr = msg_rate / max(1, n_inferred_MO)
gate = 0.25 if zs(msg_rate) > 3 else 1.0 # linear recovery to 1.0 over 30 s
# the runner multiplies EVERY directional emission by gate (§8.2)
§7.15 Event-time resiliency asymmetry — resil_asym (role: directional · seconds–minutes · label-robust)¶
Why: after a level is wiped, the exact time to rebuild reveals which side's makers are committed; the slow-rebuilding side is the fragile one. Depletion is a book fact — no labels needed.
DEPLETION = L1 level quantity reaches 0 (by any combination of removals)
on DEPLETION:
pre = ewm(that level's quantity, halflife 10 s) measured just before
dt = elapsed time until the side rebuilds to >= 0.5 * pre
at a price no worse than the old one (>= old px for bids, <= for asks)
resil_asym = median_ask(dt) - median_bid(dt) # rolling 30-min medians, ms, then zs()
# slow ask rebuild -> up-fragile -> positive, consistent with §1.5
§8. Combining signals, gates, and the calibration schedule¶
8.0 Functional-role map. Every emission is exactly one of four types, and the runner enforces the routing (§8.2) as a type system. In plain terms:
- Directional — an arrow: "price is more likely up than down right now." The only role that ever creates a position.
- Gate — a volume knob on the arrows, with no arrow of its own. It answers "how much should I trust my directional signals right now?", never "which way." Example: message rate explodes 10× because two HFTs are machine-gunning quotes at each other — that says nothing about direction, but it says the imbalance and flow signals are currently measuring their skirmish, not real supply and demand, so
gatedrops to 0.25 and every position shrinks to a quarter whichever way it points. Same logic forvpin(flow looks toxic → shrink) andsuppress_qi(that big bid was faked-and-pulled → stop believing bid-side imbalance for 3 s). Mechanically a gate multiplies the output: it can only make you smaller, never flip you. - Conditioner — a knob on the recipe, not the volume. It answers "which of my signals should be doing the talking right now, and how strongly?" Example:
sbarsays this name is pinned at a one-tick spread today → the combiner leans hard on queue imbalance and barely listens to deep-book OFI; the same name floats to a 3-tick spread after crossing a band boundary → the weights shift the other way. A conditioner never says up/down and never says more/less overall — it says reweight. Mechanically it enters only as interaction terms (weight_on_i1 = f(sbar)), never as a standalone bet. Slope belongs here too: a flat ask side doesn't mean "buy" — it means the same buy flow will move price further, i.e. it scales OFI's β. - Diagnostic / infrastructure — parts of the engine that never touch a trade decision. Diagnostics are thermometers: the contemporaneous OFI regression exists so a broken book builder fails loudly (R² collapses) instead of silently poisoning everything; the L2 cancel proxy runs in shadow purely so that a drift away from the L3 version reveals the trade join rotting. Infrastructure is plumbing other pieces stand on:
microas the marking price, the feature stack as a container. When one of these "fires," the correct response is never "trade" — it is "stop and fix," or nothing.
One sentence each: directional says which way · a gate says how much to trust the which-way · a conditioner says which which-way to listen to · diagnostics say whether the machine producing all of the above is even working.
"Horizon" in the table = where the forecast content naturally lives — the Wave-1 IC-by-horizon table (§9.1) is authoritative over this column once measured.
| Role | Emissions (§) | Horizon | Routing |
|---|---|---|---|
| Directional predictors | p_up/i1 (6.5) · i_w (6.6) · i1_ct (6.7) · micro_adj (6.8) · lagged ofi/ofi_int (6.1–6.3) · voi/tfi (6.9–6.10) · xofi (6.4) · prop_drift (6.14) · ebk (7.1) · sweep_* (7.2) · tfi_small/mid/large (7.3) · qr (7.4) · awi (7.5) · fleet_imb (7.6) · frag_imb/hhi_imb (7.7) · cx_imb/casc (7.8) · pull (7.9) · ice_support/ice_exhaust (7.10) · meta/meta_done (7.11) · hwk (7.12) · resil_asym (7.15) |
event-time → seconds for the queue/flow family; minutes only for prop_drift, xofi, meta |
z-scored inputs to the §8.1 combiner |
| Gates | vpin (6.13) · gate (7.14) · suppress_qi (7.9) · label health TR-1..4 |
n/a | multiply / zero exposure — never a direction |
| Conditioners | sbar/p1/eta/EDGE (2.2) · slope/curv/alpha/shape (6.11) · pid_* (7.13, inactive) |
slow | interaction terms only |
| Diagnostics / infrastructure | contemporaneous OFI regressions + impact β (6.1/9.2) · cxprox_imb/spread_hl (6.12, shadow) · feature stack (6.15) · micro as marking (6.8) |
n/a | never routed to the combiner |
Why the typing is enforced in code rather than by convention: route vpin as a direction and you systematically fade toxicity you should be dodging; use slope as an arrow and you are trading a spoofable state; ignore a diagnostic and you trade confidently on a corrupted book.
8.1 Directional stack. All directional emissions are combined per (tick band, session-half): take the zs() forms, fit Ridge or LASSO against forward returns at the target horizon with purged CV (§9.3), refit weekly. Interact with the continuous regime conditioner (sbar or p1, §2.2) rather than branching on the binary class; intermediate names get no stack of their own — they run both regimes' signals with sbar interactions.
8.1b Build order — full tick history available, latency not binding. With the complete history on disk, time-to-verdict is compute, not calendar, and the latency axis is descriptive, not a gate. Ordering is therefore by build effort and statistical honesty only:
- Wave 0 (≈ a day, L2 file only): §2.2 classifier + ofi_l1 + i1. Known-answer diagnostics (R² 0.4–0.7 contemporaneous, x1 > 0) that certify the recorded files before anything is built on them.
- Wave 1 — the big replay: build §4, pass T-6/TR-1..4 across the full history, then compute every accumulator-type signal in one batch pass over the tape — mlofi (both modes), all imbalances, voi, slope/curvature/alpha/shape, awi, fleet, frag, the ebk components, sweeps, bucketed TFI, cx_imb, casc, resil_asym, otr, spread_hl — marginal cost per extra signal in a replay is trivial. Rank everything with the §9 harness per (sbar decile × horizon × session-half). This output is the empirical re-derivation of the §2.6 table on our own data; it replaces the literature priors before any calibrated work is attempted.
- Wave 2 — calibrated singles, ordered by how self-contained they are: microprice (§6.8), QI logistic (§6.5), ebk ridge (§7.1), integrated-OFI PCA (§6.3), QR intensities (§7.4). Each a bounded nightly/weekly job with its own acceptance test.
- Wave 3 — the expensive and the overfit-prone: Hawkes MLE (§7.12), propagator (§6.14), cross-asset LASSO (§6.4), pattern detectors (§7.9–§7.11). Abundant history fixes their sample-size problem, not their researcher-degrees-of-freedom problem: thresholds are fixed on one half of the (universe × period) and validated cold on the other, and the cheapest honesty device is cross-sectional — a real effect must hold with the same sign across sbar deciles and across hundreds of names.
- Latency curve (§9.4): demoted from gate to descriptive statistic — compute it once as the map of where in time the information lives. The horizon menu extends upward accordingly: propagator, lagged cross-asset OFI and metaorder drift at 1–30 min are first-class citizens.
- History caveats: never pool coefficients or evaluation windows across a tick-table change for affected bands — the tick_table_version keying enforces the artefact side; the backtest windows are on you. Check archive depth before pooling across older regime changes (introduction of the closing auction, of volatility halts).
8.1c Regime-informed priority — the venue measures overwhelmingly LARGE (§2.2). Within the waves, the pinned-regime column of §2.6 fixes both the order and the evaluation target:
- Tier 1 — the pinned-regime core. i1 → p_up (§6.5) with its cleaners i1_ct/spoof_tell (§6.7), awi (§7.5), frag_imb (§7.7): the next mid move is the race between two queues, and these de-noise its crudest sufficient statistic. micro_adj (§6.8): with rtick this fat, the sub-tick estimate of where value sits inside the cell is worth up to half a tick — in bps, an enormous number; nowhere does the microprice pay better than on a board like this. ebk (§7.1): the one flow signal at full power here, because the component that leads on pinned books is cancel flow — the race is often decided by withdrawals before any trade prints. qr (§7.4): the explicit race model; fit after i1 works, since its intensity tables double as the fill-probability model.
- Tier 2 — queue-death overlays, meaningful exactly because queues here are long-lived objects worth fighting over: fleet_imb (§7.6), pull (§7.9), ice_support/ice_exhaust (§7.10), cx_imb/casc (§7.8), resil_asym (§7.15). Full history removes their event-scarcity problem — they validate in one replay.
- Tier 3 — minutes-horizon flow. prop_drift (§6.14), tfi + size buckets (§6.10/§7.3), meta (§7.11), xofi (§6.4). On pinned names these do not forecast the next tick — the queue signals own that — they forecast the drift over the next minutes, i.e. which way the sequence of queue races resolves.
- Deprioritized (reserve capacity): integrated OFI beyond its diagnostic role, slope/curvature, and the small-tick aggregation machinery — reserved for floating episodes and whatever population a future tick-table change migrates (§2.2).
Two framing rules. (1) Evaluation target: Tier 1 is judged on event-time next-mid-flip direction (AUC / hitting probability), not clock-time return regressions — the §2.6 modelling corollary is the default on this board, clock-time IC the secondary view. (2) Monetization shape: on 10–200bp ticks the per-event directional edge is usually smaller than the spread, so the natural use is maker-side — quote leaning, join-vs-cancel, avoiding the toxic side of the race — rather than crossing. The Wave-1 AUC/IC-by-sbar-decile table remains the arbiter; this ordering is the prior it replaces.
8.2 Gates and conditioners (multiplicative / interaction-only — never directional).
signal_out = gate (§7.14) * combiner(directional zs features)
vpin (§6.13) : position throttle only — e.g. scale by (1 - clip(zs(vpin)/4, 0, .75))
suppress_qi (§7.9) : zero the i1 / micro_adj contribution of the flagged side while raised
label health (TR-*) : if any of TR-1..TR-4 fail for the session, zero the label-sensitive family (§7)
sbar/p1 (§2.2), slope/shape (§6.11), rtick, EDGE flag : interaction terms only
8.3 Calibration schedule — all artefacts keyed (sym, asof_date, tick_table_version, code_version); the loader hard-fails on staleness > 5 sessions or version mismatch, no silent fallback.
| cadence | artefacts |
|---|---|
| nightly | ADV20 · seasonality bins (§5.3) · tick-regime stats sbar/p1/eta + class (§2.2) · PCA weights (§6.3) · QI logistic (§6.5) · microprice g (§6.8) · clip-size quantiles (§7.2/§7.3) |
| weekly | QR intensity tables (§7.4) · Hawkes MU/A (§7.12) · propagator kernel (§6.14) · cross-asset LASSO (§6.4) · Ridge betas for §7.1/§7.12 · λ selection (§6.6) |
§9. Evaluation harness¶
9.1 Information coefficient. IC(signal, h) = Spearman(signal_t, fwd_log_mid_return_{t,t+h}) per (sym, h, session-half, tick band); block-bootstrap CIs (1,000 draws, fixed seed). A signal is "alive" at h if the CI excludes 0 out-of-sample. Per §8.1c, Tier-1 signals on this venue are headlined by event-time next-flip AUC; clock-time IC is the secondary view.
9.2 R² suite. Contemporaneous 1-min OLS for the OFI family — the sanity anchor: on liquid names R²(ofi_l1) lands roughly 0.4–0.7; far below means transition logic or session hygiene is broken. Out-of-sample by day-forward chaining for everything calibrated.
9.3 Purged cross-validation. Whenever fitting against forward returns: k-fold with training samples whose target window overlaps the validation fold removed (purge), plus an embargo ≥ h after each fold — kills the overlap leak that inflates naive HF backtests.
9.4 Latency curve (compute once for the §7 path). Recompute IC with each signal delayed by δ ∈ {0, 1, 2, 5, 10, 20, 50} ms. With latency not binding on this stack (§8.1b) the curve is descriptive, not a gate: it is the map of where in time each signal's information lives, and the reference point if any signal is ever deployed on a latency-constrained path.
9.5 Slicing — this is where §2.6 gets re-derived on our venue. Every metric is reported per tick band (and per sbar decile), per session half; re-baselined after any tick-table version change. The output of this slicing is the empirical version of the §2.6 playbook table for this venue.
9.6 Label-quality monitoring. Daily: TR-1/TR-2/TR-3 rates, AMBIGUOUS share, mode-B match-window sensitivity (recompute at 50/100/200 ms; results should be stable), and the §6.12-vs-§7.8 cancel shadow divergence. Any drift here quarantines the label-sensitive family before it quarantines the P&L.
§10. Acceptance tests (the build is done when all pass)¶
| id | test | expected |
|---|---|---|
| T-1 | §6.1 fixture 1 | +1500 |
| T-2 | §6.1 fixture 2 | −6000 |
| T-3 | §6.1 fixture 3 (ask sentinel) | −4000 |
| T-4 | §6.2 MLOFI-1 level shift | index mode v[0]=+2000, v[1]=+5000; price mode total +2000 |
| T-5 | §4 BB-2 modify-priority sequence | exact CX/LO emissions and prio_ts behaviour |
| T-6 | §4.5 builder vs our L2 file | exact match at every shared exchSeq (excl. auctions, opening replay burst, first row after a gap); any mismatch is a defect — investigate |
| T-7 | §4.5 TI-1 inference fixture | two MO relabels (3,000 + 1,000), oidB keeps priority, exec_qty = 1,000 |
| T-8 | §4.5 TR-1 | inferred MO qty == automatch trade qty (exact mode A; ≤0.1% mode B) |
| T-9 | §4.5 TR-2 | every inferred MO at prevailing best of its passive side; violations = 0 |
| T-10 | §4.5 TR-3 | AMBIGUOUS < 0.1% of traded qty (A) / < 1% (B) |
| T-11 | §6.8 MP-1/MP-2 | exact antisymmetry; micro ∈ [bid,ask] ≥ 99.9% |
| T-12 | sign audit | one synthetic buy-pressure tape; every directional emission ≥ 0 |
| T-13 | §6.5 QI-1 | x1 > 0; p_up monotone in i1 |
| T-14 | §6.3 CAL-1 | OOS R²(ofi_int) ≥ R²(ofi_l1) on ≥ 70% of liquid-20 |
| T-15 | §7.4 MC vs crude race | mean abs diff ≤ 0.10 |
| T-16 | §7.1 acceptance | R²(ebk) ≥ R²(ofi_l1) on ≥ 70% of liquid-20 |
| T-17 | §2.2 classifier sanity | synthetic pinned tape → LARGE; synthetic diffusive tape → SMALL; a pinned real-name day → LARGE |
| T-18 | reset property | synthetic tape crossing the midday break: no accumulator survives; first post-reset transition emits zero flow |
| T-19 | determinism | two replays of the same three files ⇒ byte-identical outputs |
| T-20 | static checks | no float price comparisons; no wall-clock reads in the streaming path |
| T-21 | throughput | ≥ 200k tape events/s single core in batch replay |
| T-22 | §4.5 TR-4 TradeID continuity | per security, TradeIDs 1..N gapless net of cancels; a gap quarantines the label-sensitive family |
§11. Consolidated parameter defaults¶
bucket_ms = 1000
inference: { mode: SEQ_if_available_else_TIME, seq_claim_window: 20, match_window_ms: 100,
trd_type_book: AUTOMATCH_ONLY, # auction quarantined, off-book -> offbook stream
ambiguous_policy: EXCLUDE_BOTH, max_unreconciled: 0.1% (A) / 1% (B) }
session: { warmup_s: 5 } # suppress flow accumulation during the opening book-replay burst
regime: { p1_large: 0.85, sbar_large: 1.2, sbar_small: 2.5,
hysteresis_sessions: 5, band_edge_pct: 2,
illiquid_adv_floor_pctile: 20, illiquid_quote_floor: 0.95 }
norms: { Q_L1.halflife: 30*MIN, Q_TOP.halflife: 30*MIN, AES.window: 30*MIN }
seasonality: { bin: 5*MIN, lookback: 20 sessions, winsor: ±8 }
mlofi: { mode: INDEX, levels: 10 }
pca: { calib: 10 sessions, min_obs: 500 }
qi: { min_obs_logistic: 2000, tick_bands_bp: (10, 50) }
microprice: { imb_edges: [±0.8, ±0.6, ±0.4, ±0.2, 0], spread_bins: (1,2,3+), K: 6, min_cell_obs: 50 }
ebk: { band_ticks: 2 }
sweeps: { cluster_gap: 1*MS, large_pctl: 95 }
tfi_buckets: { q_small: 0.50, q_large: 0.90, lookback: 5 sessions }
qr: { q_max: 30, refit: weekly }
age: { tau: 30*SEC, levels: 3 }
fleeting: { tau_f: 500*MS, window: 5*MIN } # CONFIRMED cancels only
cascade: { cx_halflife: 10*SEC, mo_lookback: 5*SEC, mo_min: 0.5*AES, band_ticks: 2 }
slope_shape: { min_levels: 5, robust: THEIL_SEN, near_far_split: 5, alpha_clip: [0.2, 3.0] }
pull: { level_share: 0.30, max_dist: 2 ticks, max_lifetime: 2*SEC,
fade_halflife: 5*SEC, suppress: 3*SEC }
reload: { dt: 1*SEC, clip_ratio: 1.5, n_confirm: 3, exhaust_mult: 3 }
metaorder: { window: 20*MIN, n_min: 8, cv_t < 0.5, cv_s < 0.3, done_gap: 3x median }
hawkes: { beta_halflives: (0.1, 1, 10)*SEC, branching_cap: 0.95, refit: weekly }
otr: { window: 10*SEC, burst_z: 3, gate_floor: 0.25, recover: 30*SEC }
resiliency: { phi: 0.5, pre_depth_ew: 10*SEC, window: 30*MIN }
vpin: { V: ADV20/50, sma: 50 bars }
propagator: { grid: 1*SEC, max_lag: 600*SEC, refit: weekly }
eval: { horizons: (1,5,10,30,60,300)*SEC, latency_curve: (0,1,2,5,10,20,50)*MS }