2:15 AM monday. system’s been clean since the websocket IV fix went live friday. heartbeat healthy, colo latency normal, no stale data flags. spent most of sunday going deep on something i’ve been meaning to do since the tariff postmortem.
the april chaos gave us a live test of the new architecture. SQS held, IV rank staleness got fixed, drawdown was contained. but that’s one data point. one event in favorable conditions where we had just enough time to build the right tools before the market decided to go sideways. that’s not validation, that’s luck with decent timing.
i want to know what happens when we run the improved system through a real vol event we didn’t know was coming. august 2024. yen carry unwind. VIX hit 65 intraday on august 5th — biggest single-day spike since 2018. that week, the old system took -4.2%.
so i built a replay engine over the weekend and ran it.
why replay testing is different from backtesting #
standard backtesting: run your entry/exit signals against historical price data, see if the strategy made money. useful but limited. it tells you whether the trade logic is sound. it doesn’t tell you whether the infrastructure decisions would have worked in real-time.
replay testing: reconstruct the live system’s real-time state at a historical timestamp and run every component — signal computation, SQS scoring, Redis reads, freshness checks — exactly as they would have behaved if they’d existed then. the question isn’t “would this trade have been profitable?” it’s “would this system have behaved differently, and by how much?”
for validating the SQS and the websocket IV rank improvement, that’s the only test that matters. i need to know if the gate logic holds in conditions we’ve already lived through.
the replay architecture #
the core problem: every system component that reads “current” market state (IV rank, VIX level, sub-signal freshness scores) is pulling from Redis keys populated by live market feeds. for replay, you need a shadow Redis instance that mirrors what live Redis would have held at the simulated timestamp.
import asyncio
from datetime import datetime, timedelta
from dataclasses import dataclass
from redis.asyncio import Redis
from typing import Optional
@dataclass
class MarketSnapshot:
sim_time: datetime
spx_iv_rank: float
vix_level: float
spx_spread_pct: float
data_lag_minutes: float
sub_signal_agreement: float
btc_momentum_z: float
class ReplayEngine:
def __init__(
self,
event_start: datetime,
event_end: datetime,
replay_redis: Redis,
ts_conn,
sqs_threshold: int = 55
):
self.event_start = event_start
self.event_end = event_end
self.replay_redis = replay_redis # shadow instance, live Redis untouched
self.ts = ts_conn
self.sqs_threshold = sqs_threshold
async def seed_snapshot(self, sim_time: datetime) -> MarketSnapshot:
"""Pull 5-min historical snapshot from TimescaleDB. Populate replay Redis."""
row = await self.ts.fetchrow("""
SELECT
spx_iv_rank,
vix_level,
spx_spread_pct,
data_lag_minutes,
sub_signal_agreement,
btc_momentum_z
FROM market_snapshots_5m
WHERE bucket = (
SELECT MAX(bucket)
FROM market_snapshots_5m
WHERE bucket <= $1
)
""", sim_time)
snap = MarketSnapshot(
sim_time=sim_time,
spx_iv_rank=float(row["spx_iv_rank"]),
vix_level=float(row["vix_level"]),
spx_spread_pct=float(row["spx_spread_pct"]),
data_lag_minutes=float(row["data_lag_minutes"]),
sub_signal_agreement=float(row["sub_signal_agreement"]),
btc_momentum_z=float(row["btc_momentum_z"])
)
# Seed replay Redis to match what live Redis would have held at sim_time
await self.replay_redis.hset("iv_rank:SPX", mapping={
"value": snap.spx_iv_rank,
"timestamp": sim_time.timestamp(),
"source": "replay"
})
await self.replay_redis.hset("vix:current", mapping={
"value": snap.vix_level,
"timestamp": sim_time.timestamp()
})
# Simulate IV heartbeat at sim_time (no staleness by default)
await self.replay_redis.set("iv_heartbeat", sim_time.timestamp())
return snap
async def compute_sqs(self, snap: MarketSnapshot) -> dict:
"""
Signal Quality Score: composite 0-100.
Same four components as live system.
"""
# 1. Data freshness (30% weight)
# For old system simulation: use actual data_lag_minutes from historical records
# For new system simulation: assume websocket reduced lag to <1 min
lag = snap.data_lag_minutes
freshness = max(0.0, 100.0 - (lag / 5.0) * 100.0) # 0 at 5+ min lag
# 2. Market spread quality (25% weight)
# Normal SPX spread ~0.04%. >0.12% = degraded conditions
normal_spread = 0.04
spread_score = max(0.0, 100.0 - ((snap.spx_spread_pct - normal_spread) / 0.12) * 100.0)
# 3. Sub-signal agreement (25% weight)
# Smoothed EWA (30-min window in live system)
agreement_score = snap.sub_signal_agreement * 100.0
# 4. VIX regime (20% weight)
# VIX < 20: 100. VIX > 40: 0. Linear in between.
vix_score = max(0.0, min(100.0, (40.0 - snap.vix_level) / 20.0 * 100.0))
composite = (
freshness * 0.30 +
spread_score * 0.25 +
agreement_score * 0.25 +
vix_score * 0.20
)
return {
"composite": round(composite, 1),
"gate_open": composite >= self.sqs_threshold,
"components": {
"freshness": round(freshness, 1),
"spread": round(spread_score, 1),
"agreement": round(agreement_score, 1),
"vix_regime": round(vix_score, 1)
}
}
async def run_event(
self,
step_minutes: int = 15,
simulate_old_system: bool = False
) -> list[dict]:
"""
Step through event window. At each step: seed Redis, compute SQS, record.
simulate_old_system=True: use actual historical data_lag_minutes (reflects
the old 15-min polling lag). simulate_old_system=False: zero out lag
(reflects websocket streaming).
"""
results = []
current = self.event_start
while current <= self.event_end:
snap = await self.seed_snapshot(current)
if simulate_old_system:
# Old system had no SQS and stale IV rank - reconstruct behavior
old_sqs = await self.compute_sqs(snap)
results.append({
"sim_time": current,
"sqs": old_sqs["composite"],
"gate_open": snap.vix_level < 25, # only hard VIX gate
"snap": snap,
"system": "old"
})
else:
# New system: zero lag (websocket), SQS composite gate
snap.data_lag_minutes = 0.1 # ~6 second average lag for websocket
new_sqs = await self.compute_sqs(snap)
results.append({
"sim_time": current,
"sqs": new_sqs["composite"],
"gate_open": new_sqs["gate_open"],
"snap": snap,
"system": "new"
})
current += timedelta(minutes=step_minutes)
return results
the key design decision: simulate_old_system=True uses the actual data_lag_minutes from the historical records (reflecting the 15-minute polling cycle’s real-world lag) and only applies the hard VIX > 25 binary gate. simulate_old_system=False zeros out the lag (websocket parity) and runs the full SQS composite.
historical data in TimescaleDB goes back to november 2023 for 5-minute snapshots. august 2024 is fully reconstructed from live records — i had continuous logging running by then. the yen carry unwind window is clean data.
what august 5th looked like through both systems #
the yen carry unwind peaked on august 5th, 2024. VIX opened at ~28, hit 65 intraday, closed around 38. SPX dropped about 3% on the open and kept going. SPX options spreads went from a normal 0.04% to 0.18–0.22% bid-ask by mid-morning.
running the replay through both systems for the full week (aug 1–9):
the gap on august 5th is the whole story. old system (reconstructed): SQS around 22-31, which is below threshold — but the old system didn’t have a composite gate. it only had the binary VIX > 25 circuit breaker. VIX opened at 28 on aug 5th, which was above 25, so the old system was already throttled. but “throttled” meant max position size reduced to 40% and new entries suspended — it didn’t mean zero signal confidence assessment.
new system: SQS of 4-8 during the aug 5th open. gate hard-closed. three entry signals that the strategy logic flagged as valid got zero-ed out by the quality layer. those three entries were the ones that caused most of the -4.2% loss in the old system that week.
the drawdown picture #
old system crossed the -3% circuit breaker on august 5th. that’s when manual intervention kicked in — i was watching the account that day and pulled the plug on further activity. new system (replay) would’ve peaked at -0.8% by aug 6th as existing positions worked through, then recovered by end of week.
same market. same entry signals. completely different outcome from the gate layer.
what the replay surfaced #
a few things came up that weren’t obvious from the live system design:
sub-signal agreement needs smoothing during fast-moving sessions. the current SQS component computes raw agreement percentage across sub-signals. during the yen carry unwind, that score was cycling between 0% and 100% every 15 minutes as the market whipsawed. a smoothed EWA (exponentially weighted average over 30 minutes) stabilizes this — you want a read on the trend of agreement, not the instantaneous value. built it, going into production this week.
the freshness component is the most leveraged piece. websocket IV rank had the biggest impact on SQS during the vol event window. during the aug 5th morning session, the old 15-min polling cycle was showing data_lag_minutes of 14-16 before each refresh. that alone dropped the freshness component to near-zero, which is exactly what we fixed with the websocket streaming. the replay confirms that fix is the highest-leverage change in the entire stack.
the VIX regime component needs dynamic thresholds. fixed 20/40 band was calibrated when VIX was averaging around 16-18. VIX has averaged higher in 2025-2026. the 40-point upper bound means anything above VIX 40 scores zero, but the practical difference between VIX 65 and VIX 90 is lost. going to parameterize against trailing 90-day VIX distribution instead of fixed bounds.
none of these are blocking — the core system works and the drawdown improvement is real. these are calibration improvements.
where things stand #
replay confirmed what the april tariff week suggested: the infrastructure changes are the right ones. SQS composite gate + websocket IV rank would have cut the august 2024 drawdown from -4.2% to approximately -0.8%. that’s not marginal — that’s the difference between a bad week and a stop-loss event.
been logging a lot of this work in the algo development journal thread on NexusFi — good thread if you’re building actual live systems, lots of people going through the same signal reliability questions. the backtesting methodology article in the academy is also solid on the theory side, though replay testing is a different beast than traditional strategy backtesting.
sub-signal smoothing ships tuesday. VIX regime recalibration following that. then i want to run the replay against the march 2023 SVB banking crisis — messier data since i only have partial records from back then, but it’s the one event i was live for and still have notes on what actually happened.
dinner last night A. made this pasta thing she’s been trying to nail for a month. finally got it right. spent ten minutes telling me what she changed from the last three attempts. i mostly just ate it and said it was good. she said “you’re not even listening” and she was right and also it was really good pasta.
algos running clean. colo heartbeat steady. nothing needs me until morning.
somewhere around 1 AM i was staring at the drawdown chart and got that random thing where an image just shows up — dad sitting at the kitchen table at midnight with engineering schematics and a cup of coffee he’d forgotten about. no particular reason. just happens sometimes. this line of work runs late and so did he.
back to the code.
-AK