Optimization
walk-forward validation: stopped fooling myself with in-sample results
·2004 words
2:15 AM monday.
A. called it around 11:30. she reads for like 20 minutes and then just drops — book still open on the nightstand, her laptop sitting open on the coffee table. I turned the screen off around midnight, refilled my coffee, sat back down.
backtesting framework - vectorbt for fast parameter testing at scale
·1126 words
vectorbt = game changer for parameter testing.
10x faster than backtrader.
vectorized operations instead of event-driven.
the speed problem # traditional backtesting:
parameter tuning - avoiding overfitting with walk-forward validation
parameter optimization = dangerous.
easy to overfit historical data.
walk-forward validation = solution.
the overfitting problem # traditional optimization:
regime detection - walk-forward validation improving accuracy
·806 words
regime detection upgraded.
walk-forward validation running.
accuracy improving.
the problem # static regime parameters:
optimized on historical data.
walk-forward optimization - how i avoid overfitting my strategies
overfitting = #1 way algos fail in production.
backtest looks amazing.
live trading implodes.
walk-forward optimization prevents this.
been discussing validation techniques on NexusFi algo trading threads and walk-forward is the gold standard.
regime detection improvements - faster market adaptation working
·1101 words
week 3 april going strong.
adaptive strategy crushing it.
been refining regime detection logic.
current performance (apr 1-17) # trades: 29
migrating market data to timescaledb - 10x query speedup
·598 words
been storing market data in regular postgres.
works but slow for time-series queries.
migrated to timescaledb this week.
10x speedup on historical queries.
got the idea from NexusFi algo infrastructure discussions about optimizing market data storage. someone mentioned timescaledb and i researched it.
refactored data pipeline to async - 3x faster market data processing
·907 words
been running synchronous data fetching since january.
works but slow during market open.
refactored to async this week.
3x speed improvement.
the problem with sync code # # Old synchronous approach def fetch_market_data(symbols): results = [] for symbol in symbols: data = fetch_from_api(symbol) # Blocks here results.append(data) return results # With 10 symbols, takes 10 * 180ms = 1,800ms total each API call blocks until complete.
added redis caching - cut market data latency by 60%
·910 words
been noticing market data latency creeping up.
average fetch time: 180ms from polygon API.
slowing down entry execution.
the problem # every time algo needs current price: