backtesting is where most algo traders hurt themselves.
they optimize parameters until strategy looks perfect on historical data.
then go live and it fails immediately.
classic overfitting.
learned this the hard way. saw countless traders on NexusFi backtesting discussions make same mistake when i joined in 2023.
what is overfitting #
you tune parameters so precisely to past data that strategy captures noise instead of signal.
example:
you backtest mean reversion with lookback period from 5 to 50 days.
find that 23 days gives perfect sharpe ratio on 2023 data.
go live with 23 days.
january 2024 hits and strategy fails completely.
why?
23 days wasn’t edge.
it was coincidence.
you fit noise.
how i got burned (march 2023) #
lost $40k in one month on “perfect” backtest strategy.
what i did wrong:
tested momentum strategy with 50+ parameter combinations.
picked best sharpe ratio (2.8 on 2022 data).
parameters:
- lookback: 17 days
- entry threshold: 1.8 std dev
- exit threshold: 0.6 std dev
- position size: 2.1% risk
looked amazing.
went live march 2023.
completely failed.
lost $40k in 3 weeks.
lesson learned the expensive way.
my overfitting prevention system #
built after march disaster.
5 rules i follow religiously:
1. walk-forward analysis #
never optimize on full dataset.
split data into chunks:
- train: optimize parameters
- test: verify performance
- walk forward: repeat process
def walk_forward_analysis(data, train_period, test_period):
"""
Walk-forward optimization to prevent overfitting
"""
results = []
for i in range(0, len(data) - train_period - test_period, test_period):
# Split data
train_data = data[i:i+train_period]
test_data = data[i+train_period:i+train_period+test_period]
# Optimize on training data
best_params = optimize_strategy(train_data)
# Test on unseen data
test_results = backtest_strategy(test_data, best_params)
results.append({
'train_sharpe': best_params['sharpe'],
'test_sharpe': test_results['sharpe'],
'degradation': best_params['sharpe'] - test_results['sharpe']
})
return results
# Real usage
results = walk_forward_analysis(
data=historical_data,
train_period=252, # 1 year training
test_period=63 # 3 months testing
)
# Check for overfitting
avg_degradation = np.mean([r['degradation'] for r in results])
if avg_degradation > 0.5:
print("WARNING: Likely overfitting")
if strategy crushes training but fails testing = overfitting.
2. parameter stability check #
good parameters shouldn’t be hyper-sensitive.
test nearby values.
if sharpe drops 50% when changing lookback from 23 to 24 days = overfitting.
if sharpe stays similar from 20-30 days = sturdy edge.
def parameter_stability_test(data, base_params, param_to_test, test_range):
"""
Test parameter stability - solid edge vs curve fit
"""
results = []
base_sharpe = backtest_strategy(data, base_params)['sharpe']
for value in test_range:
test_params = base_params.copy()
test_params[param_to_test] = value
test_sharpe = backtest_strategy(data, test_params)['sharpe']
degradation = (base_sharpe - test_sharpe) / base_sharpe * 100
results.append({
'param_value': value,
'sharpe': test_sharpe,
'degradation_pct': degradation
})
# Good strategy: degradation stays under 20% for nearby values
nearby_results = [r for r in results if abs(r['param_value'] - base_params[param_to_test]) <= 5]
avg_nearby_degradation = np.mean([r['degradation_pct'] for r in nearby_results])
if avg_nearby_degradation > 20:
print(f"WARNING: Parameter {param_to_test} is unstable")
print(f"Nearby degradation: {avg_nearby_degradation:.1f}%")
return False
return True
# Test lookback period stability
stable = parameter_stability_test(
data=historical_data,
base_params={'lookback': 20, 'threshold': 1.5},
param_to_test='lookback',
test_range=range(15, 31) # Test 15-30 days
)
i want strategies that work across parameter ranges.
not pinpoint precision.
3. limit parameter count #
more parameters = more overfitting risk.
my rule: max 4 tunable parameters per strategy.
mean reversion (3 parameters):
- lookback period
- entry threshold
- exit threshold
momentum (4 parameters):
- lookback period
- entry threshold
- exit threshold
- position sizing multiplier
keep it simple.
4. out-of-sample validation #
always hold back 20% data that optimizer never sees.
final check before going live.
def final_validation(full_data, optimal_params):
"""
Final out-of-sample test on completely unseen data
"""
# Split: 80% train/optimize, 20% final validation
split_idx = int(len(full_data) * 0.8)
optimization_data = full_data[:split_idx]
validation_data = full_data[split_idx:]
# Backtest on validation data (never used for optimization)
validation_results = backtest_strategy(validation_data, optimal_params)
print(f"Optimization Sharpe: {optimal_params['sharpe']:.2f}")
print(f"Validation Sharpe: {validation_results['sharpe']:.2f}")
print(f"Degradation: {(optimal_params['sharpe'] - validation_results['sharpe']):.2f}")
# Decision criteria
if validation_results['sharpe'] < 1.0:
print("REJECT: Validation Sharpe too low")
return False
degradation_pct = (optimal_params['sharpe'] - validation_results['sharpe']) / optimal_params['sharpe'] * 100
if degradation_pct > 30:
print(f"REJECT: Degradation {degradation_pct:.1f}% too high")
return False
print("PASS: Strategy validated for live trading")
return True
if validation fails = back to drawing board.
no exceptions.
5. monte carlo simulation #
test strategy across thousands of random scenarios.
resamples historical data with replacement.
shows worst-case and best-case outcomes.
def monte_carlo_validation(data, params, n_simulations=1000):
"""
Monte Carlo simulation - test across randomized scenarios
"""
results = []
for i in range(n_simulations):
# Resample data with replacement (bootstrap)
bootstrap_data = data.sample(n=len(data), replace=True).sort_index()
# Run backtest on resampled data
sim_results = backtest_strategy(bootstrap_data, params)
results.append({
'sharpe': sim_results['sharpe'],
'max_dd': sim_results['max_drawdown'],
'win_rate': sim_results['win_rate']
})
# Analyze distribution
sharpe_dist = [r['sharpe'] for r in results]
print(f"Sharpe Distribution:")
print(f" Mean: {np.mean(sharpe_dist):.2f}")
print(f" Median: {np.median(sharpe_dist):.2f}")
print(f" 5th percentile: {np.percentile(sharpe_dist, 5):.2f}")
print(f" 95th percentile: {np.percentile(sharpe_dist, 95):.2f}")
# Decision: 5th percentile must be profitable
if np.percentile(sharpe_dist, 5) < 0.5:
print("REJECT: 5th percentile Sharpe too low")
return False
return True
# Validate with Monte Carlo
passed = monte_carlo_validation(
data=historical_data,
params=optimal_params,
n_simulations=1000
)
if 5th percentile is garbage = strategy too fragile.
red flags i watch for #
1. too-perfect backtest
sharpe > 3.0 = suspicious.
win rate > 85% = suspicious.
max drawdown < 5% = suspicious.
real trading has losses.
if backtest doesn’t show them = overfitting.
2. parameter precision
optimal value: 23.47 days lookback.
no way.
round to whole numbers.
if strategy needs decimals = overfitting.
3. degradation on new data
backtest 2023: sharpe 2.5
live trading jan 2024: sharpe 0.8
strategy failed.
overfitting confirmed.
my current mean reversion strategy #
parameters (survived all tests):
lookback: 20 days (stable from 15-25)
entry threshold: 2.0 std dev (stable from 1.8-2.2)
exit threshold: 0.5 std dev (stable from 0.4-0.7)
walk-forward results:
avg training sharpe: 1.6
avg testing sharpe: 1.4
degradation: 0.2 (acceptable)
out-of-sample validation:
optimization sharpe: 1.5
validation sharpe: 1.3
degradation: 13% (good)
monte carlo (1000 sims):
mean sharpe: 1.4
5th percentile sharpe: 0.9
passed all tests.
live performance (nov 2023 - jan 2024):
sharpe: 1.2
within expected range.
how i backtest now #
class RobustBacktester:
"""
Backtesting system with overfitting prevention built-in
"""
def __init__(self, data, train_pct=0.6, validation_pct=0.2):
self.full_data = data
# Split data
train_end = int(len(data) * train_pct)
val_end = train_end + int(len(data) * validation_pct)
self.train_data = data[:train_end] # 60% train
self.test_data = data[train_end:val_end] # 20% test
self.validation_data = data[val_end:] # 20% validation (unseen)
def optimize_parameters(self, param_grid):
"""Step 1: Optimize on training data only"""
best_sharpe = -999
best_params = None
for params in param_grid:
results = backtest_strategy(self.train_data, params)
if results['sharpe'] > best_sharpe:
best_sharpe = results['sharpe']
best_params = params
return best_params, best_sharpe
def test_stability(self, params):
"""Step 2: Test parameter stability"""
# Test lookback stability
lookback_stable = parameter_stability_test(
self.train_data, params, 'lookback',
range(params['lookback']-5, params['lookback']+6)
)
# Test threshold stability
threshold_stable = parameter_stability_test(
self.train_data, params, 'threshold',
np.arange(params['threshold']-0.3, params['threshold']+0.4, 0.1)
)
return lookback_stable and threshold_stable
def walk_forward_test(self, params):
"""Step 3: Walk-forward analysis"""
results = walk_forward_analysis(
data=self.test_data,
train_period=126, # 6 months
test_period=63 # 3 months
)
avg_degradation = np.mean([r['degradation'] for r in results])
return avg_degradation < 0.5 # Max 0.5 Sharpe degradation
def monte_carlo_test(self, params):
"""Step 4: Monte Carlo validation"""
return monte_carlo_validation(
self.test_data, params, n_simulations=1000
)
def final_validation(self, params):
"""Step 5: Final test on completely unseen data"""
results = backtest_strategy(self.validation_data, params)
# Require validation Sharpe > 1.0 and degradation < 30%
train_sharpe = backtest_strategy(self.train_data, params)['sharpe']
degradation_pct = (train_sharpe - results['sharpe']) / train_sharpe * 100
return results['sharpe'] > 1.0 and degradation_pct < 30
def run_full_validation(self):
"""Run complete validation pipeline"""
print("Step 1: Optimizing parameters...")
best_params, train_sharpe = self.optimize_parameters(param_grid)
print(f" Best training Sharpe: {train_sharpe:.2f}")
print("\nStep 2: Testing parameter stability...")
if not self.test_stability(best_params):
print(" FAILED: Parameters unstable")
return None
print(" PASSED")
print("\nStep 3: Walk-forward analysis...")
if not self.walk_forward_test(best_params):
print(" FAILED: High degradation in walk-forward")
return None
print(" PASSED")
print("\nStep 4: Monte Carlo simulation...")
if not self.monte_carlo_test(best_params):
print(" FAILED: Poor performance in Monte Carlo")
return None
print(" PASSED")
print("\nStep 5: Final validation on unseen data...")
if not self.final_validation(best_params):
print(" FAILED: Validation test failed")
return None
print(" PASSED")
print("\n✓ Strategy validated - ready for live trading")
return best_params
# Usage
backtester = RobustBacktester(data=historical_data)
validated_params = backtester.run_full_validation()
if validated_params:
print(f"\nGoing live with: {validated_params}")
else:
print("\nStrategy rejected - back to drawing board")
tonight #
backtesting saved me.
march 2023: didn’t validate properly, lost $40k.
now: 5-step validation process, strategies survive live trading.
overfitting kills accounts.
sound validation saves them.
2:35am monday. backtesting overfitting prevention. lost $40k march 2023 on overfit strategy. built 5-step validation: walk-forward, stability, parameter limits, out-of-sample, monte carlo. current mean reversion passed all tests, performing live within expected range.
-AK