checking my backtests for overfitting

worried my strategies are overfit to historical data. spent today testing for it. been reading NexusFi backtesting threads about this exact problem.

the problem

#

my backtests look great:

• sharpe 2.1
• 70% win rate
• max drawdown 8%

live trading: not so great.

this might be overfitting.

what is overfitting

#

optimizing strategy parameters until backtest looks perfect, but performance doesn’t translate to live trading.

classic example:

# Optimize these parameters
lookback_period = 47  # why 47? because backtest says so
volatility_threshold = 2.37  # why 2.37? because backtest says so
profit_target = 0.523  # suspiciously specific

# These parameters have no logical basis
# They just happened to work in backtest

if parameters are oddly specific (2.37 instead of 2.0 or 2.5), probably overfit.

tests i ran

#

1. out-of-sample testing

train on 2020-2022, test on 2023:

# Training period
train_start = '2020-01-01'
train_end = '2022-12-31'

# Optimize parameters on training data
best_params = optimize_strategy(train_start, train_end)

# Test on completely unseen data
test_start = '2023-01-01'
test_end = '2023-05-15'
test_results = backtest_strategy(test_start, test_end, best_params)

# Compare performance
print(f"Training Sharpe: {train_sharpe:.2f}")
print(f"Test Sharpe: {test_sharpe:.2f}")
print(f"Degradation: {(train_sharpe - test_sharpe) / train_sharpe * 100:.1f}%")

results:

• training sharpe: 2.08
• test sharpe: 1.43
• degradation: 31%

that’s pretty bad. strategy performs way worse on unseen data.

2. walk-forward analysis

test strategy on rolling windows:

# Walk forward with 90-day train, 30-day test
window_size = 90
test_size = 30

results = []
for start_date in date_range('2020-01-01', '2023-05-01', step=30):
    train_end = start_date + timedelta(days=window_size)
    test_end = train_end + timedelta(days=test_size)

    # Train
    params = optimize(start_date, train_end)

    # Test
    perf = backtest(train_end, test_end, params)
    results.append(perf)

# Aggregate all test periods
overall_sharpe = calculate_sharpe(results)

results:

• average test sharpe: 1.38
• consistency: 12 of 15 periods profitable
• worst period: -12% (feb 2023)

walk-forward shows strategy works but not as well as single backtest suggested.

3. parameter sensitivity

test how sensitive results are to parameter changes:

base_sharpe = 2.08  # optimal parameters

# Test small parameter changes
for delta in [-10%, -5%, +5%, +10%]:
    modified_lookback = base_lookback * (1 + delta)
    modified_sharpe = backtest_with_params(modified_lookback, ...)

    degradation = (base_sharpe - modified_sharpe) / base_sharpe
    print(f"Lookback {delta:+.0%}: Sharpe {modified_sharpe:.2f} ({degradation:+.1%})")

results:

• -10% lookback: sharpe 1.92 (-8%)
• -5% lookback: sharpe 1.98 (-5%)
• +5% lookback: sharpe 2.01 (-3%)
• +10% lookback: sharpe 1.89 (-9%)

performance cliff at ±10%. not good.

if tiny parameter changes destroy performance, strategy is fragile and probably overfit.

4. monte carlo simulation

randomly shuffle trade returns to test if performance is luck:

import random

actual_trades = get_backtest_trades()
actual_sharpe = calculate_sharpe(actual_trades)

# Run 1000 simulations with shuffled trades
simulated_sharpes = []
for i in range(1000):
    shuffled = random.sample(actual_trades, len(actual_trades))
    sim_sharpe = calculate_sharpe(shuffled)
    simulated_sharpes.append(sim_sharpe)

# Where does actual performance rank?
percentile = sum(s < actual_sharpe for s in simulated_sharpes) / 1000
print(f"Actual Sharpe: {actual_sharpe:.2f}")
print(f"Percentile: {percentile:.1%}")

results:

• actual sharpe: 2.08
• percentile: 73rd

only better than 73% of random shuffles. not great.

strong strategies should be >95th percentile.

signs of overfitting

#

found in my strategies:

1. ✅ overly specific parameters - 2.37 instead of 2.0
2. ✅ degradation on unseen data - 31% sharpe drop
3. ✅ parameter sensitivity - 8% drop with small changes
4. ✅ mediocre monte carlo - only 73rd percentile
5. ✅ too many rules - 12 entry conditions (probably 3-4 would work)

yeah, i’m overfit.

how to avoid overfitting

#

use simple parameters:

# Bad (overfit)
lookback = 47
threshold = 2.37
profit_target = 0.523

# Good (sound)
lookback = 50  # round numbers
threshold = 2.0  # logical values
profit_target = 0.50  # clean percentage

fewer rules:

before:

• IV rank > 45
• IV percentile > 70
• delta < 0.20
• delta > 0.12
• volume > 500
• bid-ask spread < 0.05
• dte between 30-45
• dte not 35-40 (weird exclusion)
• underlying price > $50
• earnings > 7 days away
• vix < 30
• no positions in same sector

12 rules. probably 6 are noise.

after:

• IV rank > 45
• delta 0.15-0.25
• dte 30-45
• earnings > 7 days away

4 rules. cleaner, more reliable.

regularization in optimization:

penalize complexity:

def fitness_function(params, trades):
    sharpe = calculate_sharpe(trades)
    complexity = count_rules(params)

    # Penalize complex strategies
    penalty = complexity * 0.1

    return sharpe - penalty

simpler strategies get higher fitness even if sharpe is slightly lower.

walk-forward not single backtest:

don’t optimize on all historical data. use rolling windows.

out-of-sample validation:

always hold back 20-30% of data for final validation.

never optimize on validation set.

fixing my strategies

#

going through each strategy:

premium selling:

• reduced from 12 rules to 5
• simplified parameters (round numbers)
• retested walk-forward

new results:

• training sharpe: 1.82 (lower)
• test sharpe: 1.65 (9% degradation, better)
• monte carlo: 89th percentile (much better)

less impressive backtest but more strong.

mean reversion:

• reduced from 8 rules to 4
• removed oddly specific parameters
• retested

new results:

• training sharpe: 1.65
• test sharpe: 1.52 (8% degradation)
• monte carlo: 91st percentile

better.

live vs backtest comparison

#

before (overfit strategies):

• backtest sharpe: 2.08
• live sharpe (4 months): 0.43
• degradation: 79%

brutal.

after (simpler strategies):

• backtest sharpe: 1.71
• expected live sharpe: ~1.45
• expected degradation: 15%

more realistic expectations.

lessons

#

1. impressive backtests are suspicious - if it looks too good, probably overfit
2. simplicity is robustness - fewer rules = more likely to work live
3. test multiple ways - out-of-sample, walk-forward, monte carlo
4. parameter sensitivity matters - strategy should work with approximate parameters
5. accept lower backtest performance - if it translates to live, worth it

my new testing protocol

#

before going live with any strategy:

1. out-of-sample test (30% degradation max)
2. walk-forward analysis (consistency check)
3. parameter sensitivity (10% parameter change = <10% performance change)
4. monte carlo simulation (>90th percentile)
5. simplicity check (≤5 rules)

if fails any test, back to drawing board.

current strategies status

#

premium selling v2:

• passes all tests
• going live next week with minimum size

mean reversion v2:

• passes 4 of 5 tests
• needs more work

momentum:

• fails everything
• scrapped entirely

reality check

#

my strategies weren’t as good as backtests suggested. they were overfit.

fixing them means accepting lower expected returns but higher probability they’ll actually work.

better to make 15% annually consistently than backtest 50% and make 0% live.

3:22am. spent all day on this. strategies are simpler now. hopefully actually work in live trading.

-AK