finally got around to documenting my monitoring setup.
been running this stack for almost 2 years now. saved my ass multiple times.
why monitoring matters #
had an algo go sideways in march 2024.
what happened:
- strategy kept opening positions
- API rate limit hit
- orders queued but never confirmed
- woke up to $8k in unintended exposure
if i had proper alerting i would’ve caught it in 5 minutes instead of 5 hours.
lesson learned. built this stack.
the architecture #
[Algo Servers] → [Prometheus] → [Grafana] → [AlertManager]
↓ ↓ ↓ ↓
metrics time-series dashboards slack/email
exporter database visualization alerts
components:
- prometheus: metrics collection and storage
- grafana: visualization dashboards
- alertmanager: notification routing
- node_exporter: server metrics (cpu, memory, disk)
- custom python exporter: trading-specific metrics
prometheus config #
running on my chicago colo box. same server as execution.
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
rule_files:
- 'alerts/*.yml'
scrape_configs:
- job_name: 'algo_metrics'
static_configs:
- targets: ['localhost:8000']
metrics_path: '/metrics'
- job_name: 'node'
static_configs:
- targets: ['localhost:9100']
- job_name: 'ib_gateway'
static_configs:
- targets: ['localhost:8001']
15 second scrape interval. fast enough for trading, not so fast it hammers the server.
custom python exporter #
this is where it gets good.
from prometheus_client import start_http_server, Gauge, Counter, Histogram
import time
from typing import Dict
from dataclasses import dataclass
# Core trading metrics
ORDERS_TOTAL = Counter(
'algo_orders_total',
'Total orders placed',
['strategy', 'side', 'symbol']
)
POSITION_VALUE = Gauge(
'algo_position_value_usd',
'Current position value in USD',
['strategy', 'symbol']
)
PNL_REALIZED = Gauge(
'algo_pnl_realized_usd',
'Realized P&L in USD',
['strategy']
)
API_LATENCY = Histogram(
'algo_api_latency_seconds',
'API call latency',
['broker', 'endpoint'],
buckets=[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]
)
FILL_RATE = Gauge(
'algo_fill_rate_percent',
'Order fill rate percentage',
['strategy']
)
@dataclass
class TradingMetrics:
"""Container for trading metrics"""
orders_placed: int = 0
orders_filled: int = 0
total_pnl: float = 0.0
open_positions: Dict[str, float] = None
def __post_init__(self):
if self.open_positions is None:
self.open_positions = {}
class MetricsExporter:
def __init__(self, port: int = 8000):
self.port = port
self.metrics = TradingMetrics()
def start(self):
start_http_server(self.port)
print(f"Metrics server running on port {self.port}")
def record_order(self, strategy: str, side: str, symbol: str):
ORDERS_TOTAL.labels(
strategy=strategy,
side=side,
symbol=symbol
).inc()
self.metrics.orders_placed += 1
def update_position(self, strategy: str, symbol: str, value: float):
POSITION_VALUE.labels(
strategy=strategy,
symbol=symbol
).set(value)
self.metrics.open_positions[f"{strategy}_{symbol}"] = value
def record_pnl(self, strategy: str, pnl: float):
PNL_REALIZED.labels(strategy=strategy).set(pnl)
self.metrics.total_pnl = pnl
def record_latency(self, broker: str, endpoint: str, latency: float):
API_LATENCY.labels(
broker=broker,
endpoint=endpoint
).observe(latency)
def update_fill_rate(self, strategy: str, rate: float):
FILL_RATE.labels(strategy=strategy).set(rate)
every algo i run imports this and calls the methods. prometheus scrapes every 15 seconds.
latency monitoring #
this is the money chart.
api latency by hour and day. market open (6:30am PST) always spikes. normal.
what i’m watching:
- baseline should be 3-4ms from chicago colo to IB
- anything over 10ms gets flagged
- sustained 20ms+ triggers alert
the heatmap makes patterns obvious. market open = spike. lunch = calm. power hour = busy.
system health dashboard #
cpu, memory, and order rate over first two weeks of january. spikes during market hours. flat overnight.
alert thresholds:
- CPU > 80% sustained 5min → warning
- CPU > 95% → critical
- Memory > 85% → warning
- Order rate = 0 during market hours → critical (algo might be dead)
grafana alerts #
the real value is alerting.
# alerts/trading_alerts.yml
groups:
- name: trading
rules:
- alert: HighLatency
expr: histogram_quantile(0.95, algo_api_latency_seconds_bucket) > 0.1
for: 2m
labels:
severity: warning
annotations:
summary: "High API latency detected"
- alert: NoOrders
expr: increase(algo_orders_total[15m]) == 0
for: 30m
labels:
severity: critical
annotations:
summary: "No orders in 30 minutes during market hours"
- alert: HighDrawdown
expr: algo_pnl_realized_usd < -5000
labels:
severity: critical
annotations:
summary: "Daily P&L below -$5000"
slack notifications for warnings. actual phone call for critical.
what this catches #
since implementing (march 2024):
- 3 API rate limit issues → caught in <5min
- 2 memory leaks → caught before crash
- 1 broker gateway disconnect → alerted immediately
- 4 strategy drift events → saw it in metrics before P&L
total incidents avoided: probably $30k+ in potential losses.
$200/month for the colo server. worth every penny.
the stack in practice #
wake up. check grafana on phone. green = good. red = problem.
takes 30 seconds.
if something’s off, i can drill into specific metrics, see when it started, correlate with market events.
been discussing monitoring setups with other algo traders on NexusFi - most people underestimate how important observability is until they get burned.
next upgrades #
planning to add:
- position tracking dashboard: real-time exposure by strategy
- correlation monitoring: detect when strategies overlap too much
- market regime detection: alert when conditions change
infrastructure is boring until it saves you.
2:34am wednesday. finally documented my monitoring stack. prometheus + grafana + custom python exporter. catches problems before they cost money. $200/month for peace of mind. been running since march 2024 after the $8k incident. hasn’t let me down since.
-AK