monitoring and alerting for algo infrastructure - grafana + prometheus setup · Pro Algo Trading

realized last month that i had a massive blind spot in my infrastructure: monitoring. my algos were running 24/7 but i had no idea if they were running WELL until i checked PnL at end of day. that’s like driving with your eyes closed and checking if you crashed when you stop.

built out a proper monitoring stack with prometheus for metrics collection and grafana for dashboards. now i get alerts on my phone when anything looks sketchy.

what i’m monitoring
#

the obvious stuff first:

strategy PnL (real-time, per-strategy)
order fill rates (how many orders actually execute vs fail)
data feed latency (am i getting stale data?)
signal generation latency (are my algos keeping up?)
system resources (CPU, RAM, disk I/O on the dell rack)

the less obvious but equally important stuff:

signal-to-fill latency (time between signal generation and order fill)
slippage per trade (expected vs actual fill price)
position drift (actual positions vs target positions)
correlation breakdown (when historically correlated strategies diverge)

prometheus metrics in python
#

prometheus has a python client that makes it dead simple to expose metrics from your trading code. here’s how i instrumented my pipeline:

import time
import logging
from prometheus_client import (
    Counter,
    Gauge,
    Histogram,
    Summary,
    start_http_server,
    CollectorRegistry,
)
from dataclasses import dataclass

logger = logging.getLogger(__name__)

# custom registry to avoid polluting default
REGISTRY = CollectorRegistry()


# ---- metric definitions ----

# order metrics
ORDERS_TOTAL = Counter(
    "algo_orders_total",
    "total orders placed",
    ["strategy", "exchange", "side", "order_type", "status"],
    registry=REGISTRY,
)

FILL_LATENCY = Histogram(
    "algo_fill_latency_seconds",
    "time from signal to fill",
    ["strategy", "exchange"],
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0],
    registry=REGISTRY,
)

SLIPPAGE_BPS = Histogram(
    "algo_slippage_bps",
    "slippage in basis points",
    ["strategy", "exchange", "side"],
    buckets=[0, 0.5, 1, 2, 3, 5, 10, 20, 50],
    registry=REGISTRY,
)

# PnL metrics
PNL_TOTAL = Gauge(
    "algo_pnl_total_usd",
    "total PnL in USD",
    ["strategy"],
    registry=REGISTRY,
)

PNL_DAILY = Gauge(
    "algo_pnl_daily_usd",
    "daily PnL in USD",
    ["strategy"],
    registry=REGISTRY,
)

POSITION_SIZE = Gauge(
    "algo_position_size",
    "current position size",
    ["strategy", "symbol"],
    registry=REGISTRY,
)

# signal metrics
SIGNAL_LATENCY = Histogram(
    "algo_signal_latency_ms",
    "signal generation time in ms",
    ["strategy"],
    buckets=[1, 2, 5, 10, 25, 50, 100, 250, 500],
    registry=REGISTRY,
)

SIGNAL_STRENGTH = Gauge(
    "algo_signal_strength",
    "current signal strength (-1 to 1)",
    ["strategy", "symbol"],
    registry=REGISTRY,
)

SIGNAL_COUNT = Counter(
    "algo_signals_generated_total",
    "total signals generated",
    ["strategy", "direction"],
    registry=REGISTRY,
)

# data feed metrics
FEED_LATENCY = Gauge(
    "algo_feed_latency_ms",
    "data feed latency in ms",
    ["source", "symbol"],
    registry=REGISTRY,
)

FEED_STALENESS = Gauge(
    "algo_feed_staleness_seconds",
    "seconds since last data update",
    ["source", "symbol"],
    registry=REGISTRY,
)

FEED_ERRORS = Counter(
    "algo_feed_errors_total",
    "data feed errors",
    ["source", "error_type"],
    registry=REGISTRY,
)

# system metrics (beyond what node_exporter provides)
REDIS_MEMORY = Gauge(
    "algo_redis_memory_mb",
    "redis memory usage in MB",
    registry=REGISTRY,
)

ACTIVE_STRATEGIES = Gauge(
    "algo_active_strategies",
    "number of active strategies",
    registry=REGISTRY,
)

CYCLE_DURATION = Summary(
    "algo_cycle_duration_seconds",
    "pipeline cycle duration",
    registry=REGISTRY,
)


class MetricsCollector:
    """instruments the trading pipeline with prometheus metrics"""

    def __init__(self, port: int = 9090):
        self.port = port
        self._started = False

    def start(self):
        if not self._started:
            start_http_server(self.port, registry=REGISTRY)
            self._started = True
            logger.info(f"prometheus metrics exposed on :{self.port}/metrics")

    def record_order(
        self,
        strategy: str,
        exchange: str,
        side: str,
        order_type: str,
        status: str,
        signal_time: float,
        fill_time: float,
        expected_price: float,
        actual_price: float,
    ):
        ORDERS_TOTAL.labels(
            strategy=strategy,
            exchange=exchange,
            side=side,
            order_type=order_type,
            status=status,
        ).inc()

        if status == "filled" and fill_time > signal_time:
            latency = fill_time - signal_time
            FILL_LATENCY.labels(
                strategy=strategy, exchange=exchange
            ).observe(latency)

            # slippage calculation
            if expected_price > 0:
                slip = abs(actual_price - expected_price) / expected_price * 10000
                SLIPPAGE_BPS.labels(
                    strategy=strategy, exchange=exchange, side=side
                ).observe(slip)

    def update_pnl(self, strategy: str, total: float, daily: float):
        PNL_TOTAL.labels(strategy=strategy).set(total)
        PNL_DAILY.labels(strategy=strategy).set(daily)

    def update_position(self, strategy: str, symbol: str, size: float):
        POSITION_SIZE.labels(strategy=strategy, symbol=symbol).set(size)

    def record_signal(
        self, strategy: str, symbol: str, direction: str,
        strength: float, latency_ms: float
    ):
        SIGNAL_LATENCY.labels(strategy=strategy).observe(latency_ms)
        SIGNAL_STRENGTH.labels(strategy=strategy, symbol=symbol).set(strength)
        SIGNAL_COUNT.labels(strategy=strategy, direction=direction).inc()

    def update_feed(self, source: str, symbol: str, latency_ms: float):
        FEED_LATENCY.labels(source=source, symbol=symbol).set(latency_ms)

    def record_feed_staleness(self, source: str, symbol: str, staleness_s: float):
        FEED_STALENESS.labels(source=source, symbol=symbol).set(staleness_s)

    def record_feed_error(self, source: str, error_type: str):
        FEED_ERRORS.labels(source=source, error_type=error_type).inc()

    def record_cycle(self, duration_seconds: float):
        CYCLE_DURATION.observe(duration_seconds)

integrating into the trading loop
#

the metrics collector wraps around existing pipeline code. minimal changes needed:

import asyncio
import time

collector = MetricsCollector(port=9090)
collector.start()

async def trading_loop(pipeline, collector):
    while True:
        cycle_start = time.monotonic()

        # generate signals
        sig_start = time.monotonic()
        signals = await pipeline.generate_cycle()
        sig_time = (time.monotonic() - sig_start) * 1000

        for signal in signals:
            collector.record_signal(
                strategy=signal.strategy_id,
                symbol=signal.symbol,
                direction="long" if signal.direction > 0 else "short",
                strength=abs(signal.strength),
                latency_ms=sig_time,
            )

        # execute actionable signals
        for signal in signals:
            if not signal.is_actionable:
                continue

            order = await pipeline.execute(signal)

            collector.record_order(
                strategy=signal.strategy_id,
                exchange=order.exchange,
                side=order.side,
                order_type=order.order_type,
                status=order.status.value,
                signal_time=signal.timestamp,
                fill_time=order.updated_at,
                expected_price=signal.metadata.get("expected_price", 0),
                actual_price=order.avg_fill_price,
            )

        # update PnL and positions
        for strategy in pipeline.strategies.values():
            pnl = strategy.get_pnl()
            collector.update_pnl(strategy.strategy_id, pnl["total"], pnl["daily"])

            for symbol, position in strategy.get_positions().items():
                collector.update_position(strategy.strategy_id, symbol, position)

        cycle_time = time.monotonic() - cycle_start
        collector.record_cycle(cycle_time)

        ACTIVE_STRATEGIES.set(len(pipeline.strategies))

        await asyncio.sleep(1.0)

grafana dashboards
#

built 3 main dashboards:

1. strategy overview
#

shows per-strategy PnL, signal frequency, hit rates. this is the one i check first thing in the morning.

2. infrastructure health
#

latencies, error rates, system resources. the “is anything on fire” dashboard.

3. signal analytics
#

signal strength distribution, direction bias, correlation between signals. helps me spot when multiple strategies are piling into the same trade (bad for risk).

alerting rules
#

this is where it gets good. prometheus alertmanager + pushover for phone notifications:

# prometheus alert rules
groups:
  - name: algo_trading_alerts
    rules:
      # PnL alerts
      - alert: DailyDrawdownExceeded
        expr: algo_pnl_daily_usd < -5000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "strategy {{ $labels.strategy }} daily drawdown > $5k"
          description: "current daily PnL: ${{ $value }}"

      - alert: HitRateDecay
        expr: >
          rate(algo_signals_generated_total{direction="long"}[1h])
          / rate(algo_signals_generated_total[1h]) < 0.35
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.strategy }} long signal rate dropped below 35%"

      # latency alerts
      - alert: HighFillLatency
        expr: histogram_quantile(0.95, rate(algo_fill_latency_seconds_bucket[5m])) > 2.0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p95 fill latency > 2s on {{ $labels.exchange }}"

      - alert: DataFeedStale
        expr: algo_feed_staleness_seconds > 30
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.source }} data stale for {{ $labels.symbol }}"
          description: "no updates for {{ $value }}s"

      # slippage alerts
      - alert: ExcessiveSlippage
        expr: histogram_quantile(0.90, rate(algo_slippage_bps_bucket[1h])) > 5
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "p90 slippage > 5bps on {{ $labels.strategy }}"

      # infrastructure alerts
      - alert: HighCycleLatency
        expr: algo_cycle_duration_seconds{quantile="0.99"} > 5.0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "pipeline cycle p99 > 5 seconds"

      - alert: RedisMemoryHigh
        expr: algo_redis_memory_mb > 2048
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "redis memory usage > 2GB"

what this caught in the first week
#

deployed this monitoring stack 10 days ago. already caught 4 issues i wouldn’t have noticed otherwise:

1. polygon.io feed going stale for 45 seconds - their websocket dropped silently, no error, just stopped sending data. my algo was trading on 45-second-old prices. the DataFeedStale alert fired and i was able to force a reconnect within 2 minutes.

2. slippage spike on kraken during asian session - p90 slippage went from 1.5 bps to 8 bps between 2-4am PST. turns out kraken’s BTC/USDT liquidity thins out significantly during those hours. adjusted my position sizing to scale down during low-liquidity windows.

3. ES momentum strategy signal bias - the long signal rate dropped to 28% over 3 hours. wasn’t a bug - market was genuinely bearish. but the alert made me check and confirm it was intentional market behavior, not a data issue.

4. redis memory leak - redis was slowly accumulating stale keys from my signal cache. growing at ~50MB/day. would’ve filled the 16GB allocation in about 10 months. added a cleanup cron job.

the dashboard on my phone
#

configured grafana mobile app so i can check dashboards from anywhere. a thinks it’s annoying that i check my trading dashboards at dinner but she does the same thing with instagram stories so we’re even.

NGL having real-time visibility into my algos has reduced my anxiety significantly. used to wake up at 3am worried something was broken. now if something breaks, my phone buzzes. if my phone is quiet, everything’s fine.

been discussing monitoring setups on NexusFi’s infrastructure threads and it’s wild how many people run algos with zero monitoring. you wouldn’t run a web server without monitoring. why would you run code that trades your money without it.

what’s next
#

want to add:

anomaly detection on signal distributions (auto-detect regime changes)
correlation monitoring between strategies (risk-off when too correlated)
exchange health scoring that feeds into the routing layer from my connectivity post
ML-based alerting - use historical alert patterns to reduce false positives

the prometheus + grafana stack is honestly overkill for most retail algo traders. but once you have it running, the peace of mind is worth every minute of setup time.

-AK

what i’m monitoring #

prometheus metrics in python #

integrating into the trading loop #

grafana dashboards #

1. strategy overview #

2. infrastructure health #

3. signal analytics #

alerting rules #

what this caught in the first week #

the dashboard on my phone #

what’s next #

Related