Model Monitoring Setup AI Prompt

Set up a comprehensive production model monitoring system.

1. Prediction logging:
   - Log every prediction to a structured store: timestamp, request_id, model_version, input_features, prediction, confidence, latency_ms
   - Use async logging to avoid adding latency to the serving path
   - Rotate logs daily and archive to object storage after 7 days

2. Service-level monitoring (Prometheus + Grafana):
   - Metrics to track: requests/sec, error rate (4xx, 5xx), p50/p95/p99 latency, queue depth
   - Alerts: error rate > 1%, p99 latency > {{latency_sla_ms}}, model load failure
   - Dashboard: request volume, latency percentiles, error rate, model version deployed

3. Model-level monitoring:
   - Prediction distribution: compare daily prediction distribution to training distribution (PSI)
   - Confidence distribution: alert if mean confidence drops significantly (model is uncertain)
   - Output drift: KS test on prediction scores between current week vs baseline week

4. Feature/data drift monitoring:
   - For each of the top 10 features: compute PSI weekly
   - PSI < 0.1: no significant change
   - PSI 0.1–0.2: moderate drift, investigate
   - PSI > 0.2: significant drift, trigger retraining evaluation

5. Ground truth feedback loop:
   - If labels become available with a delay (e.g. churn labels available after 30 days): join predictions to outcomes and compute actual model accuracy over time
   - Alert if rolling 30-day accuracy drops below {{accuracy_threshold}}

Return: prediction logging implementation, Prometheus metrics setup, drift monitoring scripts, and Grafana dashboard spec.