Multi-Model Monitoring System AI Prompt

Design a centralized monitoring system that scales to {{num_models}} production ML models.

Challenge: each model has different input features, output types, and business metrics. A one-size-fits-all approach does not work.

1. Model registry integration:
   - Each registered model provides a monitoring config alongside the model artifact
   - Monitoring config specifies: key features to monitor, output type and drift thresholds, business metric to track, retraining trigger conditions, and alert routing

2. Centralized collection layer:
   - Standardized prediction log schema with model-specific payload field for input/output details
   - All models write to the same Kafka topic, partitioned by model_name
   - Central consumer writes to a unified monitoring database partitioned by model_name/date

3. Per-model monitoring jobs:
   - Template monitoring job parameterized by model config
   - Spins up one monitoring job per registered model automatically on new model deployment
   - Each job: reads from the unified monitoring database, applies the model-specific config, and writes results to a monitoring metrics table

4. Unified monitoring dashboard:
   - Overview page: table of all models with health status (🟢/🟡/🔴) based on recent alerts
   - Drill-down per model: serving metrics, prediction distribution, drift scores, recent alerts
   - Cross-model comparison: compare drift patterns across models — correlated drift suggests a shared upstream data issue

5. Alert deduplication and routing:
   - Group alerts from the same model within a 1-hour window to avoid alert storms
   - Route to the correct on-call engineer based on model ownership in the registry
   - Escalation: if alert is not acknowledged within {{escalation_window}} minutes, page the team lead

6. Cost management:
   - Tier models by importance (Tier 1: revenue-critical, Tier 2: operational, Tier 3: experimental)
   - Different monitoring frequencies per tier: T1 = real-time, T2 = hourly, T3 = daily
   - Estimated monitoring cost per model per month

Return: monitoring config schema, centralized collection architecture, per-model job template, and unified dashboard spec.