Model Monitoring AI Prompts

Analyze and optimize the cost of the production model monitoring infrastructure. Current monitoring stack: {{monitoring_stack}} Number of models monitored: {{num_models}} Current monthly monitoring cost: {{current_cost}} 1. Cost breakdown: - Log storage: how many GB of prediction logs are stored? At what cost per GB? - Compute: how many monitoring jobs run per day? What is the compute cost per job? - Query costs: how many analytical queries run against the monitoring database? Cost per query? - Alerting: external alerting services (PagerDuty, OpsGenie) cost per seat/alert - Dashboard: Grafana Cloud or self-hosted cost 2. Sampling strategy for high-throughput models: - For models with > 1M predictions/day: log a stratified sample instead of 100% - Sample rate recommendation: 10% for >1M/day, 50% for 100k-1M/day, 100% for <100k/day - Ensure sample is stratified by prediction score bucket (preserve distribution shape) - Log ALL anomalous predictions regardless of sample rate (score > 0.95 or < 0.05 for classifiers) 3. Log retention optimization: - Tiered storage: hot (last 7 days, queryable), warm (7–90 days, compressed), cold (>90 days, archival) - Pre-aggregate daily statistics (mean, std, percentiles) and retain indefinitely - Delete raw logs after 90 days — aggregate statistics are sufficient for long-term trend analysis 4. Monitoring frequency tiering: - Tier 1 (revenue-critical): real-time serving metrics, hourly drift checks, daily performance - Tier 2 (operational): hourly serving metrics, daily drift checks, weekly performance - Tier 3 (experimental): daily serving metrics, weekly drift checks, no automatic performance tracking 5. Estimated savings from each optimization: - Sampling: saves X% on log storage and compute - Tiered storage: saves Y% on storage - Monitoring frequency tiering: saves Z% on compute Return: cost breakdown analysis, sampling implementation, tiered storage design, and total estimated savings.

Design a ground truth feedback loop that joins delayed labels to predictions for ongoing model performance tracking. Model: {{model_name}} Label delay: labels become available {{label_delay}} after prediction (e.g. 7 days for churn, 30 days for fraud) Label source: {{label_source}} 1. Prediction storage for labeling: - Store predictions with: request_id, entity_id, model_version, prediction, score, prediction_timestamp - Retain predictions for at least label_delay + 30 days buffer - Index on entity_id and prediction_timestamp for efficient label joins 2. Label ingestion pipeline: - Daily job: fetch newly available labels from {{label_source}} - Join to prediction store on entity_id and the relevant time window - Handle multiple labels per entity: use the label with the timestamp closest to the prediction - Label match rate: what % of predictions received a label? Alert if < {{min_label_rate}}% 3. Performance tracking: - Compute rolling metrics over the last 30 days of labeled predictions: - Classification: AUC-ROC, precision, recall, F1, calibration error - Regression: MAE, RMSE, MAPE, prediction bias (mean(prediction - actual)) - Compare to training/validation performance baseline - Plot metric trend: is performance stable, improving, or degrading? 4. Cohort analysis: - Break down performance by: prediction date cohort, model version, user segment - Identify if performance degradation is concentrated in a specific cohort or universal 5. Retraining trigger: - Define threshold: if rolling 30-day AUC drops below {{retrain_threshold}}, trigger retraining pipeline - Distinguish signal from noise: require the drop to persist for {{consecutive_days}} consecutive days 6. Feedback to training: - Append newly labeled examples to the training dataset for the next retraining run - Track data freshness: what % of the training set is from the last 90 days? Return: prediction storage schema, label join pipeline, performance tracking queries, and retraining trigger logic.

Build an early warning system for model performance degradation before ground truth labels arrive. Since labels often arrive days or weeks after predictions, rely on proxy signals that correlate with model quality. 1. Proxy signal monitoring (no labels required): a. Confidence score degradation: - Track the distribution of model confidence scores daily - A well-calibrated model should have a stable confidence distribution - Alert if mean confidence drops > {{confidence_drop_threshold}} or if the distribution becomes more uniform (model is less certain) b. Prediction entropy (for classifiers): - Entropy = -Σ p_i × log(p_i) across classes - Higher entropy = less confident predictions - Alert if rolling 7-day mean entropy increases > 1σ above the baseline mean entropy c. Feature anomaly rate: - Track the % of incoming requests where at least one feature falls outside the training distribution - A rising anomaly rate predicts performance degradation before it appears in labels d. Business metric correlation (if available): - Track downstream business metrics that the model influences (conversion rate, fraud rate) - Unexplained movements in business metrics may indicate model degradation 2. Composite degradation score: - Combine multiple proxy signals into a single degradation score (0–100) - Weight by historical correlation with actual performance drops - Thresholds: score > 60 → Slack alert, score > 80 → PagerDuty 3. Alert content: - Current degradation score and contributing signals - Trend: is degradation accelerating or stable? - Recommended action: monitor / investigate / retrain - Link to monitoring dashboard and recent prediction sample for manual inspection 4. Validation: - Backtest the proxy signals on historical data: did they predict known past degradation events? - Report: lead time before degradation became visible in labels, false positive rate Return: proxy signal monitoring code, composite score calculation, alerting logic, and backtest methodology.

Step 1: Define monitoring requirements — for this model, specify: what constitutes a healthy prediction distribution, the acceptable performance floor, the label availability timeline, and the business cost of undetected degradation vs false alarms. Step 2: Instrument prediction logging — add async prediction logging to the serving layer. Log: request_id, model_version, features, prediction, confidence, latency. Verify logs are flowing to the storage layer. Step 3: Establish baselines — compute reference distributions for all features and model outputs using the first 2 weeks of production data (or the validation set if launching new). Store baseline statistics. Step 4: Deploy serving metrics — instrument Prometheus metrics (RPS, latency, error rate). Set up Grafana dashboard. Configure AlertManager rules for SLA violations. Step 5: Deploy drift monitors — implement daily PSI checks for top features and prediction distribution. Set thresholds and alert routing. Run a backtest to validate alert sensitivity. Step 6: Deploy performance tracking — implement ground truth join pipeline. Set up rolling performance metric computation. Define retraining trigger condition. Step 7: Document and hand off — write the monitoring runbook: what each alert means, initial triage steps, escalation path, and how to silence a false alarm. Get sign-off from the on-call team before go-live.

Design a centralized monitoring system that scales to {{num_models}} production ML models. Challenge: each model has different input features, output types, and business metrics. A one-size-fits-all approach does not work. 1. Model registry integration: - Each registered model provides a monitoring config alongside the model artifact - Monitoring config specifies: key features to monitor, output type and drift thresholds, business metric to track, retraining trigger conditions, and alert routing 2. Centralized collection layer: - Standardized prediction log schema with model-specific payload field for input/output details - All models write to the same Kafka topic, partitioned by model_name - Central consumer writes to a unified monitoring database partitioned by model_name/date 3. Per-model monitoring jobs: - Template monitoring job parameterized by model config - Spins up one monitoring job per registered model automatically on new model deployment - Each job: reads from the unified monitoring database, applies the model-specific config, and writes results to a monitoring metrics table 4. Unified monitoring dashboard: - Overview page: table of all models with health status (🟢/🟡/🔴) based on recent alerts - Drill-down per model: serving metrics, prediction distribution, drift scores, recent alerts - Cross-model comparison: compare drift patterns across models — correlated drift suggests a shared upstream data issue 5. Alert deduplication and routing: - Group alerts from the same model within a 1-hour window to avoid alert storms - Route to the correct on-call engineer based on model ownership in the registry - Escalation: if alert is not acknowledged within {{escalation_window}} minutes, page the team lead 6. Cost management: - Tier models by importance (Tier 1: revenue-critical, Tier 2: operational, Tier 3: experimental) - Different monitoring frequencies per tier: T1 = real-time, T2 = hourly, T3 = daily - Estimated monitoring cost per model per month Return: monitoring config schema, centralized collection architecture, per-model job template, and unified dashboard spec.

Build a system to monitor the distribution of model predictions over time and detect output drift. Model type: {{model_type}} (binary classifier / multiclass / regression / ranking) 1. Baseline distribution: - Compute the prediction distribution on a held-out reference dataset (or first 2 weeks of production logs) - For classifiers: positive rate, score distribution (histogram with 20 bins), confusion matrix on labeled data - For regression: mean, std, percentiles (5th, 25th, 50th, 75th, 95th), histogram - Store baseline statistics in a metadata table for comparison 2. Daily distribution comparison: - Compute the same statistics on the last 24 hours of predictions - Statistical tests: - Classifier scores: Kolmogorov-Smirnov test (KS test) vs baseline distribution - Classifier positive rate: two-proportion z-test vs baseline positive rate - Regression outputs: KS test + t-test for mean shift - Population Stability Index (PSI): - PSI < 0.1: no significant shift - PSI 0.1–0.2: moderate shift — investigate - PSI > 0.2: significant shift — alert 3. Temporal patterns: - Plot rolling 7-day mean prediction score over time - Plot rolling positive rate (for classifiers) over time - Flag: sudden jumps (step change) vs gradual drift (slow trend) - Annotate with model deployment dates to distinguish drift from deployment effects 4. Segment-level monitoring: - Compute prediction distribution separately for key segments (region, user type, device) - Flag any segment where distribution diverges significantly from the population 5. Alerting: - PSI > 0.2 on overall prediction distribution: Slack alert to ML team (P2) - Positive rate changes > 2× std from 30-day rolling average: Slack alert (P2) - Positive rate changes > 5× std: PagerDuty (P1) — likely a model or feature pipeline failure Return: baseline computation script, daily comparison script, PSI calculation function, and alerting configuration.

Design and implement a production prediction logging system for this ML model. Model: {{model_name}} Serving framework: {{serving_framework}} Expected throughput: {{requests_per_second}} requests/sec 1. What to log per prediction: - request_id: unique identifier for traceability - model_name and model_version: which exact artifact served this request - timestamp: ISO 8601, UTC - input_features: the feature vector sent to the model (after preprocessing) - raw_input: the original unprocessed input (for debugging preprocessing bugs) - prediction: the model's output (class label, score, or generated value) - prediction_probability or confidence: confidence score where applicable - latency_ms: total inference time - serving_node: which pod/instance served the request (for debugging node-specific issues) 2. Async logging (never block the serving path): - Write to an in-memory queue in the request handler - Background thread drains the queue and writes to the log sink in batches - If the log sink is unavailable: drop logs gracefully, do not fail the prediction 3. Log sink options by throughput: - < 1k RPS: write directly to a structured log file, ship with Fluentd/Logstash - 1k–100k RPS: write to Kafka topic, consume to object storage and OLAP table - > 100k RPS: write to a high-throughput sink (Kinesis Data Firehose, Pub/Sub) with batching 4. Storage and retention: - Raw logs: object storage (S3/GCS), partitioned by model_name/date, retained for 90 days - Queryable table: OLAP warehouse (BigQuery/Snowflake), retained for 1 year - PII handling: mask or hash any PII fields in the feature log before storage 5. Log schema versioning: - Version the log schema alongside the model version - Never remove fields from the log schema — add new fields with NULL backfill for old records Return: prediction log schema (JSON), async logging implementation, sink configuration for the given throughput, and PII masking approach.

Design a production model serving metrics dashboard using Prometheus and Grafana. Model: {{model_name}} SLA targets: p99 latency < {{latency_sla_ms}}ms, error rate < {{error_rate_sla}}%, availability > {{availability_sla}}% 1. Prometheus metrics to instrument (add to the serving application): Service-level metrics: - model_requests_total (counter): labeled by model_version, endpoint, status_code - model_request_duration_seconds (histogram): labeled by model_version, endpoint Buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0] - model_errors_total (counter): labeled by model_version, error_type - model_in_flight_requests (gauge): current concurrent requests Model-level metrics: - prediction_score_distribution (histogram): distribution of output confidence scores - feature_value_distribution (histogram): one per key feature, for drift detection - model_load_time_seconds (gauge): time to load model at startup Infrastructure metrics: - gpu_utilization_percent (gauge): per serving node - gpu_memory_used_bytes (gauge): per serving node 2. Grafana dashboard panels: Row 1 — SLA Overview: - Current request rate (RPS) - p50 / p95 / p99 latency (time series) - Error rate % (time series, red threshold at SLA) - Availability % (stat panel, green/red) Row 2 — Model Health: - Prediction score distribution (heatmap over time) - Request volume by model version (stacked bar — useful during rollouts) - Top error types (table) Row 3 — Infrastructure: - GPU utilization per node (time series) - GPU memory used per node (time series) - Pod count (gauge) 3. Alerting rules (Prometheus AlertManager): - HighErrorRate: error_rate > {{error_rate_sla}}% for 5 minutes → PagerDuty (P1) - HighLatency: p99 > {{latency_sla_ms}}ms for 10 minutes → Slack (P2) - ModelDown: no successful requests for 2 minutes → PagerDuty (P0) - LowThroughput: RPS drops > 50% vs 1-hour average → Slack (P2) Return: Prometheus instrumentation code, AlertManager rules YAML, and Grafana dashboard JSON.

Implement shadow mode deployment to evaluate a new model version in production without serving its predictions to users. In shadow mode: all requests are served by the champion model. The challenger model receives a copy of every request, runs inference, and logs its predictions — but its output is discarded and never returned to the user. 1. Shadow mode architecture: - Duplicate every incoming request to the challenger model asynchronously - The challenger call must never block or slow the champion response - Use a fire-and-forget async call with a timeout of {{shadow_timeout_ms}}ms - If the challenger times out or errors: log the failure, continue without impact to the user 2. Shadow prediction logging: - Log champion and challenger predictions with the same request_id for comparison - Schema: request_id, champion_prediction, champion_score, challenger_prediction, challenger_score, timestamp 3. Comparison analysis (run daily): - Agreement rate: % of requests where champion and challenger produce the same prediction - Score correlation: Pearson correlation between champion and challenger scores - Distribution comparison: KS test between champion and challenger score distributions - Disagreement analysis: for cases where they disagree, which model is likely correct? Sample 50 and manually inspect - Latency comparison: challenger p99 vs champion p99 (challenger must meet latency SLA) 4. Promotion criteria: - Run shadow mode for {{shadow_duration}} days minimum - Challenger must: pass all serving metric requirements, show better or equal distribution quality, meet latency SLA - If labels are available: measure challenger performance on labeled shadow period data 5. Shadow mode cost: - Shadow mode doubles compute cost — plan for this in the infrastructure budget - Use a smaller replica count for the challenger during shadow mode Return: shadow mode routing implementation, comparison analysis script, and promotion decision criteria.