Model Performance Degradation Alert AI Prompt

Build an early warning system for model performance degradation before ground truth labels arrive.

Since labels often arrive days or weeks after predictions, rely on proxy signals that correlate with model quality.

1. Proxy signal monitoring (no labels required):

   a. Confidence score degradation:
   - Track the distribution of model confidence scores daily
   - A well-calibrated model should have a stable confidence distribution
   - Alert if mean confidence drops > {{confidence_drop_threshold}} or if the distribution becomes more uniform (model is less certain)

   b. Prediction entropy (for classifiers):
   - Entropy = -Σ p_i × log(p_i) across classes
   - Higher entropy = less confident predictions
   - Alert if rolling 7-day mean entropy increases > 1σ above the baseline mean entropy

   c. Feature anomaly rate:
   - Track the % of incoming requests where at least one feature falls outside the training distribution
   - A rising anomaly rate predicts performance degradation before it appears in labels

   d. Business metric correlation (if available):
   - Track downstream business metrics that the model influences (conversion rate, fraud rate)
   - Unexplained movements in business metrics may indicate model degradation

2. Composite degradation score:
   - Combine multiple proxy signals into a single degradation score (0–100)
   - Weight by historical correlation with actual performance drops
   - Thresholds: score > 60 → Slack alert, score > 80 → PagerDuty

3. Alert content:
   - Current degradation score and contributing signals
   - Trend: is degradation accelerating or stable?
   - Recommended action: monitor / investigate / retrain
   - Link to monitoring dashboard and recent prediction sample for manual inspection

4. Validation:
   - Backtest the proxy signals on historical data: did they predict known past degradation events?
   - Report: lead time before degradation became visible in labels, false positive rate

Return: proxy signal monitoring code, composite score calculation, alerting logic, and backtest methodology.