Ground Truth Feedback Loop AI Prompt

Design a ground truth feedback loop that joins delayed labels to predictions for ongoing model performance tracking.

Model: {{model_name}}
Label delay: labels become available {{label_delay}} after prediction (e.g. 7 days for churn, 30 days for fraud)
Label source: {{label_source}}

1. Prediction storage for labeling:
   - Store predictions with: request_id, entity_id, model_version, prediction, score, prediction_timestamp
   - Retain predictions for at least label_delay + 30 days buffer
   - Index on entity_id and prediction_timestamp for efficient label joins

2. Label ingestion pipeline:
   - Daily job: fetch newly available labels from {{label_source}}
   - Join to prediction store on entity_id and the relevant time window
   - Handle multiple labels per entity: use the label with the timestamp closest to the prediction
   - Label match rate: what % of predictions received a label? Alert if < {{min_label_rate}}%

3. Performance tracking:
   - Compute rolling metrics over the last 30 days of labeled predictions:
     - Classification: AUC-ROC, precision, recall, F1, calibration error
     - Regression: MAE, RMSE, MAPE, prediction bias (mean(prediction - actual))
   - Compare to training/validation performance baseline
   - Plot metric trend: is performance stable, improving, or degrading?

4. Cohort analysis:
   - Break down performance by: prediction date cohort, model version, user segment
   - Identify if performance degradation is concentrated in a specific cohort or universal

5. Retraining trigger:
   - Define threshold: if rolling 30-day AUC drops below {{retrain_threshold}}, trigger retraining pipeline
   - Distinguish signal from noise: require the drop to persist for {{consecutive_days}} consecutive days

6. Feedback to training:
   - Append newly labeled examples to the training dataset for the next retraining run
   - Track data freshness: what % of the training set is from the last 90 days?

Return: prediction storage schema, label join pipeline, performance tracking queries, and retraining trigger logic.