Online Drift Detection AI Prompt

Implement online (stream-based) drift detection that detects drift in real time as predictions arrive, rather than in daily batch jobs.

Use case: model serving at > {{throughput}} RPS where drift needs to be detected within {{detection_window}} minutes.

1. ADWIN (Adaptive Windowing) for concept drift:
   - ADWIN maintains a sliding window of recent accuracy values
   - Automatically adjusts window size based on detected distribution changes
   - When the mean of the window changes significantly (using Hoeffding bound), drift is flagged
   - Suitable for: streaming accuracy monitoring when labels are near-real-time
   - Implementation: use the River library (formerly scikit-multiflow)

2. DDM (Drift Detection Method):
   - Tracks error rate mean and standard deviation over a stream of binary correct/incorrect outcomes
   - WARNING level: error_rate + std > baseline + 2×std_baseline
   - DRIFT level: error_rate + std > baseline + 3×std_baseline
   - Reset warning level statistics when drift is detected
   - Lightweight: O(1) memory, suitable for very high throughput

3. KSWIN (Kolmogorov-Smirnov Windowing):
   - Sliding window KS test on a chosen feature or prediction score
   - Compare the oldest {{reference_window}} samples vs newest {{detection_window}} samples
   - Drift flagged when KS p-value < {{alpha}} (e.g. 0.001)
   - Suitable for: feature drift detection in streaming pipelines

4. Integration with serving pipeline:
   - Run drift detectors as a side-car process alongside the serving container
   - Consume from the prediction log stream
   - Emit drift events to an alert topic when drift is detected
   - Circuit breaker: if drift exceeds a critical threshold, automatically route traffic to a fallback model

5. False positive management:
   - Online detectors are sensitive — apply a minimum detection window (don't alert on single anomalous batch)
   - Require drift to be sustained for {{min_sustained_window}} consecutive windows before alerting

Return: ADWIN, DDM, and KSWIN implementations, serving integration design, and false positive management.