Feature Store Integration AI Prompt

Design the integration between this ML model serving system and a feature store (Feast / Tecton / Hopsworks).

1. Feature retrieval at inference time:
   - Online store lookup: retrieve pre-computed features for entity_id in < 5ms
   - Handle missing entities: define fallback values or reject the request
   - Batch feature lookup for batch inference: use get_online_features with list of entity IDs

2. Feature freshness:
   - Define the maximum acceptable feature age for each feature group
   - Add feature timestamp to the inference request response for debugging
   - Alert if feature freshness degrades beyond threshold

3. Training-serving skew prevention:
   - Use the exact same feature definitions for both training (offline store) and serving (online store)
   - Log features served at inference time to a feature log table
   - Compare feature distributions in the log vs training data to detect skew

4. Point-in-time correct training data:
   - Use feature store's point-in-time join to generate training data
   - Ensure no future feature values leak into training features

5. Feature store client configuration:
   - Initialize client with retry logic and connection pooling
   - Circuit breaker: if feature store is unavailable, fall back to default features with a flag in the response

6. Monitoring:
   - Log feature store latency per request
   - Alert on feature store connection errors

Return: feature retrieval code, training data generation script, skew detection setup, and circuit breaker implementation.