Inference Caching Strategy AI Prompt

Design and implement an inference caching strategy to reduce redundant computations and improve throughput.

1. Request-level caching:
   - Identify if this model is likely to receive repeated identical inputs (recommender systems, classification of common queries)
   - Implement LRU cache with maximum {{cache_size}} entries
   - Cache key: SHA256 hash of the serialized input tensor
   - Cache hit/miss rate monitoring: log and alert if hit rate drops below expected

2. KV cache (for transformer/autoregressive models):
   - Implement key-value cache for incremental generation
   - Pre-allocate KV cache to avoid dynamic memory allocation during generation
   - Cache eviction policy for long-context scenarios

3. Embedding cache:
   - If the model has a lookup-table style embedding layer for entity IDs, cache frequently accessed embeddings in a dict
   - Warm up the embedding cache with the top {{topk}} most frequent entity IDs at startup

4. Preprocessing cache:
   - Cache the result of expensive preprocessing steps (tokenization, feature extraction) keyed by raw input
   - Use Redis for distributed caching across multiple serving replicas

5. Cache invalidation:
   - When a new model version is deployed, invalidate the entire cache
   - Version the cache key with the model version string

6. Staleness handling:
   - Set TTL (time-to-live) per cache tier based on how frequently inputs change

Return: caching implementation, Redis integration, cache warming script, and monitoring setup.