Latency Optimization AI Prompt

Optimize inference latency for this model serving endpoint to meet a p99 latency target of {{latency_target_ms}}ms.

Current p99 latency: {{current_latency_ms}}ms

Work through these optimizations in order of impact:

1. Profile first:
   - Break down request latency into: network, preprocessing, model inference, postprocessing
   - Identify which component dominates

2. Model-level optimizations:
   - Convert to TorchScript (torch.jit.trace or torch.jit.script)
   - Export to ONNX and run with ONNX Runtime (often 2–5× faster than PyTorch for inference)
   - Enable ONNX Runtime execution providers: CUDAExecutionProvider for GPU, TensorRT for maximum speed

3. Batching optimizations:
   - Implement dynamic batching: collect requests for {{batch_wait_ms}}ms, then process as a batch
   - Find optimal batch size: benchmark batch sizes 1, 2, 4, 8, 16, 32 and plot throughput vs latency tradeoff

4. Hardware optimizations:
   - Warm up the model at startup with dummy forward passes to trigger JIT compilation
   - Pin model to a specific GPU with CUDA_VISIBLE_DEVICES
   - Use CUDA streams to overlap data transfer and computation

5. Application-level optimizations:
   - Response caching for repeated identical inputs (LRU cache with size limit)
   - Connection pooling and keep-alive for HTTP
   - Reduce serialization overhead: use MessagePack or protobuf instead of JSON for high-throughput

Return: profiling methodology, optimization checklist with estimated gains, and benchmark code.