Batch Inference Pipeline AI Prompt

Build an efficient batch inference pipeline for running predictions on {{dataset_size}} records.

1. Data loading strategy:
   - Stream from source (S3, database, or file) without loading all into memory
   - Use a DataLoader with appropriate batch size for maximum GPU utilization
   - Parallelize I/O with prefetching: load next batch while GPU processes current

2. Inference optimization:
   - model.eval() and torch.no_grad()
   - Mixed precision inference with torch.autocast
   - Disable gradient computation globally: torch.set_grad_enabled(False)
   - TorchScript or ONNX export for faster inference if model is compatible

3. Output handling:
   - Buffer predictions in memory and write to output in chunks (avoid one write per sample)
   - Write to Parquet for efficient downstream use
   - Include input ID and model version in output for traceability

4. Fault tolerance:
   - Checkpoint progress: track which batches are complete
   - Resume from last successful batch on failure
   - Handle individual batch errors without killing the whole pipeline

5. Throughput optimization:
   - Profile to find the bottleneck: I/O, CPU preprocessing, or GPU inference
   - Dynamic batching: collect samples until batch is full or timeout is reached
   - Multi-process inference for CPU-only models

6. Monitoring:
   - Log throughput (samples/sec) and ETA every 100 batches
   - Log GPU memory usage and utilization
   - Alert if error rate exceeds 1%

Return: complete batch inference script with progress tracking and fault tolerance.