Post-Training Quantization AI Prompt

Apply post-training quantization (PTQ) to reduce model size and inference latency.

1. INT8 static quantization (PyTorch):
   - Prepare model: torch.quantization.prepare with a QConfig
   - Calibrate on a representative dataset (100–1000 samples): run forward passes to collect activation statistics
   - Convert: torch.quantization.convert to replace float ops with int8 ops
   - Save and measure: model size before vs after, inference latency before vs after

2. INT8 dynamic quantization:
   - torch.quantization.quantize_dynamic for models where activation ranges vary greatly
   - Suitable for: LSTMs, linear layers in NLP models
   - No calibration step needed

3. Quantization-aware training (QAT) if accuracy drops > 1%:
   - Insert fake quantization nodes during training
   - Fine-tune for {{qat_epochs}} epochs at a lower learning rate
   - Convert to fully quantized model after training

4. Accuracy validation:
   - Evaluate quantized model on the full validation set
   - Acceptable accuracy drop: < 1% for most production use cases
   - If accuracy drops significantly: try QAT, or quantize only the later layers

5. ONNX + ONNX Runtime INT8:
   - Export to ONNX, then apply ONNXRuntime quantization
   - ort.quantization.quantize_dynamic or quantize_static
   - Often faster than PyTorch native quantization on CPU

Return: PTQ implementation, QAT setup, accuracy comparison table, and latency/size improvement metrics.