Memory Optimization AI Prompt

Optimize GPU memory usage for this model to fit a larger batch size or a bigger model on available hardware.

Target: fit on {{gpu_vram}}GB GPU with maximum batch size.

Apply these techniques in order of implementation complexity:

1. Immediate wins (< 1 hour to implement):
   - Enable mixed precision (fp16/bf16) — saves 40–50% memory
   - Set optimizer to use 8-bit Adam (bitsandbytes) — saves optimizer state memory (~75% of optimizer memory)
   - Use set_to_none=True in optimizer.zero_grad()
   - Detach intermediate tensors not needed for backprop
   - Delete unused variables and call torch.cuda.empty_cache() at epoch end

2. Gradient techniques:
   - Gradient accumulation — simulate larger batches with smaller physical batch
   - Gradient checkpointing (activation checkpointing) — recompute activations during backward pass instead of storing them. Trade compute for memory (~30–40% memory reduction, ~20% slower)

3. Model architecture changes:
   - Replace nn.Linear with nn.Linear in 8-bit (bitsandbytes) for inference
   - Use flash attention instead of standard attention (for transformer models)
   - Reduce model width or depth if memory is the primary constraint

4. Advanced: Parameter-Efficient Fine-Tuning:
   - LoRA: train only low-rank adapter matrices (< 1% of parameters)
   - Prefix tuning or prompt tuning for large language models

5. Memory profiling:
   - torch.cuda.memory_summary() after each forward/backward
   - Record peak memory: torch.cuda.max_memory_allocated()
   - Identify which layer consumes the most memory

Return: memory optimization implementation ordered by complexity, expected savings per technique, and memory profiling code.