Gradient Accumulation AI Prompt

Implement gradient accumulation to simulate a larger effective batch size on limited GPU memory.

Target effective batch size: {{effective_batch_size}}
Physical batch size that fits in GPU memory: {{physical_batch_size}}
Accumulation steps: effective_batch_size / physical_batch_size

1. Basic gradient accumulation loop:
   - Accumulate gradients for N steps before calling optimizer.step()
   - Divide loss by accumulation steps to maintain consistent gradient scale
   - Zero gradients only after optimizer.step(), not every batch

2. Mixed precision compatibility:
   - Use GradScaler correctly with accumulation — only call scaler.update() after optimizer.step()
   - Correct scaler.scale(loss) placement

3. DDP compatibility:
   - Use model.no_sync() context manager for accumulation steps to prevent premature gradient sync
   - Only sync on the last accumulation step

4. Learning rate adjustment:
   - Learning rate should be tuned for the effective batch size, not physical batch size
   - Linear scaling rule: lr = base_lr × (effective_batch_size / reference_batch_size)

5. Scheduler compatibility:
   - Step scheduler based on optimizer steps (after accumulation), not raw batches

6. Verification:
   - Show how to verify that accumulation of N small batches produces identical gradients to 1 large batch

Return: complete gradient accumulation training loop with DDP and mixed precision support.