Knowledge Distillation AI Prompt

Implement knowledge distillation to train a smaller student model to match a larger teacher model.

Teacher model: {{teacher_model}} (large, high-accuracy, slow)
Student model: {{student_model}} (small, faster, to be trained)

1. Soft target distillation (Hinton et al. 2015):
   - Get teacher soft probabilities: softmax(teacher_logits / temperature)
   - Student loss = α × KL_divergence(student_soft, teacher_soft) + (1-α) × CrossEntropy(student, hard_labels)
   - Temperature T: higher T produces softer distributions (try T=3, T=5, T=10)
   - α: weight between distillation loss and task loss (try α=0.7)

2. Intermediate layer distillation (better for deep networks):
   - Match intermediate feature maps between teacher and student layers
   - Use an adapter layer if teacher and student have different hidden dimensions
   - Feature distillation loss: MSE(student_features, teacher_features)

3. Training procedure:
   - Freeze teacher model (no gradients)
   - Train student with combined loss
   - Use a slightly higher learning rate than training from scratch
   - Run for same number of epochs as training student from scratch

4. Evaluation:
   - Student accuracy vs teacher accuracy
   - Student accuracy vs same architecture trained from scratch (distillation should outperform)
   - Student inference latency vs teacher inference latency

5. Self-distillation variant:
   - If no pre-trained teacher exists: use the model's own earlier epochs as the teacher

Return: distillation training loop, temperature sweep results, student vs teacher benchmark, and comparison to training from scratch.