Knowledge Distillation

Training a smaller student model to match the outputs of a larger teacher model, retaining much of the quality at a fraction of the cost.