Robust Knowledge Distillation by pytorch

Implementation of several KD papers using pytorch.

Dependencies

• python>=3.7
• torch>=1.6
• torchvision>=0.7

TODO

• Add experimental codes for poison attack under KD
• Add experimental codes for adversarial training under KD
• Implement core procedure in attention transfer paper
• Implement core KD procedures in FitNets paper
• Add tests for network performance on MNIST and CIFAR10
• Add FitNet
• Add Maxout Network

Reference

• Distilling the Knowledge in a Neural Network
• Maxout Networks
• FitNets: Hints for Thin Deep Nets
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Related repos

• maxout
• FitNets
• PyTorch_CIFAR10