| Paper ID | ASPS-7.5 |
| Paper Title |
SQWA: STOCHASTIC QUANTIZED WEIGHT AVERAGING FOR IMPROVING THE GENERALIZATION CAPABILITY OF LOW-PRECISION DEEP NEURAL NETWORKS |
| Authors |
Sungho Shin, Yoonho Boo, Wonyong Sung, Seoul National University, South Korea |
| Session | ASPS-7: Data Science & Machine Learning |
| Location | Gather.Town |
| Session Time: | Thursday, 10 June, 16:30 - 17:15 |
| Presentation Time: | Thursday, 10 June, 16:30 - 17:15 |
| Presentation |
Poster
|
| Topic |
Applied Signal Processing Systems: Signal Processing Systems [DIS-EMSA] |
| IEEE Xplore Open Preview |
Click here to view in IEEE Xplore |
| Virtual Presentation |
Click here to watch in the Virtual Conference |
| Abstract |
Low-precision deep neural networks (DNNs) are very needed for efficient implementations, but severe quantization of weights often sacrifices the generalization capability and lowers the test accuracy. We present a new quantized neural network optimization approach, stochastic quantized weight averaging (SQWA), to design low-precision DNNs with good generalization capability using model averaging. The proposed approach includes (1) floating-point model training, (2) direct quantization of weights, (3) capturing multiple low precision models during retraining with cyclical learning rates, (4) averaging the captured models, and (5) re-quantizing the averaged model and fine-tuning it with low-learning rates. With SQWA training, we could develop the best performing QDNNs for image classification on ImageNet datasets and also for semantic segmentation on Pascal VOC 2012 dataset. |
