A Greedy Algorithm for Quantizing Neural Networks
Authors: Eric Lybrand, Rayan Saab
JMLR 2021 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide numerical experiments, on multi-layer networks, to illustrate the performance of our methods on MNIST and CIFAR10 data, as well as for quantizing the VGG16 network using Image Net data. (Abstract) We present three stylized examples which show how our proposed quantization algorithm affects classification accuracy on three benchmark data sets. (Section 6, Numerical Simulations) |
| Researcher Affiliation | Academia | Eric Lybrand EMAIL Department of Mathematics University of California, San Diego San Diego, CA 92121, USA Rayan Saab EMAIL Department of Mathematics, and Halicioglu Data Science Institute University of California, San Diego San Diego, CA 92121, USA |
| Pseudocode | No | The paper describes the algorithm using mathematical equations (2) and (3) and explains the steps in prose, but it does not present a formally labeled "Pseudocode" or "Algorithm" block. |
| Open Source Code | Yes | We have made our code available on Git Hub at https://github.com/elybrand/quantized_ neural_networks. |
| Open Datasets | Yes | We provide numerical experiments, on multi-layer networks, to illustrate the performance of our methods on MNIST and CIFAR10 data, as well as for quantizing the VGG16 network using Image Net data. (Abstract) We look at classifying digits from the MNIST data set... classifying images from the CIFAR10 data set... and finally looking at classifying images from the ILSVRC2012 data set, also known as Image Net. (Section 6, Numerical Simulations) |
| Dataset Splits | Yes | We trained the unquantized network on the full training set of 60, 000 digits without any preprocessing. 20% of the training data was used as validation during training. We then tested on the remaining 10, 000 images not used in the training set. (Section 6.1, Multilayer Perceptron with MNIST) We trained the unquantized network on the full set of 50, 000 training images. For training the quantization we only used the first 5, 000 images from the training set. (Section 6.2, Convolutional Neural Network with CIFAR10) This data set contains 50,000 images with 1,000 categories. 1500 images were randomly chosen to learn the quantization. To assess the quality of the quantized network we used 20000 randomly chosen images disjoint from the set of images used to perform the quantization. (Section 6.3, VGG16 on Imagenet Data) |
| Hardware Specification | Yes | We trained both networks using Keras (Chollet et al., 2015) with the Tensorflow backend on a a 2020 Mac Book Pro with a M1 chip and 16GB of RAM. |
| Software Dependencies | No | We trained both networks using Keras (Chollet et al., 2015) with the Tensorflow backend... (Section 6, Numerical Simulations) The paper mentions Keras and TensorFlow but does not provide specific version numbers for these software dependencies. |
| Experiment Setup | Yes | We used categorical cross entropy as our loss function during training and the Adam optimizer... for 100 epochs with a minibatch size of 128. After training the unquantized model we used 25, 000 samples from the training set to train the quantization. We used the same data to quantize each layer rather than splitting the data for each layer. For this experiment we restricted the alphabet to be ternary and cross-validated over the alphabet scalar Cα {1, 2, . . . , 10}. (Section 6.1, Multilayer Perceptron with MNIST) Finally, we train the network to minimize categorical cross entropy using stochastic gradient descent with a learning rate of 10 4, momentum of 0.9, and a minibatch size of 64 for 400 epochs. ...we cross-validated the alphabet scalars Cα over the range {2, 3, 4, 5, 6} and chose the best scalar for the benchmark MSQ network and the best GPFQ quantized network separately. Additionally, we cross-validated over the number of elements in the quantization alphabet, ranging over the set M {3, 4, 8, 16} which corresponds to the set of bit budgets {log2(3), 2, 3, 4}. (Section 6.2, Convolutional Neural Network with CIFAR10) |