Memory-augmented neural networks (MANNs) refer to a class of neural network models equipped with external memory (such as neural Turing machines and memory networks). These neural networks outperform conventional recurrent neural networks (RNNs) in terms of learning long-term dependency, allowing them to solve intriguing AI tasks that would otherwise be hard to address. This paper concerns the problem of quantizing MANNs. Quantization is known to be effective when we deploy deep models on embedded systems with limited resources. Furthermore, quantization can substantially reduce the energy consumption of the inference procedure. These benefits justify recent developments of quantized multilayer perceptrons, convolutional networks, and RNNs. However, no prior work has reported the successful quantization of MANNs. The in-depth analysis presented here reveals various challenges that do not appear in the quantization of the other networks. Without addressing them properly, quantized MANNs would normally suffer from excessive quantization error which leads to degraded performance. In this paper, we identify memory addressing (specifically, content-based addressing) as the main reason for the performance degradation and propose a robust quantization method for MANNs to address the challenge. In our experiments, we achieved a computation-energy gain of 22× with 8-bit fixed-point and binary quantization compared to the floating-point implementation. Measured on the bAbI dataset, the resulting model, named the quantized MANN (Q-MANN), improved the error rate by 46% and 30% with 8-bit fixed-point and binary quantization, respectively, compared to the MANN quantized using conventional techniques.
|Title of host publication||32nd AAAI Conference on Artificial Intelligence, AAAI 2018|
|Number of pages||8|
|State||Published - 2018|
|Event||32nd AAAI Conference on Artificial Intelligence, AAAI 2018 - New Orleans, United States|
Duration: 2 Feb 2018 → 7 Feb 2018
|Name||32nd AAAI Conference on Artificial Intelligence, AAAI 2018|
|Conference||32nd AAAI Conference on Artificial Intelligence, AAAI 2018|
|Period||2/02/18 → 7/02/18|
Bibliographical noteFunding Information:
This research was supported by Ministry of Science, ICT and Future Planning (Basic Science Research Program [2016M3A7B4911115] and Project for Research and Development of Police science and Technology [PA-C000001]) and Samsung Research Funding Center of Samsung Electronics [SRFC-IT1601-05].
Copyright © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.