I am looking to implement a neural network on hardware using Verilog. I have completed and tested with floating point representation and a 20 bit fixed point representation. I want to further reduce the memory utilisation by using dynamic fixed point. I want to identify the less important weights, and assign them a smaller bit-width, while keeping the more important weights with a higher bit-length. How can I identify the more/less important weights? What I can think of is, reducing bit-width for all the small weights [-0.001, 0.001]. But isn't that like pruning? Thanks!
$\begingroup$
$\endgroup$
-
1$\begingroup$ An accuracy of 20 bits sounds like too much. You can make do with dramatically less – as few as one bit. $\endgroup$ – Yuval Filmus Jun 28 '17 at 18:24
$\begingroup$
$\endgroup$
Start by reading the research literature on this topic. There's a large number of published papers that discuss how to quantize the weights or intermediate values used in deep learning. For instance, here are a few papers:
- https://arxiv.org/abs/1412.7024, https://arxiv.org/abs/1605.06402, https://arxiv.org/abs/1612.01064, https://arxiv.org/abs/1701.08978
Start by reading them, and do a literature search to identify other relevant papers (e.g., read their related work section and find other earlier papers that are relevant; find papers that cite them using Google Scholar and read them; repeat).
You might also look at the Ristretto tool and this project.
