1
$\begingroup$

I am looking to implement a neural network on hardware using Verilog. I have completed and tested with floating point representation and a 20 bit fixed point representation. I want to further reduce the memory utilisation by using dynamic fixed point. I want to identify the less important weights, and assign them a smaller bit-width, while keeping the more important weights with a higher bit-length. How can I identify the more/less important weights? What I can think of is, reducing bit-width for all the small weights [-0.001, 0.001]. But isn't that like pruning? Thanks!

$\endgroup$
  • 1
    $\begingroup$ An accuracy of 20 bits sounds like too much. You can make do with dramatically less – as few as one bit. $\endgroup$ – Yuval Filmus Jun 28 '17 at 18:24
1
$\begingroup$

Start by reading the research literature on this topic. There's a large number of published papers that discuss how to quantize the weights or intermediate values used in deep learning. For instance, here are a few papers:

Start by reading them, and do a literature search to identify other relevant papers (e.g., read their related work section and find other earlier papers that are relevant; find papers that cite them using Google Scholar and read them; repeat).

You might also look at the Ristretto tool and this project.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.