0
$\begingroup$

The conventional wisdom for convolution neural networks (CNNs) is to make the batch size a power of 2 because of hardware utilization/optimizations done in the convolution layers.

A similar logic applies to the Fast Fourier Transform (FFT). However, the FFT is actually best used when it is a size of a power of primes (e.g. $2^M3^N5^P$ where $M,N,P$ are positive integers.) I was wondering if this also applies to CNNs.

$\endgroup$
  • 1
    $\begingroup$ This depends entirely on the implementation details. Have you had a look at the internals of whatever you care about or simply tried it? $\endgroup$ – Juho Jan 2 '19 at 17:37
  • $\begingroup$ I am using cudnn which doesnt have source available that I am aware of. Also, I have tried to measure performance it seems powers of primes is just as good as power of 2. However, this could be by sure luck or some other inefficiency in my training regime. $\endgroup$ – Isaac Gerg Jan 2 '19 at 17:45
  • $\begingroup$ There are plenty of implementations of both CNNs and FFTs that you can peruse; many of them are open source. $\endgroup$ – D.W. Feb 11 '19 at 0:03
0
$\begingroup$

No. It doesn't apply to CNNs. CNNs are doing the same operation on each item in the batch; it's much simpler than what's happening in a FFT (which involves interaction between the elements of the array; you can't compute a FFT by just separately doing something to each element of the array in parallel).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ How do you know this? $\endgroup$ – Isaac Gerg Feb 10 '19 at 19:48
  • $\begingroup$ @IsaacGerg, I explained my reasoning in the answer. If that's not sufficient, I suggest you try implementing it and see what happens! $\endgroup$ – D.W. Feb 10 '19 at 20:14
  • $\begingroup$ Its possible that the twiddle factors are being reused between kernels. The convolution can be implemented in frequency domain in )(nlogn) where if you compute the convolution directly its O(n^2) $\endgroup$ – Isaac Gerg Feb 10 '19 at 20:43
  • 1
    $\begingroup$ @IsaacGerg, great, but I don't see how that has any relevance to my answer. $\endgroup$ – D.W. Feb 11 '19 at 0:03
  • $\begingroup$ Your answer states that the power rules for FFT sizing don't apply to CNNs because FFT's dont' apply to CNNs. You make this claim without fact. I am providing evidence that this is not a fact. $\endgroup$ – Isaac Gerg Feb 11 '19 at 0:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.