I'm not 100% sure this is the right stackexchange, please feel free to redirect me to another one.
I know that two 3x3 convolution layers can be equivalent to one 5x5 convolution layer.
I also know that in many cases (maybe all of them), the training of both options is equivalent (in terms of result, not speed or optimisation)
Suppose I have a dataset on which I KNOW that no information whatsoever can be derived from any 3x3 window, but it can be from a 5x5 window.
If I wanted to, I could train a network with a 5x5 convolution layer, then manually transform it into two 3x3 convolution layers.
My question is: could I train a network directly with two 3x3 convolution layers ?
My reasoning: What could the first convolution learn apart from overfitting the training dataset? Supposedly nothing. So, the only layer that could some generalization of my data is the second one. But then, my problem sums up to comparing ONE 3x3 convolution versus ONE 5x5 convolution.
In the general case, the 5x5 convolution would perform better. But with my restrictions on the dataset, maybe it becomes equivalent to the 3x3 convolution? I which case I could train 3x3 convolution.
Thank you !