Are fully connected layers required for convnets?

Question

I have a reinforcement learning model I'm trying to create for a grid based game. One of the features of the game is that the game board can get bigger mid game, although I know this is a problem that machine learning isn't very good at. However, one idea that I had was to train a convolutional neural net model that doesn't have a fully connected layer at the end, so that the input and output size of the net can vary together, and the model will still work because it won't need any additional weights to produce an output. In addition, adding a fully connected layer or two at the end would be a very large amount of extra weights to train, since the action space of the problem is very large (15232 discrete actions if I remove the expanding grid portion of the game). If my calculations are correct, a single fully connected layer would become 464 million more weights. Which seems like a lot compared to my existing 223,350 trainable parameters.

Here is one iteration of this network type I've tested: For this model, the observation space is 92 layers deep, and 28 wide by 16 tall. And the action space is 34 layers deep, and 28 wide by 16 tall. I'm not getting amazing results from this network (although better than random choice), which could be due to a multitude of reasons. However, I don't know if this is bad idea as I'm having trouble reasoning about this design generally.

Is a fully connected layer at the end necessary for the network to learn global ideas about the observation space?

Answer 1

There are no rules. You can do whatever you like. The ultimate test is how well it works; you'll have to try it and see.

But the short answer is: no, fully connected layers are not required. You can build a network that is purely convolutional, with no fully connected layers. It's hard to know how well it will work, without trying it.

Answer 2

What you are thinking of is a Fully Convolutional Network (FCN). It's already discussed for a couple of years, and you might want to check out this famous piece of paper.

So the architecture you are designing conceptually has no problem. With less parameters, you may achieve faster learning and better performance (or not, maybe). Is a FCN better than a traditional ConvNet with fully connected layers? I don't know, I think there's no solid answer to that.