I'm fairly new to deep learning, so if terminology makes no sense, please let me know so I can clarify what I mean.
We're working with a neural network for applying classes to inputs. That is, each output unit represents a target class, and its activation denotes the amount of relevance for a given input. The output layer is fully connected to the previous one.
This NN is used for prioritizing and ordering search results, so learning happens online (based on user choices), and the function/mapping we're approximating may shift over time.
Since the domain of our search results is open-ended, the set of target classes may dynamically change over time. Therefore, a fixed set of output units representing output classes doesn't work for us.
Is there any best practice (or at least some research) on adding/removing output layer nodes on already-trained neural networks? To our surprise, we couldn't find any research papers or other literature discussing this topic.
For what it's worth, our current best idea is to deal with changes in the target set the following way:
- For classes vanishing from the target set, remove their corresponding output layer units.
- For new classes appearing in the target set, create new output layer units and fully connect them
with weight zero(or alternatively, average of existing other edges, or randomized).
With such a solution, we'd expect the NN to remain stable on existing classes, and potentially giving a "disadvantage" to new ones, but this seems acceptable. Are there any other bad side effects that might emerge?