I'm fairly new to deep learning, so if terminology makes no sense, please let me know so I can clarify what I mean.

We're working with a neural network for applying classes to inputs. That is, each output unit represents a target class, and its activation denotes the amount of relevance for a given input. The output layer is fully connected to the previous one.

This NN is used for prioritizing and ordering search results, so learning happens online (based on user choices), and the function/mapping we're approximating may shift over time.

Since the domain of our search results is open-ended, the set of target classes may dynamically change over time. Therefore, a fixed set of output units representing output classes doesn't work for us.

Is there any best practice (or at least some research) on adding/removing output layer nodes on already-trained neural networks? To our surprise, we couldn't find any research papers or other literature discussing this topic.

For what it's worth, our current best idea is to deal with changes in the target set the following way:

  • For classes vanishing from the target set, remove their corresponding output layer units.
  • For new classes appearing in the target set, create new output layer units and fully connect them with weight zero (or alternatively, average of existing other edges, or randomized).

With such a solution, we'd expect the NN to remain stable on existing classes, and potentially giving a "disadvantage" to new ones, but this seems acceptable. Are there any other bad side effects that might emerge?

  • $\begingroup$ "For new classes appearing in the target set, create new output layer units and fully connect them with weight zero" I can already tell you that this won't work. Hint: what is the derivative w.r.t the hidden layer if all weights are zero? $\endgroup$ – orlp Jul 2 '18 at 13:40
  • $\begingroup$ @orlp Ah, I see. So we should probably use either an average value or random ones. $\endgroup$ – Cedric Reichenbach Jul 2 '18 at 13:58
  • $\begingroup$ Alternative strategy: keep your neural network topology the same, with $N$ output classes, where $N$ is the maximum number of output classes you can support. And any unused output classes simply group into "Other". $\endgroup$ – orlp Jul 2 '18 at 14:15
  • $\begingroup$ That's what we've been doing so far. We build a network with millions of output units and basically assign them "meaning" along the way. But unless we re-use/re-assign old nodes, we'll eventually hit the limit. And re-assigning would likely cause issues with fairness and consistency. $\endgroup$ – Cedric Reichenbach Jul 2 '18 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.