Explain this Neural Network used for Knowledge Graph Embedding

Question

I am interested in better understanding the neural network used in the paper LogicENN A Neural Based Knowledge Graphs Embedding Model with Logical Rules. To my knowledge this is the most advanced and complex knowledge graph embedding.

The neural network is described as follows:

Entity pairs, $[h,t]$ , are input and relations are output.

The score of a given triple $(h, r, t)$ is $f^r_{h,t}$ and is given by:

The beta vector are the embedding of relations.

The phi vector is the feature mapping of the hidden layer of the network which is shared between all relations.

This neural network works with a single hidden layer.

The first problem is: when I look at the picture, I think this neural network should have multiple hidden layers but in the text it's written "we use a single hidden layer".

The second problem is: when I look at the picture, I think that there is no direct relation between the outputs and the hidden layers but in text it's written "the feature mapping of the hidden layer of the network which is shared between all relations".

Please, explain this neural network. Any help is appreciated.

Answer 1

Well basically, without loss of generality, they use a single hidden layer for the neural network to show theoretical capabilities of those networks. Practically, sometimes it is better to use more hidden layers and sometimes, on the other hand, adding more hidden layers only adds complexity and it is more time consuming for a model to be trained. if your data is linearly separable (which you often know by the time you begin coding a NN) then you don't need any hidden layers at all. Of course, you don't need an NN to resolve your data either, but it will still do the job.

In their experiment, they are using 3 hidden layers. People usually come up with the best number of hidden layers by experiment. There is no magic formula for selecting the optimum number of hidden neurons. However, some thumb rules are available for calculating the number of hidden neurons. A rough approximation can be obtained by the geometric pyramid rule proposed by Masters (1993). For a three layer network with n input and m output neurons, the hidden layer would have $ \sqrt{n \times m} $ neurons.