# How to represent symbolic knowledge using real numbers - theory about neural networks and natural/analog computing?

One can define the semantics of one definite word using the references to real world entities, relationships with the other words and other concepts and represent all this knowledge about this one word using logical symbolic expressions. And then one can encode all this set of symbolic expressions into vector of real numbers. This is word embedding that is used in natural language processing, distributional semantics of the word in opposite of the formal semantics of the word.

One can consider function of software program (e.g. functional program or any other program). One can encode this program in the multiple vectors and matrices of real numbers that are used for the definition of neural network.

One can consider symbolic meta-knowledge and encode them into vectors or neural networks as well.

The decoding process can be more tricky. There is more or less elaborate decoding of neural network - e.g. see Google queries "logical program extraction from neural networks" or "symbolic rule extraction from neural networks". But I have not seen the work about extraction of more or less static knowledge base from the word-embedding-vector.

So - I have two questions regarding this matter:

• Is there symbolic knowledge extraction from the word-embedding vectors - some kind of decoding algorithm from vector of real numbers to the set of logical formulas?
• Is there general theory of mentioned encoding algorithms? The usual approach is to train neural networks and to arrive at the encoded form using non-symbolic methods, non-algorithmic methods, implicit way. I have heard about embedding of symbolic knowledge in neural networks to speed-up training, but such kind of work is scarce. But what about general encoding algorithms?

There are discrete, natural Goedel numbers (encoding algorithms) that can be assigned to any theorem of first order logic. But what about such Goedel numbers for the sets of formulas or for some computational program (as a set of commands)? Can we enumerate all such sets using natural numbers only or maybe the real numbers are needed instead naturally. Or maybe even set of real numbers are required for encoding the set of symbolic formulas or program statements? Is there such research work which I can develop further? If no, then what ideas can be mentioned for such encoding/decoding schemes?

Such encoding-decoding algorithms can be related to biological computing and ultimately they can lead to the explanation of brain activity.

• Essentially - I am seeking methods how to convert compositional semantics into distributional semantics of natural language and back. – TomR Oct 3 '18 at 6:45
• Can you share any specific experience that you know about encoding of knowledge about specific real numbers? For example, how does Wolfram treat the $\pi$, for example when it appears in input such as wolframalpha.com/input/…? Can knowledge about $\pi$ be trained into a neural network and how? – John L. Oct 4 '18 at 23:37
• Representation of the π is interesting question, but it is not highly relevant here. I am more interested in legal or similar non-mathematical domains. – TomR Oct 5 '18 at 6:06
• The question of extraction of symbolic knowledge from word embedding looks intriguing and stated well enough while other questions included in this post seem a bit vague. For the first question there is a significant body of research on adding constraints and structure to word embeddings that seems related to this post. Perhaps splitting the post and giving the question a more concise formulation should improve the chances of getting a useful answer. – Dmitri Chubarov Oct 7 '18 at 5:56