2
$\begingroup$

One can define the semantics of one definite word using the references to real world entities, relationships with the other words and other concepts and represent all this knowledge about this one word using logical symbolic expressions. And then one can encode all this set of symbolic expressions into vector of real numbers. This is word embedding that is used in natural language processing, distributional semantics of the word in opposite of the formal semantics of the word.

One can consider function of software program (e.g. functional program or any other program). One can encode this program in the multiple vectors and matrices of real numbers that are used for the definition of neural network.

One can consider symbolic meta-knowledge and encode them into vectors or neural networks as well.

The decoding process can be more tricky. There is more or less elaborate decoding of neural network - e.g. see Google queries "logical program extraction from neural networks" or "symbolic rule extraction from neural networks". But I have not seen the work about extraction of more or less static knowledge base from the word-embedding-vector.

So - I have two questions regarding this matter:

  • Is there symbolic knowledge extraction from the word-embedding vectors - some kind of decoding algorithm from vector of real numbers to the set of logical formulas?
  • Is there general theory of mentioned encoding algorithms? The usual approach is to train neural networks and to arrive at the encoded form using non-symbolic methods, non-algorithmic methods, implicit way. I have heard about embedding of symbolic knowledge in neural networks to speed-up training, but such kind of work is scarce. But what about general encoding algorithms?

There are discrete, natural Goedel numbers (encoding algorithms) that can be assigned to any theorem of first order logic. But what about such Goedel numbers for the sets of formulas or for some computational program (as a set of commands)? Can we enumerate all such sets using natural numbers only or maybe the real numbers are needed instead naturally. Or maybe even set of real numbers are required for encoding the set of symbolic formulas or program statements? Is there such research work which I can develop further? If no, then what ideas can be mentioned for such encoding/decoding schemes?

Such encoding-decoding algorithms can be related to biological computing and ultimately they can lead to the explanation of brain activity.

Improve this question
$\endgroup$
4
  • $\begingroup$ Essentially - I am seeking methods how to convert compositional semantics into distributional semantics of natural language and back. $\endgroup$
    – TomR
    Commented Oct 3, 2018 at 6:45
  • $\begingroup$ Can you share any specific experience that you know about encoding of knowledge about specific real numbers? For example, how does Wolfram treat the $\pi$, for example when it appears in input such as wolframalpha.com/input/…? Can knowledge about $\pi$ be trained into a neural network and how? $\endgroup$
    – John L.
    Commented Oct 4, 2018 at 23:37
  • $\begingroup$ Representation of the π is interesting question, but it is not highly relevant here. I am more interested in legal or similar non-mathematical domains. $\endgroup$
    – TomR
    Commented Oct 5, 2018 at 6:06
  • $\begingroup$ The question of extraction of symbolic knowledge from word embedding looks intriguing and stated well enough while other questions included in this post seem a bit vague. For the first question there is a significant body of research on adding constraints and structure to word embeddings that seems related to this post. Perhaps splitting the post and giving the question a more concise formulation should improve the chances of getting a useful answer. $\endgroup$
    – Dima Chubarov
    Commented Oct 7, 2018 at 5:56

1 Answer 1

Reset to default
2
$\begingroup$

No, probably not. I think you're expecting too much from the current state of the art in word embeddings. Word embeddings don't magically capture all semantic knowledge. They don't reflect perfect understanding of the language. Instead, they're just useful mappings where similar words often have similar embeddings. Moreover, the way word embeddings are constructed has nothing to do with logical formulas.

I don't think you're going to find that word embeddings solve the problem of converting from natural language to formulas.

I don't know what would count as a general theory of encoding algorithms for you. There are certainly multiple papers that propose different methods of constructing different word embeddings; you could read those to understand the state of the art.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Neural network training (as I mentioned) also has nothing to do with the symbolic formulas, yet there are papers that extract logic programs from trained networks, or even natural language grammar from the networks that do question answering or machine translation. So - similar approach for word embedding should be possible. $\endgroup$
    – TomR
    Commented Oct 3, 2018 at 8:14
  • $\begingroup$ @TomR, if you think it should be possible, that's an empirical claim you can test. So, I suggest you go test it! Prove me wrong -- prove it is possible, by demonstrating how to do it, and showing some concrete experiments that demonstrate it is possible and measure how well it works. That's an experiment you'll need to do yourself -- it's something you can only settle definitively with an experiment and data, not a question here. My answer is mere opinion, and data trumps opinion any day. $\endgroup$
    – D.W. ♦
    Commented Oct 3, 2018 at 17:33
  • $\begingroup$ One developing approach cl.cam.ac.uk/~sc609/talks/groningen13.pdf $\endgroup$
    – TomR
    Commented Oct 4, 2018 at 5:32
  • $\begingroup$ And this one coling2016.anlp.jp/doc/tutorial/slides/T1/… $\endgroup$
    – TomR
    Commented Oct 4, 2018 at 5:56
  • $\begingroup$ Well - the papers and presentations mentioned in my two comments are about the distributional meaning of the phrases and sentences that can be calculated from the distributional meaning of the words. The distributional meaning of the words itself is not explained/calculated in explicit manner (as I suggested) in this DisCoCat framework. $\endgroup$
    – TomR
    Commented Oct 4, 2018 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.