I’m curious about an embedding technique where every possible “tokenization” of a text gets an embedding - not just individuals words, but every single 2-gram, 3-gram, and n-gram.

Does this exist?

Or is it the bag of words approach?

What I would really like is to explore in real time how the addition of a token changes a range of suggested likely tokens and their perplexity scores.

For example, after creating the embeddings, I can write a “dictionary” where for every n-gram up to some manageable length N, the M most “predicted” tokens (of any length) are written in an adjacent column, according to a clustering metric like Euclidean distance, cosin similarity, or a different one.

You could then observe how the context of predicted tokens and their likelihood “updates” based on the addition of any other token. You can also return a distribution if predicted and anti-predicted tokens.

So, (dog, cat) may return (animal, pet, house, Brazil) - animal has the strongest association (including because of it being a more common word, in general), pet because it’s commonly associated, house is weakly or neutrally associated, and Brazil is anti-associated, there is no reason to expect it above randomness.

When you update it to (dog, cat, bird), the co-occurrence token scores update to, (pet, housepet, animal, pet store, pet food, rug, clang, symphony).

This tells us information about an entity, (bird | dog, cat) - while token (bird) may independently be (fly, sky, jungle, canary, cave, egg, waterfall, heart, gym, Christmas sale, firefighter, salvia), (bird, dog, cat) emphasizes bird as “pet”.

I know there is some stuff here related to LSA (latent semantic analysis), like essentially, pet = cat + dog + bird - fly - bone - meow (the equality can be replaced by a similarity percentage - the cosin similarity is maybe 67%), but I’m curious to know the straightforward algorithm to set this up, using a good library suited to the task, perhaps. Gensim? Or native algorithms, in Python?

My current approach would be:

Assume N texts, each of some number of tokens, T_i.

Let’s just use a Spacy tokenizer to keep this simple.

For each text, tokenize the text in a Spacy doc object.

From 0 to T_i (the length of the text), split the text on context windows of that length, and add it to a table. So for 1, you have a column in a matrix:

1: a, abs, and, Andromeda, …, zebra, Zeus 2: “a book”, “a chair”, “after all”, …, “back home”, “cheese, the”, etc. 10: “This is not very fun given that I am tired.”, “hope so. But on the other hand, there’s possibly a”

Combine all multi-word expressions into a single set.

Again, count co-occurrences for every context window length. For example:

“albino” and “chimichangas so”: total number of times co-occurred adjacently (context window of 0): 1 time

co-occurrence in context window of 20 words in between: 4

And so on.

If it’s too computationally expensive, everything can be reduced to 2-grams, perhaps. Or, 2-grams can be generalized, and then you can see how performance changes as you increase N to 3, 4, 5, etc.

I think the standard way is to order the tokens alphabetically, and each token is an axis on a vector space. The value for each vector is the co-occurrence count with that other token. In some sense, you have one axis for every word in the texts. Maybe you could simplify this by assuming the context window is just the number of words, in other words, always adjacent.

So, the vector of all words, call V. Call the co-occurrence space V^N, where N is V is the order of V (the number of words). This is our domain. Define f: V^N -> N as maybe a normalized, relativized co-occurrence number, rather than a total tally. Like, translated by the average, multiplied to make a standard deviation of 1, or something.

Also, V can appear as a binary vector, so coordinate (dog, cat) is actually (1,1,0,0,0,0,0), a selector which selects the positions of dog and cat and dis-selects the presence of a different word. This makes it easy to calculate a context window: simply sum the terms of the vector.

Can this be calculated with some gigantic tensor computations, maybe?


  1. Choose max N
  2. Map the text to ordered pairs of n-grams and their counts - take all 1-grams and accumulate the count of identical 1-grams. Do the same up to N.
  3. Create word vector V, a binary vector where the values index an alphabetical list of tokens (/1-grams).
  4. Map 2-grams to a corresponding vector, like (1,1,0,0,0,0). And so on, for N.

We have an N-dimensional finite space where each point is mapped to a natural number.

This is effectively a vector space - each point is assigned a “magnitude”, like a differential equation. Right?

Perhaps from there it’s a few simple equations to be able to query the data with any tokens and see what tokens are considered “nearest”?

If we think of the associated magnitude as the norm of the vector at that point, maybe we need a change of basis? Is that the linear algebra this is heading towards?

Thank you

  • $\begingroup$ This is a very long post. Please ask only one question per post. I count 8 different sentences that end with "?". If you have multiple questions, please ask them separately. What exactly is the problem you are trying to solve? "Does this exist?" might not be the best question; by describing it, you have essentially proven that it does exist, but I suspect that's not what you mean to ask. Is there a problem you're trying to solve? This site works best when you're trying to solve some actual, concrete problem. $\endgroup$
    – D.W.
    Jun 27, 2023 at 7:53


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.