I am trying to study a research paper about VQ-VAE: Neural Discrete Representation Learning, Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, NeurIPS 2017.

I have difficulty to understand the value of embedding vectors $e_j$ in this part (page 3 from the research paper):

The posterior categorical distribution $q(z|x)$ probabilities are defined as one-hot as follows:

$q(z=k|x)=\begin{cases}1, & \text{for k = }argmin_j ||z_e(x) - e_j||_2\\ 0, & \text{otherwise} \\ \end{cases}$

Here we define latent embedding space $e\in R^{K×D}$ and there are K embedding vectors $e_i \in R^D$.

My question is how to initialize latent embedding space and its value. Is it random initialization? Because we need to subtract our $z_e(x)$ with $e_j$ so embedding space $e$ must have some value.

  • $\begingroup$ Have you looked at the code of any VQ-VAE implementations? $\endgroup$
    – D.W.
    Commented May 10, 2023 at 17:00

1 Answer 1


In initializing codebook embeddings, a common practice is to employ random initialization due to its simplicity and effectiveness in various contexts. A widely adopted method among practitioners is to sample values uniformly from a specified range. Specifically, the range is typically chosen to be between $-1/K$​ and $1/K$​, where K represents the total number of embeddings in the codebook.

Here is an example of an implementation using this initialization:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.