Initialization of embedding space value for VQ-VAE

Question

I am trying to study a research paper about VQ-VAE: Neural Discrete Representation Learning, Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, NeurIPS 2017.

I have difficulty to understand the value of embedding vectors $e_j$ in this part (page 3 from the research paper):

The posterior categorical distribution $q(z|x)$ probabilities are defined as one-hot as follows:

$q(z=k|x)=\begin{cases}1, & \text{for k = }argmin_j ||z_e(x) - e_j||_2\\ 0, & \text{otherwise} \\ \end{cases}$

Here we define latent embedding space $e\in R^{K×D}$ and there are K embedding vectors $e_i \in R^D$.

My question is how to initialize latent embedding space and its value. Is it random initialization? Because we need to subtract our $z_e(x)$ with $e_j$ so embedding space $e$ must have some value.

Answer 1

In initializing codebook embeddings, a common practice is to employ random initialization due to its simplicity and effectiveness in various contexts. A widely adopted method among practitioners is to sample values uniformly from a specified range. Specifically, the range is typically chosen to be between $-1/K$​ and $1/K$​, where K represents the total number of embeddings in the codebook.

Here is an example of an implementation using this initialization: