Assumption of a generation of the dataset by a probability distribution

Question

Consider the following paragraph from the deeplearningbook

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other, and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example. The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{data}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.

Bolded area is difficult for me to comprehend. Here I have the following issues in interpreting.

1) How probability distribution is generating a dataset?

2) Are the generation process and probability distribution the same?

3) What is the sample space and random experiment for the underlying probability distribution?

Answer 1

The main takeaway from this paragraph is that the authors assume that the training data and test data are sampled i.i.d. from the same distribution, because then if we are able to estimate some statistic of the distribution we sample the training data from, we will be able to do the same on the test data.

As for your specific points:

1) How probability distribution is generating a dataset?

We sample from a large set according to some distribution. The precise method depends on the problem.

2) Are the generation process and probability distribution the same?

No, the generation process provides a sample from a distribution. A probability distribution is an abstract mathematical object, while a generation process is a specific algorithm that provides a dataset. In a way, the distribution provides a probabilistic description of what the generation process should return.

3) What is the sample space and random experiment for the underlying probability distribution?

The sample space depends on the problem in question. I'm not sure what you mean by 'random experiment', but the definition of the random distribution is problem-specific.