Reporting environment details for reproducibility of results in machine learning

Question

I'm a computer science master's student, researching deep learning models.

At my university we have a cluster of computer nodes with GPUs for training machine learning models.

I recently developed a model for my thesis, and I want to guarantee its out-of-the-box reproducibility not just within our organization.

I report the following environment details programmatically for each job:

• CPU architecture (e.g., x86_64)
• Linux kernel version (e.g., 5.14.0-427.18.1.el9_4.x86_64)
• glibc version (e.g., 2.34)
• GPU model (e.g., NVIDIA GeForce RTX 3090)
• NVIDIA driver version (e.g., 555.42.02)
• CUDA version (e.g., 12.1)
• cuDNN version (e.g., 8902)
• Python version (e.g., 3.11.9)
• pip version (e.g., 24.0)
• NumPy version (e.g., 2.0.0)
• PyTorch version (e.g., 2.3.1+cu121)
• PyTorch Lightning version (e.g., 2.3.0)
• Package versions (output of pip list --format=freeze)

However, we have two sets of nodes that, given that all the above are the same (and the same data, hyperparameters, code revision and random seed are used, as well as deterministic=True is set on the trainer), in one of them my results are reproduced, while in the other they are not, when performing full training from the beginning. So it seems that even such detailed report is incomplete.

I don't know what else is important to report about the environment in order to guarantee reproducibility. I don't know of any differences in compiler settings, and also I'm not sure what other hardware differences may impact reproducibility, but I would like my report to be as specific as possible, because from my experience during my master's program, reproducing other people's works is difficult, and I don't want other people to experience the same thing with my model. I simply don't agree with the current state of nonreproducibile research, and I opt to change that and help the machine learning community set new standards for reproducible research by reporting the environment details as best as I can.

A few things come to mind regarding the differences between the two sets of nodes:

• CPU models may differ between nodes. Does it have any impact on reproducibility, even though they use the same architecture?
• GPU models may ship in different packages of cooling and overclocking brands. Does it have any impact on reproducibility, even though they consist of the same chipset?
• As far as I can see, the two sets of nodes differ in the amount of both CPU cores and RAM. Do these have any impact on reproducibility?

I am not sure if I can get the full descriptions of hardware of every node in the cluster, but what other properties should I report about in case I can?

Unfortunately, to the best of my knowledge, the current literature on reproducibility in machine learning does not have a comprehensive list of all environment details that may impact reproducibility. Some useful resources include:

• ML Reproducibility Tools and Best Practices
• ML Reproducibility Challenge -> Resources
• Reproducibility in Machine Learning-Driven Research
• Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers
  • "In general, it is questionable to what extent reproducibility can be ensured out-of-the-box."

Answer 1

A standard way to address many of the software issues with reproducibility is to provide a Docker image (or VM image) containing all of the software pre-installed. This ensures that everyone is working with the same version of software you used.

If you want deterministic reproducible computation, this is more challenging. Some steps towards that goal including setting fixed seeds for all random number generators (including seeds for the prngs used internally by language-level standard libraries), use only single-threaded computation (no parallelism, no multi-threading), avoid making network connections, and make sure none of your software uses/produces timestamps anywhere. (There are various methods to address the issue of timestamps, e.g., setting the clock to a fixed time, libfaketime.)

There still may be other implementation-level artifacts, either in software or hardware, that could cause non-reproducibility. Those implementation-level details are likely beyond the scope of this site.