I've been reading about Google's TensorFlow, and the way it represents calculations with graphs that are then executed by an engine. While the concept is interesting, I would like to understand why they made that choice instead of the arguably simpler imperative programming found e.g in PyTorch.
The TensorFlow documentation lists several advantages:
- Parallelism. By using explicit edges to represent dependencies between operations, it is easy for the system to identify operations that can execute in parallel.
- Distributed execution. By using explicit edges to represent the values that flow between operations, it is possible for TensorFlow to partition your program across multiple devices (CPUs, GPUs, and TPUs) attached to different machines. TensorFlow inserts the necessary communication and coordination between devices.
- Compilation. TensorFlow's XLA compiler can use the information in your dataflow graph to generate faster code, for example, by fusing together adjacent operations.
- Portability. The dataflow graph is a language-independent representation of the code in your model. You can build a dataflow graph in Python, store it in a SavedModel, and restore it in a C++ program for low-latency inference.
Portability makes sense, I'm more interested in the performance aspect. It seems that doing parallel and distributed computations doesn't require graph execution, since PyTorch does it too. I assumed that creating a graph enabled the engine to do optimizations that could not be done with an imperative program.
But then I read about TensorFlow's upcoming eager mode, which basically does away with the graph API and lets us use imperative programming like in other libraries. The documentation for eager mode suggests that it approaches and could reach the performance of graph mode:
For compute-heavy models, such as ResNet50 training on a GPU, eager execution performance is comparable to graph execution. But this gap grows larger for models with less computation and there is work to be done for optimizing hot code paths for models with lots of small operations.
One thing I didn't find is distributed training with eager mode, but as mentioned earlier, other imperative libraries seem to offer that despite not using graphs.
I'm not sure what to make of this. Does graph execution have a performance advantage over imperative programming after all?