People have recently started talking about "differentiable programming". I have listened to some people talk about this at a philosophical level, but I don't see the practical difference between this and just existing Autograd libraries like PyTorch.

I understand that (at least according to some interpretation of "differentiable programming"), what makes it different is that it is an integrated part of a programming language. For example, in Julia, backprop/autograd is apparently a fundamental element of the language.

However, it seems to me that there isn't such a big difference between calling backprop as a library call to PyTorch, vs having it more integrated in the language. The library call to PyTorch is already very simple. So how is "differentiable programming" actually different in a meaningful way from just calling a library like PyTorch or TensorFlow?


You are essentially right: "differentiable programming" and automatic differentiation libraries (e.g. PyTorch and TensorFlow) are morally the same thing. However, when people talk about differentiable programming they are more likely to be considering the problem from a theoretical lens, working e.g. in programming language theory or type theory. On the other hand, PyTorch and TensorFlow or more practical software engineering tools rather than interested in a formal or theoretical study of what automatic differentiation is.

Put another way: programming languages theorists are interested in differentiable programming because they are interested in understanding the fundamental principles behind how PyTorch and TensorFlow work.

For example, here's a prototypical paper on differential programming from ICFP 2018:

In the paper, Elliot proposes a foundational theory of differentiable programs, how they can be written and composed, and how differentiation and techniques like back-propagation can be derived as part of this theory. That may or may not be relevant to engineers working on PyTorch and TensorFlow, who just want to "get the job done" and write a usable tool that works well in practice.

  • $\begingroup$ This makes sense to me, except people excited about Julia seem to talk about differentiable programming as a quite important next step for practical engineers $\endgroup$
    – user56834
    Dec 5 '21 at 9:23
  • 1
    $\begingroup$ Theoretical advances are always quite important next steps for practical engineers, even though engineers do not like to admit it. $\endgroup$ Jan 4 at 6:57
  • $\begingroup$ @AndrejBauer Agreed & well-said. $\endgroup$
    – 6005
    Jan 4 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.