I have some experiences with Convolutional Neural Networks before. I have a question regarding the Bilinear Sampler used in "Unsupervised Monocular Depth Estimation With Left-Right Consistency" (the link to it is provided here: https://arxiv.org/pdf/1609.03677.pdf and https://www.youtube.com/watch?v=jI1Qf7zMeIs)
I understand that the author had made the convolutional neural network predict the disparity map of the left image, then uses the disparity map to transform the right image back into the predicted left image.
However, I don't understand how the transformation is done (shown in 5:35 in the youtube video, and "Our network generates the predicted image with backward mapping using a bilinear sampler, resulting in a fully differentiable image formation model").
I understand that the system needs to be fully differentiable so that backpropagation can be used, and I understand that Spatial Transformer Networks can map images from a transformed grid, but I cannot understand how the bilinear sampler in the spatial transformer network is used to transform the right view image into a left view image given the left disparity map. In particular, in spatial transformer networks, the sampling stage requires a set of sampling points (which is a 2x3 matrix) and an image so that it can sample a new image from the old one. However, in the depth estimation paper, I can't understand how they extract the set of sampling points (the 2x3 matrix) from the disparity map. How does that work?