Backward mapping with bilinear sampler

I have some experiences with Convolutional Neural Networks before. I have a question regarding the Bilinear Sampler used in "Unsupervised Monocular Depth Estimation With Left-Right Consistency" (the link to it is provided here: https://arxiv.org/pdf/1609.03677.pdf and https://www.youtube.com/watch?v=jI1Qf7zMeIs)

I understand that the author had made the convolutional neural network predict the disparity map of the left image, then uses the disparity map to transform the right image back into the predicted left image.

However, I don't understand how the transformation is done (shown in 5:35 in the youtube video, and "Our network generates the predicted image with backward mapping using a bilinear sampler, resulting in a fully differentiable image formation model").

I understand that the system needs to be fully differentiable so that backpropagation can be used, and I understand that Spatial Transformer Networks can map images from a transformed grid, but I cannot understand how the bilinear sampler in the spatial transformer network is used to transform the right view image into a left view image given the left disparity map. In particular, in spatial transformer networks, the sampling stage requires a set of sampling points (which is a 2x3 matrix) and an image so that it can sample a new image from the old one. However, in the depth estimation paper, I can't understand how they extract the set of sampling points (the 2x3 matrix) from the disparity map. How does that work?

• Cross-posted: cs.stackexchange.com/q/91879/755, stackoverflow.com/q/50323130/781723. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. – D.W. May 14 '18 at 20:44
• did you find your answer? so please share – Aqib Mumtaz May 19 '18 at 12:05
• Actually yes, after receiving information about website rules instead of what i was looking for, i looked at the source codes. The author first creates a meshgrid (a uniform mesh grid with each pixel as a cell). He then shifts the grid to the left (supposed the input image is a right image) according to the value in the disparity map. The final image is the backward-sampled image of the right image using the transformed meshgrid. The floating point indexes are solved with bilinear interpolation (as the name bilinear sampler implies) – Derpson May 24 '18 at 18:15