# Registering 3D-NIR image to thermal image and vice versa

In the past I have thought a bit about how to register a NIR-image and a thermal image and noticed that this is not trivial - one statement was that if I had the depth information for each pixel, the task would be much easier.

Now consider having a 3D-NIR camera (e.g. asus xtion) and a thermal camera and I want to map the thermal information to the depth map (the 3d cloud) or vice versa and the NIR information to the thermal information (and vice versa).

I thought I could do it simply like this:

1. Conduct a stereo camera calibration for the two cameras (-> R1,R2,T)
2. Use R1, R2 and T to transform the 3D points from 3D-NIR to the thermal camera's coordinate system
3. Use the camera matrix of the thermal camera to project the 3D points to its image plane
4. Now I know where the 3D points "fall" in the thermal image, which gives me a depth value (and intensity value, because the depth map and NIR-image are aligned) for each pixel in the thermal image.

Which to me sounds similar to the procedure described in this answer by D.W.. However, I can't find any article describing this method. Always there seems to be some form of feature matching step, e.g. here.

Also, thinking about this, the above solution can not really work, I believe. Consider the case where the two cameras look at an object from different sides. The 3D camera will calculate the depth for a point p1 in the world. When projecting p1 to the thermal camera's image plane it will fall in pixel x. However, since the thermal camera looked at another side of the object, another point in the world p2 actually formed pixel x when the thermal camera took the image - and thats what the measured temperature in x is for (i.e., p1 and p2 roughly lie on a line viewed from the thermal camera).

So, all in all:

1. Is my statement that the outlined solution can't always work correct?
2. Under which circumstances does the outlined solution work?
3. What do I need to keep in mind when building the setup?