# Registering 3D-NIR image to thermal image and vice versa

In the past I have thought a bit about how to register a NIR-image and a thermal image and noticed that this is not trivial - one statement was that if I had the depth information for each pixel, the task would be much easier.

Now consider having a 3D-NIR camera (e.g. asus xtion) and a thermal camera and I want to map the thermal information to the depth map (the 3d cloud) or vice versa and the NIR information to the thermal information (and vice versa).

I thought I could do it simply like this:

1. Conduct a stereo camera calibration for the two cameras (-> R1,R2,T)
2. Use R1, R2 and T to transform the 3D points from 3D-NIR to the thermal camera's coordinate system
3. Use the camera matrix of the thermal camera to project the 3D points to its image plane
4. Now I know where the 3D points "fall" in the thermal image, which gives me a depth value (and intensity value, because the depth map and NIR-image are aligned) for each pixel in the thermal image.

Which to me sounds similar to the procedure described in this answer by D.W.. However, I can't find any article describing this method. Always there seems to be some form of feature matching step, e.g. here.

Also, thinking about this, the above solution can not really work, I believe. Consider the case where the two cameras look at an object from different sides. The 3D camera will calculate the depth for a point p1 in the world. When projecting p1 to the thermal camera's image plane it will fall in pixel x. However, since the thermal camera looked at another side of the object, another point in the world p2 actually formed pixel x when the thermal camera took the image - and thats what the measured temperature in x is for (i.e., p1 and p2 roughly lie on a line viewed from the thermal camera).

So, all in all:

1. Is my statement that the outlined solution can't always work correct?
2. Under which circumstances does the outlined solution work?
3. What do I need to keep in mind when building the setup?

Thanks for reading this long post!

• What are your assumptions? Are you assuming the two images were taken from the same position (i.e., the two cameras are at the same position)? Or might the NIR-image have been taken from a different position from the thermal image? That's going to fundamentally change the nature of the problem. In general if they are in different positions there will be limits on what you can achieve. – D.W. Jul 28 '16 at 19:41
• I think your Q1 and Q2 is a dup of cs.stackexchange.com/q/47234/755. Also, the site doesn't work so well when you have multiple questions in a single post. I think it'd be better to split this into two posts: one asking under what conditions your proposed solution work; another asking for a solution to the problem that meets some specific requirements that your existing proposal doesn't achieve. – D.W. Jul 28 '16 at 19:44