Alignment of RGBD with a thermal camera

Question

I have a thermal camera that I would like to overlay, automatically, to the output of a RGB-D camera (e.g. Kinect).

I have retrieved the intrinsic and extrinsics parameters of both cameras.

My objective is to calculate the disparity map between the two. That is, to map the matrix resulting from the depth image to the thermal image, in order to estimate the distance of each point in the thermal image. From an intuitive standpoint, I should be able to align them because I have depth information in this case.

A solution that comes to my mind, it is as follows:

• Step 1: for each pixel of the depthmap, estimate its corresponding 3D point. Let's call it Depth_3D.

• Step 2: transform Depth_3D in Ther_3D. That is, given the 3D position of the point of the depth camera, find the 3D position of the same 3D point (object in the real world), but respect to the thermal camera. The relative position and orientation of the two cameras is known.

Problem: to perform step 2, I need the 3D rotation matrix (let's call it R3D) and 3D translation vector (let's call it T3D) to be able to map from the depth camera to the thermal camera, in the real world.

• Step 3: Finally, using the intrinsic parameters of the thermal camera, project each Ther_3D into the thermal image.

• Step 4: assign the depth to the obtained pixel, so that, at the end of the loop, the depth map of the thermal image is complete.

My doubt is about how to calculate R3D and T3D (please see step 2) using their relative position and orientation. Other solutions are welcome of course.

Thanks in advance.

Answer 1

Step 2 is just a change of coordinate system from one camera's coordinate system to the other camera's coordinate system. It's easy to do, using a "change of basis".

The extrinsic parameters of the depth camera tell you how to transform from the depth camera's coordinate system to the world coordinate system (a transform between two 3D coordinate systems). The extrinsic parameters of the thermal camera tell you how to transform from the 3D world coordinate system to the thermal camera's coordinate system. If you combine these two transformations, you have a way to transform from Depth_3D to Ther_3D.

So if you have the extrinsic parameters, you should already know how to do this.

See, e.g., https://en.wikipedia.org/wiki/Pinhole_camera_model and https://en.wikipedia.org/wiki/Camera_matrix and https://en.wikipedia.org/wiki/Camera_resectioning, or any textbook that describes camera intrinsic and extrinsic parameters.