# Alignment of RGBD with a thermal camera

I have a thermal camera that I would like to overlay, automatically, to the output of a RGB-D camera (e.g. Kinect).

I have retrieved the intrinsic and extrinsics parameters of both cameras.

My objective is to calculate the disparity map between the two. That is, to map the matrix resulting from the depth image to the thermal image, in order to estimate the distance of each point in the thermal image. From an intuitive standpoint, I should be able to align them because I have depth information in this case.

A solution that comes to my mind, it is as follows:

• Step 1: for each pixel of the depthmap, estimate its corresponding 3D point. Let's call it Depth_3D.

• Step 2: transform Depth_3D in Ther_3D. That is, given the 3D position of the point of the depth camera, find the 3D position of the same 3D point (object in the real world), but respect to the thermal camera. The relative position and orientation of the two cameras is known.

Problem: to perform step 2, I need the 3D rotation matrix (let's call it R3D) and 3D translation vector (let's call it T3D) to be able to map from the depth camera to the thermal camera, in the real world.

• Step 3: Finally, using the intrinsic parameters of the thermal camera, project each Ther_3D into the thermal image.

• Step 4: assign the depth to the obtained pixel, so that, at the end of the loop, the depth map of the thermal image is complete.

My doubt is about how to calculate R3D and T3D (please see step 2) using their relative position and orientation. Other solutions are welcome of course.