# using orientation sensor data to predict image points

I want to use Android's gyro/accelerometer/magnetometer to predict (over a very short time interval) how image points corresponding to a fixed object will change (without trying to scan for them in the image). In particular, suppose we have an object of known size with model points $v_1, \dots, v_N$. In this question, all vectors are assumed to be in homogenous form. Here is what I am trying:

1. Detect the object. Let the pixel coordinates of the model points be denoted by $u_1, \dots, u_N$. Given the calibration matrix $K$ and negligible lens distortion, solve the PnP problem to get rotation $R_0$ and translation $t_0$ from the coordinates in which the vectors $v_1, \dots, v_N$ are defined into camera-centric coordinates, shown in black in the image above. In particular, the solution to the PnP problem solves the following equation. $$u_i = K \begin{bmatrix} R_0 & t_0 \\ 0^\top & 1\end{bmatrix} v_i.$$

2. Simultaneously query the Android rotation sensor for a rotation matrix.
This is the relative orientation from Earth coordinates to the device coordinates (colored frame in the picture above). The direction of the this transformation is discussed informally in the Android developer site:

The rotation vector represents the orientation of the device as a combination of an angle and an axis, in which the device has rotated through an angle $\theta$ around an axis ($x$,$y$, or $z)$.

Let $R_1$ denote this rotation matrix.

1. Rotate the phone a small amount and query the rotation sensor again for $R_2$.

2. Render the predicted location to the pixel coordinates. For this, lets assume the sensors and the camera are collocated, so that the transformation from camera coordinates to device coordinates is a pure rotation: $$D= \begin{bmatrix} 0 & -1 & 0 & 0\\ -1 & 0 & 0 & 0 \\ 0 & 0 & -1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$ This matrix maps the black frame to the colorful frame in the picture. Note that the $+z$ axis of the camera frame (black in the picture) is meant to point out of the back of the phone, so that it constitutes a right-handed coordinate frame.

The transformation from the body coordinates of the object $v_1, \dots, v_N$ to the image coordinates in the perturbed camera frame $u_1', \dots, u_N'$ are given by $$u_i' =KD\begin{bmatrix}R_2 R_1^\top & 0 \\ 0^\top & 1 \end{bmatrix} D \begin{bmatrix} R_0 & t_0 \\ 0^\top & 1 \end{bmatrix} v_i,$$ where we used the fact that $D=D^{-1}.$

Right now, I am stuck tying to verify that this equation actually works. Here is a simple experiment (that I have tried in Python). Let $v = (0, 0, 1)^\top,$ and suppose it is already defined in the camera coordinate frame, i.e., $R_0=I$ and $t_0=0.$ Moreover assume that $R_1= I,$ Then, I rotate the device by a small angle about the $x$ or $y$ axis (which are the same for the device and for the Earth because $R_1=I.$) This rotation constitutes $R_2$ from the equations above. When I try this in Python, this is what I observe for various counter-clockwise rotations:

test rotation of device :: expected result :: observed result

rotate device about $+x$ :: $x$ pixel coordinate increases :: opposite behavior

rotate device about $-x$ :: $x$ pixel coordinate decreases :: opposite behavior

rotate device about $+y$ :: $y$ pixel coordinate decreases :: as expected

rotate device about $-y$ :: $y$ pixel coordinate increases :: as expected

• Do you know the distance to the object? Without knowing that, I don't think it's possible. If the object is close by, then a small rotation can cause a large change to its location in the image. If it is far away, then its location in the image will change only by a small amount. You can't infer the distance to an object (in general) given just a single image.
– D.W.
Jun 2, 2017 at 22:31
• Yes, you can determine the distance because we know the exact model points $v_1, \dots, v_N$. This gives us scale information so that you can solve the PnP problem. Jun 2, 2017 at 22:33

• Even though the rotation matrix reports the rotation from Earth to Device, I should have been applying the opposite rotation to the point. It took me a day to convince myself of this. Basically, if you rotate the whole frame about, e.g., $+x$, then coordinates that remained fixed w.r.t. the device will rotate w.r.t. the Device frame about $-x$.