1
$\begingroup$

I observe a scene with two cameras, c1 and c2, that produce two images i1 and i2, respectively. What I now want is to bring i1 and i2 into alignment, that is I want to know where pixel (x,y) in i1 is in i2. This "alignment-matrix" should be valid in general, that is not only for i1 and i2, but also for further images of the cameras c1 and c2 as long as they haven't been moved. I believe this problem is called "correspondence problem".

I have now read quite a bit about camera calibration, image rectification, essential and fundamental matrix. But there are still open questions and I would not know how to achieve my task.

So, the question basically is:

The essential matrix E and the fundamental matrix F both only give constraints for the correspondence problem (point p in i1 must lie on line l in i2). How do I actually solve the problem?

Thanks a lot for your time and answers!

Improve this question
$\endgroup$
2
  • $\begingroup$ That's 3 separate questions. We prefer you stick to one question per question, please. I suggest you edit out bullet items 2 & 3 and post each one separately as its own question. Thank you! $\endgroup$
    – D.W. ♦
    Commented Jul 29, 2015 at 17:06
  • $\begingroup$ okay, will do, sorry :) $\endgroup$
    – user1809923
    Commented Jul 30, 2015 at 7:05

1 Answer 1

Reset to default
2
$\begingroup$

For 2D cameras, you can't. There's a reason why the matrices only give constraints that let you say that point p in i1 must lie on line l in i2 -- that's because that's all you can say.

In particular, for a particular point p in image i1, with 2D cameras, you don't know the depth of the corresponding object. The depth affects where it appears in image i2, so without knowing the depth, you can't know which pixel in i2 corresponds to p. If the object is very close, then there might be a very large offset; if the object is very far away, there might be a very small offset.

This is in fact the basis of binocular vision. Some 3D sensing methods work by somehow finding the correspondence using various heuristics (e.g., image similarity metrics), and then using this to infer the depth of the object.

So, the intrinsic and extrinsic parameters are not enough to uniquely deduce the correspondence, if both cameras are 2D cameras. With a 3D camera, this problem can be solved.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Oh, then I am in trouble. How do people use information from two different image sources simultaneously then? E.g., I have an optical and a thermal image and want to use both information in my algorithms. How do I do that? At the moment I could only think of: fundamental matrix -> rectify -> disparity map. Use the disparity to associate individual pixels. This however seems very error prone, especially since thermal and optical images can look very different... $\endgroup$
    – user1809923
    Commented Jul 30, 2015 at 7:09
  • $\begingroup$ @user1809923, "How do people use information from two different cameras?" is a very broad question; try doing some research on the correspondence problem, and on inferring 3D scenes -- there's lots written in the computer vision literature. For combining a 2D camera and a 3D camera, see cs.stackexchange.com/q/44838/755. But: This is not a discussion forum; it's a question-and-answer site. Please don't use comments to ask new follow-up questions. Comments should only be used to help improve answers. (cont.) $\endgroup$
    – D.W. ♦
    Commented Jul 30, 2015 at 17:39
  • $\begingroup$ Instead, make sure that your original question explains the problem you want to solve, do your research before asking so you ask the right question, and post new follow-up questions as a separate, new question. Thank you! $\endgroup$
    – D.W. ♦
    Commented Jul 30, 2015 at 17:39
  • $\begingroup$ Okay, I see. Even though my "follow-up" question basically also is the correspondence problem and therefore part of my original question. I guess the difference could be that I asked for a general/exact solution which you answered by saying it does not exist :) $\endgroup$
    – user1809923
    Commented Aug 6, 2015 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.