I have multiple cameras that records the 2D pixel coordinates of the features each camera sees. After some searching, I came across this paper with a method to calculate the 3D coordinates of the features.

My problem is that the input of that algorithm requires a "feature tracks" for each feature. Meaning for each feature, there is a list of 2D coordinates corresponding to the camera that recorded them. In my case, I can have any number of features, how do I know which of the pixel coordinates belong to which feature? Is there a widely accepted algorithm for this? What would be the key word to search for the algorithm for this problem, would it be just a clustering algorithm I should look for?

Edit: In other words, I want to determine the 3D positions of a set of points by a set of 2D images taken by surrounding cameras. I do not know how many points I have.

  • $\begingroup$ What do you mean by "list of 2D coordinates corresponding to the camera that recorded them"? I don't understand what you're referring to. What do you mean by a "feature", specifically? $\endgroup$ – D.W. Apr 20 '19 at 22:35
  • $\begingroup$ @D.W. When a camera takes an image, a point on an object is projected onto that image. The position of that image is expressed as [x ,y], which are indices of a 2D array. By feature, you can think of a specific point on the object I am concerned with. $\endgroup$ – klWu Apr 21 '19 at 13:42
  • $\begingroup$ OK. See my updated answer. $\endgroup$ – D.W. Apr 21 '19 at 19:14

In general, without additional information, you can't, due to scale ambiguity; for example, you can't tell the difference between a 1" bird that's 10' away, vs a 2" bird that's 20' away. See, e.g., Using Headpose Vector and 2D Points to Compute Distances, calculting distance between a person and the object in the image, Is it possible to transfer a point from one camera to another, given n corresponding points?, find the actual size of an object from pixel coordinates.

If you don't have additional information but want to do the best you can, there are some heuristics; see Algorithms to convert 2D videos to 3D ones.

If you know the camera's instrinsic parameters, or the positions of the cameras, or other extra information, this is solvable Relevant technology:

  • object detection lets you find the 2D coordinates of an object feature in a 2D image (you can apply this to each 2D image)

  • algorithms for the correspondence problem let you match these points between multiple 2D images, even without first doing object detection

  • camera calibration lets you figure out the camera parameters

See also Fundamental and essential matrices, value of intrinsic parameters, How to obtain the camera extrinsic matrix without using chessboard.

  • $\begingroup$ I see. Camera parameters are available to me, so looks like I need to look into correspondence problem. Thank you. $\endgroup$ – klWu Apr 22 '19 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.