# Can computer vision process 3D “images” directly?

Most CV algorithms deal with 2D images produced by cameras. However, sometimes we need to process 3D “images” (I don’t know how to call them). For example, CT and MRI produce 3D radiographs. Traditionally these radiographs are cut into many slices for doctors to read. Because human eyes can only see 2D images (and most people lack the ability to comprehend complex 3D structures), doctors need to analyze them slice by slice, which is very time consuming. Computers on the other hand can process 3D patterns as efficiently as 2D. So can we design a CV algorithm to process 3D images directly? For example, instead of cutting the radiographs into many slices and mapping the edges (which marks the cross sections of blood vessels), the algorithm maps the surface of blood vessels directly. Will such direct 3D processing be more efficient and more accurate than 2D processing?

This is a very general question, so I'll give some ideas.

The short answer is yes. If you're talking about the "processing" as some function, then yes, we can use some function $$f_3: \mathbb{R}^3 \to A$$ in our computer algorithm rather than a $$f_2: \mathbb{R}^2 \to B$$. In fact, there are many computer algorithms that already do this. In fact, you can give any $$n$$-dimensional input, and have some algorithm process the $$n$$ inputs into anything you want (e.g. lots of econ models do this).

In terms of efficiency, it depends what you consider efficiency. But I think what you're trying to get at is, "Is there a 2D algorithm that can be more efficient if done in 3D?", and I would say no, because 2D algorithms are designed for 2D, and 3D are designed for 3D. E.g. if you want to do edge detection in 2D, it doesn't even make sense to use it in the context of 3D (there would be 3D collision or edge detection instead, a different version).

However, to entertain your question about efficiency, there are ideas of "creating features" from lower dimensional data, moving it to upper dimensional data and then doing analysis. PCA and neural networks are some examples that do that, and are sometimes very good at it. E.g. the classic is number digit recognition, where the 2D problem is quite difficult, but you might be able to find more patterns in higher dimensions and recognize the numbers more easily (neural networks are great for this).

Also, even though you might think of your vision as 2D (because that's how cinema portrays it), you have depth perception and many other senses that help you perceive 3D, which actually would make your visual senses a 3D rather than 2D function as you might think.

Doctors look at CAT scans in 2D because it might help them target certain areas they want to look at more. There are of course software that can process these slice by slices into 3D, and I would bet doctors use those as well. Also, you have to remember that CAT scans are "looking inside" of something in the first place, so it might be more helpful to see cross-sections rather than the whole thing.

• Are there any functions that can only be realized in authentic 3D processing? For example, edge detection only works in 2D. If you cut a 2D image into many stripes, you will detect a lot of false positives (like dots).
– 哲煜黄
Jun 19, 2022 at 9:15
• Again, specific algorithms for specific spaces only make sense in certain contexts, so in your case, maybe edge detection works in 2D, but it doesn't make sense to do that in 1D, so yes, you can say that's the case. You can do boundary detection in 3D as well as 2D and 1D, which is what edge detection sort of it, so you can say that's technically the same thing. But end of the day, it really boils down to what you want to do in each space. Some things will be extendable and others won't, depending on the application. Jul 17, 2022 at 2:30