Image2Mesh: Single Image 3D Reconstruction

https://arxiv.org/abs/1711.10669

by cpheinrich • 8 years ago

97 10 8 years ago

arxiv.org

malux85• 8 years ago

No source code :(

consultutah• 8 years ago

No code yet, but promised: http://www.cs.cmu.edu/~chenk/

stealthcat• 8 years ago

analogous to neural net super-resolution -- producing plausible hallucinations (in this case, 3D objects instead of more pixels)

andreyk• 8 years ago

TLDR They use a neural net to process an image to find a similar known 3D model to the object in the image as well as parameters for how to deform that model to be like the object in the image, and also then perform a linear combination of some related meshes (this is a follow up to their prior work "Compact Model Representation for 3D Reconstruction"). Pretty exciting work, but still far from being able to robustly do Image2Mesh. I happen to have worked on something quite similar, if any of you are curious: https://deformnet-site.github.io/DeformNet-website/

epberry2• 8 years ago

Interesting, thanks for the link. I tried a few published methods for doing this a year or so and I found they were quite slow. As in, you'd rather just use a stereo camera to quickly generate a decent pointcloud than use a single view plus a network like these. What did you find your performance (ms/frame) was like?

bsenftner3• 8 years ago

With separate dedicated identifiers and matching reconstruction pipelines, while only caring about a limited number of things, as in our specializing in faces, we can reconstruct 25 to 30 independent people, their heads and shoulder tops only, in real time on a 3.4 GHz i7. Note the reconstruction is accurate only as far as it matters for FR, as in we tend to ignore hair. So the models are bald, but they have the frame to frame facial expressions for everyone. Here's a gif of a degraded lighting test at 70 fps with one head doing a 3D reconstruction independently performed each frame: https://www.dropbox.com/s/0jtee4ofh2vdeg0/BlakeHeadMeshes.gi... I'm just visualizing the face, but the entire head and shoulder tops are reconstructed.

bsenftner3• 8 years ago

Surprised this is getting any notice at all. I work at a facial recognition firm pioneering this technique 20 years ago. (www.CyberExtruder.com) The TLDR description by andreyk pretty much sums the process, as it originated long ago. It is very similar now, just with a team of very smart people iterating over the pipeline for so long has made it sophisticated to the point the TLDR description could be argued as too light.

I even made a 3D avatar API & service, now closed, from the technology 10 years ago. The twitter site is still around, if you want to see the quality of reconstruction possible: https://twitter.com/3DAvatarStore/media

pokoleo• 8 years ago

Arxiv Vanity: https://www.arxiv-vanity.com/papers/1711.10669/

Image2Mesh: Single Image 3D Reconstruction

Related Stories