Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

https://worldsheet.github.io/

by ArtWomb • 6 years ago

122 33 6 years ago

worldsheet.github.io

tunesmith• 6 years ago

I'm wondering when we'll get to easy Blade Runner tech, where taking a photo will let you browse it in a 3d sense, extrapolating to some level of certainty what certain aspects "behind" objects or out of view would like look, based off the analysis of the visible elements in the photo.

ArtWomb• 6 years ago

In lab settings, it seems very close ;)

Detecting Invisible People

http://www.cs.cmu.edu/~tkhurana/invisible.htm

moron4hire6• 6 years ago

That's not the same thing. This paper is talking about extrapolating objects that become occluded in the video stream until they become visible again, to provide a continuous sense of tracking.

bflesch• 6 years ago

Amazing work, well done!

moron4hire6• 6 years ago

I wonder how long it takes to generate these images. It's not terribly hard to do by hand with rudimentary 3D modeling skills (i.e the most generous way to describe my own 3D modeling capability), if you know the height from the floor and shooting angle at which the camera took the picture. I've done it with 360 photos (therefore shooting angle didn't matter) in Blender with a VR headset to aid in positioning vertices. You can get results like this in about 15 minutes.

Complex scenes obviously take longer, but results can be extremely convincing for a limited range of view angles.

I learned the technique from this video: https://youtu.be/RSC40B8kHT8

sitkack• 6 years ago

15 minutes. I dont know about this particular technique, but others are in the 10-50ms range. Roughly 20k times faster.

moron4hire6• 6 years ago

My point is, if you need this sort of imagery, you don't need to wait for Facebook to never actually release a usable product out of this paper. Because that's generally what happens with these kinds of papers.

philip142au• 6 years ago

Could you combine many of these images together to make a 3D environment for a computer game, for example inside the bedroom take many photos and then interpolate between the 3D sheets as the player moves around?

moron4hire6• 6 years ago

That is basically what photogrammetry is all about. It's not easy, but it can work really well when done well.

mrfusion2• 6 years ago

Could this be a camera app on the latest iPhones with the built in lidar like technology? You’d snap an image and a mesh, right?

ben_w2• 6 years ago

You could, but you don’t need to use a modern phone with a depth sensor – this works with pure RGB and doesn’t need depth information.

mrfusion2• 6 years ago

In the future you could train a model to predict the correct mesh for a given image. Then things get interesting!

moron4hire6• 6 years ago

That's what they are doing. The rest of the process is not novel.

yorwba• 6 years ago

They're not trying to get the correct mesh, they just treat everything as part of one big continuous surface.

That's good enough for sideways views, but fails if you try to look behind objects, because in their approach, there's no "behind", just a big, stretchy sheet covering everything.

A model with more knowledge about the world would be able to predict that a tree trunk is roughly cylindrical and not connected to the background.

ackbar03• 6 years ago

Didn't read the paper in detail, but seems just an extension / application for single image depth prediction problem?

btown2• 6 years ago

To quote the paper:

Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time.

Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, but the title seems misleadingly magical.

kfarr• 6 years ago

Very curious to understand this as well. Does that imply that images with similar attributes (like other hotel rooms in one of their examples with slightly different dimensions or furniture) will work without additional training? Or did each example require multiple image training of that scene before a 3d mesh can be computed from arbitrary angle? In the latter case it can still be useful as compared to photogrammetry which would require hundreds of photos to achieve similar results, so maybe this could work with a dozen or less?

hijklmno• 6 years ago

Unfortunately that's most(all) of academia!

steerablesafe• 6 years ago

I don't see how that would work with the paintings.

svantana• 6 years ago

I think you're misunderstanding the paper. They are using multiple images in the learning phase to infer the 3d structure. At test time, it works on novel scenes with just a single image.

btown2• 6 years ago

I believe from a second reading this is correct - after training on the Matterport real estate image dataset, it appears they can transform novel images (which happen to be from that dataset, so they are still home interiors but can handle completely distinct scenes with distinct furniture etc.). Which is actually really fascinating and leverages decades of innovation. This is actually really cool and fully justifies the hype in the title.

nayuki• 6 years ago

It'd be nice to see this new technique replace the very crude warping animation in Google maps' street view.

crazygringo2• 6 years ago

Google Maps Street View already collects some low-res lidar data, so they don't even need this.

They just need to build it into their viewer, which I also wish they would do.

moron4hire6• 6 years ago

They do use the Lidar data in the viewer. You can see it with the positioning of the cursor on "surfaces" when you're looking around. The reprojection animation they do between photospheres is based on the Lidar data, it's just that's about the best you can do with the low-res data they've collected.

Imagine a situation where you're standing on a street corner with a mailbox or something standing on the corner. You move to the next photo position, which takes you past the mailbox. The smearing and warping you are seeing is from the continous mesh they've created from the Lidar data not matching the real world mesh in areas that were occluded from the original point of view. You get a moment of seeing "behind" the mailbox, but there is no data for what is behind the mailbox, so it all gets interpolated from the data that is known in the surrounding visual area. The mailbox becomes a rectangular prism that extends all the way from the mailbox's location through to the intersection with the ground that we can see behind the mailbox.

These are just the problems with single-image mesh recreation. You can't really get around them without some form of inference of the data that doesn't exist. You even see it in Facebook's images in the linked article, if you look closely. They try to cut the videos early so you don't see it, but it's there if you know what to look for.

londons_explore• 6 years ago

I believe the crude warp is caused by the difficult combination of:

* Need to do animation with very low latency (can't go to the network to collect data for the animation when the user clicks)

* Need to do the animation without much CPU/GPU power in the browser.

* Need to not download much data for the animation ahead of time (browsing panoramas is already very data heavy).

anorakoverflow• 6 years ago

I’ve tried out Apple’s Look Around recently and was impressed with the transition animation compared to Google Maps. If you have an Apple device, try it out in London for instance: Moving past a red telephone box shows how the object kind of retains its shape during the transition. Traffic moving around you also looks quite different compared to Google.

taneq2• 6 years ago

Does anyone know what the Facebook "3D Photos" feature is doing and how/if it's different to this? Because that is generally terrible (much worse than the 'failure mode' examples here) and the depth map generated seems largely arbitrary.

andybak• 6 years ago

It's worth making a distinction between "3D Photos where the user supplies depth data from the dual cameras/LIDAR etc" vs "3D Photos where Facebook's magic algorithms try to infer depth". It's the latter that I presume you're criticising here. (Might be obvious to you but I thought it worth spelling out)

ben_w2• 6 years ago

I think this paper is also about the latter, it’s just a much better version of the latter - the abstract says RGB image not RGBZ, and one of the examples is a painting.

taneq2• 6 years ago

True, and the latter is indeed what I was referring to. This looks like it does the same thing as Facebook's (train a neural net to infer depth from image contents) but it actually works fairly well.

Generating new views from an rgb+depth image is relatively straightforward and I'd expect any reasonable implementation to work pretty well.

moetsi• 6 years ago

the latter was 6d.ai's whole thing. I say "was" because they were bought by Niantic

Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

Related Stories