Forge – mobile 3-D capture [video]

by acjohnson55 • 12 years ago

82 35 12 years ago

therobot24• 12 years ago

nice! I'd be interested in the method they use to put everything together. My best bet is some basic structure from motion weighted by the depth sensor...or maybe it's simpler than that...

nobbis15• 12 years ago

Author here. It uses color info as well as depth for tracking. Otherwise, it'd fail if you pointed the camera at featureless geometry, e.g. walls, floors.

phorese• 12 years ago

Looks great :)

In the video it looks like you are using a volumetric representation, perhaps an Octree+isosurface extraction?

joshvm• 12 years ago

Looks neat - what depth sensor were you using to generate that video? It's not totally clear to me from the comments.

ihnorton2• 12 years ago

At the end of the video, one camera is from PrimeSense, and the other (appears) to be a SoftKinetic (or Senz3D); is that correct? Does anyone have experience comparing those cameras for accuracy/fov/framerate/etc.?

nobbis15• 12 years ago

Correct. It's a SoftKinetic DS325.

Its FOV is higher than the PrimeSense sensor, it's capable of higher frame rate but it's a little less accurate and it's range is currently shorter.

The two technologies (structured light vs time of flight) have different strengths/weaknesses, but I believe ToF is the future.

ihnorton2• 12 years ago

Thanks. Have been playing with the Senz3D (rebranded DS325) for a rather different application, but was a bit underwhelmed so far with both accuracy and range. (On the other hand, it's kind of amazing for $125).

bromagosa• 12 years ago

I tried the URL that's shown at the end of the video and it didn't work :(

Any live demos?

nobbis15• 12 years ago

No, sorry. I'm moving coast to coast next week, but hope to start beta testing late April.

Enter your email on the website and I'll let you know before beta testing starts.

diafygi• 12 years ago

Great demo! A few questions:

1) What previously existing libraries are you using (if any)?

2) Since the video capture (assuming it's from the tablet's camera) is detached from the depth capture, how do you coordinate the two to make the grid layout in the video? Structure.io seems to require attaching the depth device to the video device (so the perspectives are relatively fixed).

nobbis15• 12 years ago

1) The core reconstruction technology is written from scratch (it's been a full-time project for over 18 mths.)

Some open source projects it uses: Eigen, OpenNL, OpenMesh, OpenCTM, GLM, Protobuf, Redis, Gluster, Node.

2) Video capture is actually from the RGB-D sensor, so it's registered with the depth. I believe the Structure sensor requires that the camera is fixed to the mobile device so that its IMU can be used for tracking. No such requirement here.

jrpowers• 12 years ago

Jeff from the Structure Sensor team here. Structure Sensor comes with an attachment to hold it rigidly attached to an iPad (and CAD so you could 3D print attachments for other devices). Color-depth registration is handled by the SDK and is registered using vision (not IMU) -- but IMU is available if desired. Hope to see this project ported to be compatible with Structure Sensor!

mierle• 12 years ago

No Ceres?

bayesianhorse2• 12 years ago

Will it be free? Will the models be free? Would be very neat when used with blender...

nobbis15• 12 years ago

I'd really like for light usage to be free, but I'm undecided as to the best monetization strategy.

One option is for model creators to be paid when users download their public models.

bayesianhorse2• 12 years ago

The most profitable route would be aiming for a sharing platform, getting early stage investment, build a userbase, then sell the company. Both Facebook and Google are eyeing this field.

It's not even clear if this is more risky than a paid model, because the internet landscape is littered with corpses of paid-model web applications which didn't take off "enough".

hatuman• 12 years ago

Seems like the tip of a very big iceberg. I sent this on to a friend who creates 3D construction and real estate models. I think he needs to diversify before too long...

nobbis15• 12 years ago

I agree. Also, Forge's underlying streaming 3D technology has lots of applications - mobile capture is just the first.

rmc2• 12 years ago

I did not know you could buy off the shelf 3d cameras. Where can I buy one?

nobbis15• 12 years ago

DepthSense 325 - http://www.softkinetic.com/Store/tabid/579/ProductID/6/langu...

DepthSense 311 - http://www.softkinetic.com/Store/tabid/579/ProductID/2/langu...

Creative Senz3D - http://us.store.creative.com/Creative-Senz3D-Depth-and-Gestu...

Structure Sensor - https://store.structure.io/preorder

Asus Xtion Pro - http://www.newegg.com/Product/Product.aspx?Item=N82E16826785...

MechSkep2• 12 years ago

This looks great! I'd like to use it to ease up the tech overhead for some mobile robotics applications. Is there anyway that the user's position in the world frame can be accessed? i.e. can this do SLAM over longish distances?

Basically I'm looking for Google's Project Tango, without the hardware constraints.

Pxl_Buzzard• 12 years ago

Is it possible to use a Leap Motion? I don't think it has a video camera in it, but could it still be used to do 3D scanning?

jc_dntn• 12 years ago

this is going to totally change the way we send dick pics.

yogrish• 12 years ago

This seems to be in line with Project Tango of Google. https://www.google.com/atap/projecttango/

nobbis15• 12 years ago

Project Tango has a 320x180 depth sensor running at 5 fps, i.e. 290k depth measurements per second. Compare this with off-the-shelf depth cameras (e.g. DS325) that generate 320x240 at 60 fps, i.e. 4.6m measurements per second.

The reason for this is that mobile processors aren't fast enough to process more information. So the Tango prototype has to have, in addition to its depth camera, a special motion tracking camera with a fish-eye lens and 2 dedicated processors in order to robustly track.

Even then, with less depth information, the quality of any Tango reconstruction will be far inferior. Maybe in 5-10 years, mobile processors can approach what desktop GPU's are capable of today.

In any case, it remains to be seen if Google can persuade cellphone manufacturers to include 2 special cameras + 2 extra processors in their future devices.

Forge – mobile 3-D capture [video]

Related Stories