SceneScript, a novel approach for 3D scene reconstruction

https://ai.meta.com/blog/scenescript-3d-scene-reconstruction-reality-labs-research/

by Klaster_1 • 2 years ago

132 29 2 years ago

ai.meta.com

ganzuul• 2 years ago

I would love to have this as a Blender add-on.

dTal3• 2 years ago

I'm sure the tech is neat. But I get a real sinking feeling from how hard tech giants like "Meta" are trying openly to create dystopia. The "Metaverse" is not aspirational!

That understanding would let AR glasses tailor content to you and your individual context, like seamlessly blending a digital overlay with your physical space

Horrifying, if it ever takes off in the way that "Meta" wants it to. Take the modern smartphone disease and amplify it a few orders of magnitude.

While LLMs rely on vast amounts of training data that typically comes from a range of publicly available text sources on the web, no such repository of information yet exists for physical spaces at the scale needed for training an end-to-end model.

Considering the tremendous scale in question, the "yet" is also terrifying. The only way to collect such a dataset would be if some large fraction of people were wearing always-on, network connected cameras. Meta clearly sees this future and wants it; the kind of future where everyone wears a Meta headset, all the time, and has their experience of reality mediated through it - optimized to maximize shareholder value of course, under the watchful eye of The Algorithm. A panopticon AI shoggoth.

Meta is an AI paperclip maximizer, and it is getting uncomfortably close to the "RELEASE THE HYPNODRONES" stage.

globalnode• 2 years ago

A panopticon AI shoggoth

That made my day. I do agree, it is creepy but its what theyre pushing.

idle_zealot• 2 years ago

Re: that "yet". I noticed that after a recent update on my Quest the OS gave me a pop-up about a new feature, that when enabled would allow the device to remember my play boundaries and restore them automatically rather than requiring they be re-entered every time the system gets confused about where I am. I was a little confused why it didn't just turn that on by default, it seemed like a straight upgrade to the current handling and shouldn't require anything but a bit of local storage. Scrolling down in the pop-up I was informed that enabling the feature would authorize Facebook to scan my surroundings and upload a complete map of them to use for whatever purposes they desired. I guess they legally need consent for that, and tying a feature improvement to data collection is still totally allowed somehow. So I imagine that they are building that real-world dataset presently.

dTal3• 2 years ago

I guess they legally need consent for that, and tying a feature improvement to data collection is still totally allowed somehow.

As egregious as this is, it still doesn't strike me as nakedly insulting as Google's refusal to let you turn on the (entirely passive!) GPS receiver on your phone without uploading your precise location to their servers, for similar data collection purposes.

I'd love to see this sort of thing be made illegal, but the only regulatory domain with an Overton window even close to that is the EU, and it's such a minefield of potential gray areas that I doubt it would get off the ground even there.

ricardobeat2• 2 years ago

The only way to collect such a dataset would be

You must have missed the part where they created a synthetic dataset of 100000 environments, trained the network on that, and released it to the public?

Not saying I’m not worried about their plans too, but that is clearly addressed in the text.

bobajeff3• 2 years ago

They still have to solve for:

* How uncomfortable the headsets are to wear for long periods if time

* How useless they are (what are they solution for?)

* How ridiculous they look

* How expensive they are

* How expensive it is to produce content for

After they solve these issues then I will worry about the dystopia we are in. VR headsets may look cool and futuristic in the movies but in reality they are just screens you wear on your head and that's it.

gibmeat• 2 years ago

All of these (except maybe the "useless" part) are almost always solved by time (Moore's law)

dTal3• 2 years ago

Much has been written about the dystopia we are already in, but I am specifically concerned about the dystopia they are earnestly building. Meta is aware of these problems and is trying very hard to solve them.

And they won't be useless! That's the worst part. They will provide compelling value, like smartphones, and as such will be impossible to boycott. At first the value will be simple, obvious utility, like route overlays. Later it will be value within the system, much like how the main utility of smartphones now is to run the mandatory apps to participate in aspects of society that previously did not require a smartphone. And of course they will be addictive.

Here is a short sci-fi film depicting what I fear: https://www.youtube.com/watch?v=YJg02ivYzSs

FeepingCreature• 2 years ago

The Metaverse is not aspirational!

Wait, Snowcrash was horror for you? I thought the whole point of post-cyberpunk was to not assume all tech would be dystopian.

jprete• 2 years ago

Snowcrash is a dystopian libertarian world cleverly hidden by the viewpoint characters treating it as completely normal.

ImHereToVote• 2 years ago

No code. Only for internal research.

ricardobeat2• 2 years ago

Obviously, this is likely to be part of a future Quest software update. They did release the training dataset though.

FlyingBobcat• 2 years ago

Want more data? Create an mobile app to cartograph new rental appartments. I know in other countries you often rent with furniture in it. We in germany for example take everything with us and have to check if/how the furniture fits.

With a system like the one in the video you walk through the old place, and walk through the new place, and output for example 2 Sweet Home 3d plan's.

No need to have 3d glasses for that.

WhatIsDukkha2• 2 years ago

This was a very important missing piece for mixed/augmented and virtual reality.

The semantic understanding of the objects around you (without upload to a server).

A detailed set of actual meshes of the objects in your living and working spaces.

So much can be built on this key building block, to actually help people with their daily lives, low vision people are only one group that will see massive benefits.

I also think this technique could and should be extended generally into computer vision applications, understanding semantic meaning, volumes and masses via the often highly structured environments around us.

Starting from a principled position of meaning, volumes and meshes and then training to assign gaussian splats to those meshes seems like something clever people could solve a lot of problems with.

nox101• 2 years ago

A huge problem is privacy

without upload to a server

Doesn't matter. If the app has to render them then it can still send them to the server. If the app does not render directly it can still infer them by doing collision testing against them. If it can't even check for collisions then there isn't much point in having the data in the first place.

It gets worse. The more you want AR/MR to understand the scene the more fine grained data you need. The camera sees things like: the underwear you left on the floor, the medicines/pills on your table, the posters on your wall, the bank account browser tab still on your computer screen, the adult toys you forgot to put away, your undressed body, your dressed body, your political baseball cap or t-shirt, your naked roommate/spouse/kids who just walked into the room.

I don't see how this can be solved in a trust-less way. In other words, you'll have to trust each and every individual AR/MR app you use, that they aren't spying on you. The platform/device/OS itself can not prevent this and still allow the majority of the AR/MR uses that people imagine.

littlestymaar• 2 years ago

Companies don't spy on you because they don't like you, they spy on you because they make money out of it. Outlaw making money out of personal data (by outlawing targeted ads, data purchase for insurance, and data brokers) and the incentives to spy on you disappear.

You can't be probusiness and pro-privacy at the same time though, you need to limit businesses in order to protect the people.

thoughtlede• 2 years ago

Yes, the camera sees everything. But we could avoid teaching the model certain things. For instance, an undressed body and a dressed body could both be taught as a body. Likewise, medicine pills as well as regular mints could just be taught as mints.

While this approach doesn’t address privacy completely, it avoids certain elements of our life that are considered really private.

I think half of the point of using the simulated dataset to train the model was to safeguard the above mentioned kind of privacy (with the other half being lack of real-world dataset).

jonplackett• 2 years ago

I think there must be a way of doing this so you just have to trust the OS. The camera feed itself could be essentially private and only pass on locations and types of objects. In this way you could whitelist / blocklist items or categories.

But you would defo have to trust the OS, and in this case Meta…

vergessenmir• 2 years ago

Isn't this more of A semantic understanding of how language maps onto geometry rather than understanding of actual space and geometry?

I sometimes feel like this may not work without language helping it along

WhatIsDukkha2• 2 years ago

With a multimodal llm watching a stream of these objects, should be able answer the question -

"where are my keys?"

It should be able to respond -

"Here on the dining room table" with an astar path drawn to them in the headset.

It's a stream of tokens that you can almost just reverse search on to find the last instance of "keys" in it (thats a simplification but...).

pyinstallwoes• 2 years ago

So, potentially how humans turn language into a rendered version of reality? :)

doctorhandshake2• 2 years ago

My issue with this approach is that the synthetic data, and the language used to describe it, necessarily encode a viewpoint - a bias - on the part of the creators. I am firmly behind the need to use language as an intermediary signifier, not just for utility purposes, but because it unlocks the ability to use language as an anchor for arbitrary AR data. My problem with this particular approach is that it requires us all to agree on what things are called, and requires the training dataset to cover everything users might come across.

My feeling is the better approach is to train a generalized segmentation model, derive multi-parameter representations of those segmented ‘things’, derive embeddings from those things for compression purposes, then allow users to ‘tag’ or ‘label’ or ‘describe’ those things as a service to other users. Then we could choose whose language about the world we ‘subscribe’ to, and anyone could attach any digital payload to any real thing/person/place without anyone else’s help or permission.

mycall• 2 years ago

My problem with this particular approach is that it requires us all to agree on what things are called

Walk around your home and notice [almost] everything already has a name for it as it comes from some store (unique handmade items are an outlier). Let's just label things we all agree they are called (or give them multiple labels).

doctorhandshake2• 2 years ago

I think you overestimate the degree to which your choice of words overlaps with those of others, especially when you try to be more specific than a generic noun by applying modifiers or less-used nouns. What you call a daybed I might call a couch or a sofa or a lounge chair or a chaise or a hassock or a flat couch or a downstairs bed or a family bed … and all this presumes I speak the same flavor of English you do, or that whoever made the data set is familiar with the type of furniture I live around, or how it’s used. The examples are literally limitless, before you even get into the truly subjective things - not just names but opinions about things - that people might want to use to connect digital things to real things. Stuff like ‘larger than average’, ‘wasteful’, ‘beautiful,’ ‘sacred’, ‘durable’, etc etc etc

Believing there is one map from the real world to the world of symbols is solipsism.

wdb• 2 years ago

Wished big companies would stop naming their stuff using existing terms, like in this case, scene script.

Alifatisk• 2 years ago

This reminds me a little of SegmentAnything

SceneScript, a novel approach for 3D scene reconstruction

Related Stories