A “meta-optics” camera that is the size of a grain of salt

https://cacm.acm.org/news/a-camera-the-size-of-a-grain-of-salt-could-change-imaging-as-we-know-it/

by rbanffy • 2 years ago

244 131 2 years ago

cacm.acm.org

taosx• 2 years ago

That's a nice innovation that I'm not that happy about, as there would be even less privacy...

Maybe on the other side it's good news as ppl are usually their best selves when they are being watched.

mandmandam• 2 years ago

ppl are usually their best selves when they are being watched.

I don't think that view holds up.

A, it very much depends on who is watching, what their incentives are, and what power they hold.

And B, it also depends on who is being watched - not everyone thrives under a microscope. Are they the type to feel stifled? Or rebellious?

Scarblac• 2 years ago

Also, whose definition of "best self" are we using, that of the person being watched or of the person controlling the camera?

hypeatei• 2 years ago

Unless you're in your own home, I think it's basically a guarantee at this point that you're being recorded. Could be CCTV, trail cameras, some random recording a TikTok, etc...

ninalanyon• 2 years ago

That will only hold while being watched is rare. See Clarke and Baxter's Light of Other Days for an examination of the consequences of ubiquitous surveillance.

krunck• 2 years ago

The watchers would be able to blackmail/control anybody who engages in private activities that they don't want to be public. So who watches the watchers? And who watches them? No. Privacy is 100% required in a free society.

jdalgetty• 2 years ago

This won’t be good for society.

whynotminot• 2 years ago

How will it be all that different than the ubiquitous imaging we have now?

hk__22• 2 years ago

It will be the same, but worse.

timdiggerm• 2 years ago

You can sometimes find a hidden camera today.

rapnie2• 2 years ago

The rayban metaglasshole comes to mind. Now its just journalists who fool people in the street with AI face recognition tricks, and its all still fun and games. But this is clearly a horror invention, merrily introduced by jolly zuck, boss of facelook.

roflmaostc• 2 years ago

This is no news?

Has been published in 2021. Also here https://news.ycombinator.com/item?id=29399828

bhaney3• 2 years ago

produce full-color images that are equal in quality to those produced by conventional cameras

I was really skeptical of this since the article conveniently doesn't include any photos taken by the nano-camera, but there are examples [1] in the original paper that are pretty impressive.

[1] https://www.nature.com/articles/s41467-021-26443-0/figures/2

andrepd• 2 years ago

How does this work? If it's just reconstructing the images with nn, a la Samsung pasting a picture of the moon when it detected a white disc on the image, it's not very impressive.

nateroling• 2 years ago

I had the same thought, but it sounds like this operates at a much lower level than that kind of thing:

Then, a physics-based neural network was used to process the images captured by the meta-optics camera. Because the neural network was trained on metasurface physics, it can remove aberrations produced by the camera.

Intralexical3• 2 years ago

Ultrathin meta-optics utilize subwavelength nano-antennas to modulate incident light with greater design freedom and space-bandwidth product over conventional diffractive optical elements (DOEs).

Is this basically a visible-wavelength beamsteering phased array?

itishappy• 2 years ago

Yup. It's also passive. The nanostructures act like delay lines.

roelschroeven• 2 years ago

Those images are certainly impressive, but I certainly don't agree with the statement "equal in quality to those produced by conventional cameras": they're quite obviously lacking in sharpness and color.

neom3• 2 years ago

conventional ultra thin lens cameras are mostly endoscopes, so it's up against this: https://www.endoscopy-campus.com/wp-content/uploads/Neuroend...

card_zero• 2 years ago

I wonder how they took pictures with four different cameras from the exact same position at the exact same point in time. Maybe the chameleon was staying very still, and maybe the flowers were indoors and that's why they didn't move in the breeze, and they used a special rock-solid mount that kept all three cameras perfectly aligned with microscopic precision. Or maybe these aren't genuine demonstrations, just mock-ups, and they didn't even really have a chameleon.

queuebert• 2 years ago

Tiny cameras will always be limited in aperture, so low light and depth of field will be a challenge.

baxtr• 2 years ago

Also interesting: the paper is from 2021.

dcreater• 2 years ago

No link in the article to the actual paper?

hk__22• 2 years ago

It’s in the "Further Reading" section at the bottom: https://www.nature.com/articles/s41467-021-26443-0

mwigdahl2• 2 years ago

Chalk another one up for Vernor Vinge. This tech seems like it could directly enable the “ubiquitous surveillance” from _A Deepness in the Sky_. Definitely something to watch closely.

arethuza• 2 years ago

I wonder if someone tried to build a localizer how small they could actually be made?

PS It's "Vernor"

mwigdahl2• 2 years ago

Thanks, I typed that on my phone and it "fixed" it for me without me noticing.

cmpb2• 2 years ago

The other side to the localizers is the communication / mesh networking, and the extremely effective security partitioning. Even Anne couldn't crack them! It's certainly a lot to package in such a small form

ben_w• 2 years ago

I've been interested in smart dust for a while; recently the news seems to have dried up, and while that may have been other stuff taking up all the attention (and investment money), I suspect that many R&D teams went under government NDAs because they are now good enough to be interesting.

12907835202• 2 years ago

I haven't read deepness in the sky but it's interesting how wrong alot of scifi got this. Cameras are always considerably bigger than grains of sand

cmpb2• 2 years ago

Well, Deepness is set a few thousand years in the future, so we've got some time to work on it.

KineticLensman• 2 years ago

Also the scatterable surveillance cameras used in his other great novel, 'The Peace War' [0]. Although IIRC they were the size of seeds or similar.

[0] https://en.wikipedia.org/wiki/The_Peace_War

EdwardCoffin• 2 years ago

3 or 4 mm in diameter, according to a scene in chapter 6, big enough to have similar resolution to that of a human eye, according to Paul, but able to look in any direction without physically rotating.

In chapter 13 the enemy describes them as using Fourier optics, though that seemed to be their speculation - not sure whether it was right.

gcanyon2• 2 years ago

Or Rudy Rucker’s Postsingular, where the “orphidnet” utility fog enables universal perception/visualization.

aredox• 2 years ago

"Light of Other Days"/"Other Days, Other Eyes" by Bob Shaw is much closer - and poignant - take on that idea.

curiousObject• 2 years ago

If that’s true, maybe it would allow you to put a 10,000 camera array (100x100) on a smartphone, and do interesting things with computational imaging?

bhaney3• 2 years ago

Some rough numbers:

The paper says that reconstructing an actual image from the raw data produced by the sensor takes ~58ms of computation, so doing it for 10,000 sensors would naively take around ten minutes, though I'm sure there's room for optimization and parallelization.

The sensors produce 720x720px images, so a 100x100 array of them would produce 72,000x72,000px images, or ~5 gigapixels. That's a lot of pixels for a smartphone to push around and process and store.

fragmede2• 2 years ago

72,000*72,000* say, 24 bits per color * 3 colors, equals ~43 GiB per image.

edit: mixed up bits and bytes

jajko• 2 years ago

Sensor size is super important for resulting quality, that's why pros still lug around huge full frame (even if mirrorless) cameras and not run around with phones. There are other reasons ie speed for sports but lets keep it simple (also speed is affected by data amount processed, which goes back to resolution).

Plus higher resolution sensors have this nasty habit of producing too large files, processing of which slows down given devices compared to smaller, crisper photos and they take much more space, even more so for videos. That's probably why Apple held to 12mpix main camera for so long, there were even 200mpix sensors available around if wanted.

delegate• 2 years ago

First thought that came to mind - insect-sized killer drones. I guess that's the informational context we are in right now.

Dma54rhs• 2 years ago

You would still have to power the thing and store the data etc. This is just about the lense.

cjameskeller• 2 years ago

The Air Force was already publicly talking about such things in 2009: https://m.youtube.com/watch?v=_5YkQ9w3PJ4

pizza234• 2 years ago

as well as implementing unique AI-powered image post-processing to create high-quality images from the camera.

They're not comparable, in the intuitive sense, to conventional cameras.

Etheryte• 2 years ago

Are they not? Every modern camera does the same thing. Upscaling, denoising, deblurring, adjusting colors, bumping and dropping shadows and highlights, pretty much no aspect of the picture is the way the sensor sees it once the rest of the pipeline is done. Phone cameras do this to a more extreme degree than say pro cameras, but they all do it.

PittleyDunkin2• 2 years ago

To point out the obvious, film cameras don't, nor do many digital cameras. Unless you mean modern in the sense of "cameras you can buy from best buy right now", of course. But that isn't very interesting: best buy has terrible taste in cameras.

cubefox3• 2 years ago

"AI-powered image post-processing" is only done in smartphones I believe.

stevenae• 2 years ago

Pro cameras do not do this to any degree.

Edit: by default.

rippeltippel• 2 years ago

It's interesting how they mention beneficial impacts on medicine and science in general, but everyone knows that the first applications will likely be military and surveillance.

BiteCode_dev• 2 years ago

And since it's AI improved, all of th will hurt people because of hallucinations.

I don't trust human to avoid taking shorcuts once the tech is available, it's too convenient to have "information" for so cheap, and less costly to silence the occasional scandal.

meiraleal• 2 years ago

but everyone knows that the first applications will likely be military and surveillance.

military, surveillance and porn

jansan• 2 years ago

Can we agree that in the field of cameras we surpassed science fiction?

I can remember watching a TV series as a child where a time traveler went back to the 80s and some person told him that everything is about miniaturization. Then he pointed to a little pin on the time traveler's jacket, which was actually a camera, and said: "This little pin for example could one day hold a full video camera", which seemed a bit ridiculous at that time.

ripe• 2 years ago

Maybe I'm being too skeptical, and certainly I am only a layman in this field, but the amount of ANN-based post-processing it takes to produce the final image seems to cast suspicion on the meaning of the result.

At what point do you reduce the signal to the equivalent of an LLM prompt, with most of the resulting image being explained by the training data?

Yeah, I know that modern phone cameras are also heavily post-processed, but the hardware is at least producing a reasonable optical image to begin with. There's some correspondence between input and output; at least they're comparable.

mahoho• 2 years ago

I've seen someone on this site comment to the effect that if they could use a tool like dall-e to generate a picture of "their dog" that looked better than a photo they could take themselves, they would happily take it over a photo.

The future is going to become difficult for people who find value in creative activities, beyond just a raw audio/visual/textual signal at the output. I think most people who really care about a creative medium would say there's some kind of value in the process and the human intentionality that creates works, both for the creator who engages in it and the audience who is aware of it.

In my opinion most AI creative tools don't actually benefit serious creators, they just provide a competitive edge for companies to sell new products and enable more dilettantes to enter the scene and flood us with mediocrity

guerrilla2• 2 years ago

This seems like it's going to ne a serious problem for privacy... not that anyone cares.

nachox9992• 2 years ago

It is possible to create realistic images and videos with AI, making anyone do anything. Whether a photo or video is real or not will soon be impossible to distinguish, and it won't matter to those who want to cause harm

guerrilla2• 2 years ago

Whether a photo or video is real or not will soon be impossible to distinguish,

You're forgetting something. Chain of custody, trust and reputation. The source of an image or video matters as to whether it is considered a reliable representation of reality or not.

We will develop better methods of verifying sources as well, possibly using cryptography and new social networks where members authenticate each other in-person and trust is built up.

xg15• 2 years ago

I don't want a camera the size of a grain of salt! At least not while surveillance capitalism and creeping authoritariarism are in full swing...

rapnie2• 2 years ago

Just file a complaint with the United Nations Ethics Czar. Oh.. wait.

nachox9992• 2 years ago

For those who want to cause harm (discredit), they don't need a real photo; AI is enough

DCH34165• 2 years ago

We've always had this sort of stuff. Back in the 70s you had cameras the size of lighters. There's solutions for anyone determined enough. Even with authoritarian states, you'll find counter measures with sufficient demand. It's reed in the wind shit. Hopefully we won't kill ourselves in the process.

foul• 2 years ago

How would someone detect sensors so small?

How would someone excrete an array of these cameras if ingested?

kaimac2• 2 years ago

you could detect the supporting electronics with a nonlinear junction detector but they are not cheap

thfuran• 2 years ago

If you eat something the size of a grain of salt that isn't digestible, excreting it poses no problem.

ep_jhu• 2 years ago

Everyone here is thinking about privacy and surveillance and here I am wondering if this is what lets us speed up nano cameras to relativistic speeds with lasers to image other solar systems up close.

skandinaff• 2 years ago

we would also need a transmitter of equivalent size to send those images back. also an energy source

Workaccount22• 2 years ago

Just do round trip!

NL8072• 2 years ago

Honestly even if they are size of a jellybean, it would be a massive boon for space exploration. Just imagine sending them for reconnaissance work around the solar system to check out potential bodies to explore for bigger probes later down the track. Even to catch interesting objects suddenly appearing with minimal delay, like ʻOumuamua.

TeMPOraL2• 2 years ago

Thank you!

It's been a while since I've heard anyone talk about the Starshot project[0]. Maybe this would help revitalize it.

Also even without aiming for Proxima Centauri, it would be great to have more cameras in our own planetary system.

[0] https://en.wikipedia.org/wiki/Breakthrough_Starshot

hindsightbias• 2 years ago

Gilster writes about it every few months

https://www.centauri-dreams.org/2024/01/19/data-return-from-...

kaimac2• 2 years ago

Not a single mention of the obvious privacy concerns in the article

api• 2 years ago

This kind of thing -- that humans can do today with current technology -- is why if an ET intelligence that could travel interstellar distances wanted to observe us we would never know unless they wanted us to know.

Their probes could be the size of sand grains, maybe even dust. Maybe not quite sophons, but not much better as far as our odds of finding anything. I suppose there would have to be something larger to receive signals from these things and send them back (because physics), but that could be hanging out somewhere we'd be unlikely to see it.

Yet another Fermi paradox answer: we are looking for big spacecraft when the universe is full of smart dust.

cubefox3• 2 years ago

The article was published in 2021. Why do they repost this as "news" three years later?

d--b2• 2 years ago

There is some optics thing that looks cool, but it doesn't say how the image is actually recorded.

Then there is the whole "neural" part. Do these get "enhanced" by a generative AI that fills the blur based on the most statistically likely pixels?

The article is pretty bad.

DCH34165• 2 years ago

From what I can tell it's using a neural network to derive an image from the interference patterns of light.

I imagine you could do this using a standard computational model, it would just be very intensive. So I guess it would be 'enhanced' in the same way a JPEG stores an image in a lossy format.

d--b2• 2 years ago

My question was more about what it was that records the patterns of light.

alexpotato• 2 years ago

Years ago I saw an interview with a futurist that mentioned the following:

"One day, your kids will go to the toy store and get a sheet of stickers. Each sticker is actually a camera with an IPv6 address. That means they can put a sticker somewhere, go and point a browser at that address and see a live camera feed.

I should point out: all of the technology to do this already exists, it just hasn't gotten cheap enough to mass market. When economies of scale do kick in, society is going to have to deal with a dramatic change in what they think 'physical privacy' means."

brokensegue• 2 years ago

I'm very skeptical this technology already exists. Maybe if you vastly change the meaning of "sticker"

Workaccount22• 2 years ago

"PCB-with-onboard-battery-and-adhesive-backing-icker"

TeMPOraL2• 2 years ago

Nah, the only thing I believe will turn out inaccurate here is IPv6.

petra• 2 years ago

Maybe it's possible but i can't i seem to think of an energy harvesting Method that would fit that system without direct sunlight.

NL8072• 2 years ago

If these devices (one day) can run on the same power as a solar powered calculator, it might be possible to place them in ambient light conditions.

0xdeadbeefbabe• 2 years ago

And when you scratch the stickers they smell like strawberry.

burnt-resistor2• 2 years ago

(2021) The real story: https://light.princeton.edu/publication/neural-nano-optics/

casey2• 2 years ago

Is it possible to make an orbital death laser with this?

DCH34165• 2 years ago

I mean. If you find a way of harnessing enough energy.

ashoeafoot• 2 years ago

as an optic scientists i would protest my work being lumped zogether with the psychedelics of AIchemists

dmitrygr• 2 years ago

Grains of rice are pretty big, and the images they demonstrate are NOT that impressive. There are cameras you can BUY right now whose size is 1x1x2 mm (smaller than a grain if rice) which produce images that compare. Here is one example: https://www.digikey.com/en/products/detail/ams-osram-usa-inc...

It is pretty easy to interface with too - i did it with a pi pico microcontroller: https://x.com/dmitrygr/status/1753585604971917313

eurleif• 2 years ago

The OP describes them as the size of a grain of salt, not a grain of rice.

Nevermark• 2 years ago

Wow.

Given the tiny dimensions, and wide field, adding regular lenses over an array could create extreme wide field, like 160x160 degrees, for everyday phone cameras. Or very small 360x180 degree stand-alone cameras. AR glasses with a few cameras could operate with 360x160 degrees and be extremely situationally aware!

Another application would be small light field cameras. I don't know enough to judge if this is directly applicable, or adaptable to that. But it would be wonderful to finally have small cheap light field cameras. Both for post-focus adjustment and (better than stereo) 3D image sensing and scene reconstruction.

1024core• 2 years ago

The meta-optics camera is the first device of its kind to produce full-color images that are equal in quality to those produced by conventional cameras, which are an order of magnitude larger. In fact, the meta-optics camera is 500,000 times smaller than conventional cameras that capture the same level of image quality.

That would make them 6 orders of magnitude larger.

shrx• 2 years ago

It seems that they are comparing volumes.

gatkinso• 2 years ago

All kinds of exciting implications for small cameras and lens assemblies in VR/AR

isogon• 2 years ago

Becomes a lot less interesting when you consider that there's no way to power such a camera for any meaningful period of time without a much larger battery (ignoring the issue of storage/transmission).

kouru225• 2 years ago

lmao of course the image looks like those monster pov shots from a 1980s horror movie

legohead• 2 years ago

In The Culture series books, there's a concept of "smart dust". Basically dust sized surveillance drones that they cover planets with, that lets them see and hear everything that's going on.

burnt-resistor2• 2 years ago

This reminds me of Lytro, a "light field" camera, that could be focused digitally after-the-fact to any focal plane using online software.

Does anyone know why Lytro couldn't be shrunk to fit in smartphones? Because this seems like similar technology.

A “meta-optics” camera that is the size of a grain of salt

Related Stories