Recording 660FPS Video on a $6 Raspberry Pi Camera (2019)

https://blog.robertelder.org/recording-660-fps-on-raspberry-pi-camera/

by tosh • 5 years ago

293 63 5 years ago

blog.robertelder.org

adolph4• 5 years ago

The major limitation throughout this process is the speed at which memory can be copied and transmitted. Only 20-40 seconds of video can be recorded at a time due the memory exhaustion of the Raspberry Pi. Also limited is the resolution of the recording: On the $6 camera, a maximum of 640x64 resolution can be recorded due to limitations on memory bandwidth.

tosh• 5 years ago

Does anyone know if that means the Pi 4 can record for a longer period of time (it supports up to 8GB RAM iirc)?

bick_nyers• 5 years ago

I haven't read the article, but the parent comment makes it sound as though it's not so much the capacity of the bus, but rather the speed of the bus itself. This is usually expressed in terms of a combination of CAS latency and frequency in MHz, the Pi 4 being DDR4, is likely in the neighborhood of 2133-3600Mhz. We don't often think of reading/writing to RAM as taking any time at all, but in all actuality the speed of memory is often a significant bottleneck in high performance systems.

If we are just talking about 720p 30fps, then we likely aren't really that close to those memory bottlenecks, so the only limiting factor would be how much storage space you have available to that device.

LaputanMachine• 5 years ago

According to the video in the article, the data rate is 66 MB/s (100kB per frame). So a Pi with 8GB of RAM should be able to capture almost 2 minutes of footage.

monocasa• 5 years ago

Probably some, but not to the degree that you may think. This relies on DMA in the CSI controller, but only a fraction of a pi4's RAM is accessible by DMA. Basically from the video core side, there's only a 32-bit bus to main memory, and it has to share that with MMIO space and there's mirrors with/without caching sort of like a MIPS. Probably maxes out after a couple GB or so.

justin66• 5 years ago

It looks like the repo [0] hasn't been touched in a couple of years so it's tough to say what if anything would need to change, but this application sounds like it would benefit from a Pi 4.

[0] https://github.com/RobertElderSoftware/PatientTurtle

randomluck040• 5 years ago

Would it be possible to go for a lower framerate and gain more flexibility regarding the resolution?

Forgeties795• 5 years ago

That’s how it works with more conventional camera set ups. I guess it just boils down to how they program it but there’s no reason that shouldn’t be the case.

up6w6• 5 years ago

Would parallelizing it to two distinct cores help with the issue?

jandrese2• 5 years ago

Not if you're bottlenecked on memory bandwidth or capacity.

adolph4• 5 years ago

From my reading of the issue, no. The limiting factor on the Pi side is available storage buffer. The camera sends raw files down the wire and the app caches them in RAM before concatenating them with ffmpeg. Could a Pi4 with USB3 NVMe be used? Maybe, if the writes are faster than the camera.

https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o...

seba_dos1• 5 years ago

No, this is about memory bandwidth.

squarefoot• 5 years ago

Then why not tunneling the sensor output through the lowest available network layer using the NIC, then reassembling it on a faster machine connected point to point (no IP stack involved) to the RPi?

sydthrowaway• 5 years ago

At that point, the NRE cost would dictate to buy a high end system

AlexAltea• 5 years ago

This is possible by interfacing directly with the sensor MMIO registers. I've always found fascinating how much flexibility hardware provides, compared with the typically rigid functionality the drivers expose to userland.

I wonder what we could make out of many high-end cameras if only we had enough patience and motivated people to reverse engineer everything that has been hidden away by comfort abstractions.

bserge• 5 years ago

Magic Lantern is a firmware mod for Canon cameras and it gives you more features and control over your hardware. I love this kind of low level mods.

https://magiclantern.fm/features.html

pstoll• 5 years ago

Link is dead.

sidpatil• 5 years ago

Also see CHDK [1], a FOSS firmware replacement for Canon point-and-shoot cameras.

[1] https://chdk.fandom.com/wiki/CHDK

Forgeties795• 5 years ago

Magic Lantern basically is the reason my t3i got me into film festivals lol

pishpash• 5 years ago

Ironically, I find the popularity of hardware hacks to be a sign of general stagnation, because there were periods of time in the past when they would not give as much boost in performance as a new version of something in a few months to a year. They were typically not worth the time and would be rendered obsolete, but now more and more they are worthwhile.

helge92102• 5 years ago

everything that has been hidden away

It's not hidden, it's stripped according to actual product specs.

Industrial cameras (Basler/IDS) will give you access to most of the sensor capabilities if you want to prototype something.

ClumsyPilot• 5 years ago

Yeah but for the price of a good systens canera you will get only a basic industrial one with a tiny sensor and no AF

Forgeties795• 5 years ago

If you want the answer to that question, look no further than the magic lantern project. It’s not supported like it used to be, but those guys were incredible at unlocking what Canon DSLR’s could really do. I was a legend in my film community when I booted it onto a 5Diii and got it shooting 1080p raw. All I did was follow their instructions lol

austinpena3• 5 years ago

Those guys were my intro to tech! I remember shooting RAW on an old 50d and Dual ISO on my 6d!

lovelyviking2• 5 years ago

Last time I've checked there was nothing like that for Nikon? What is the reason? Or there is something?

vhiremath4• 5 years ago

for anyone wondering, "MMIO registers" stands for "memory-mapped I/O" registers:

https://en.wikipedia.org/wiki/Memory-mapped_I/O

mahalol• 5 years ago

Great timing on this article. I have a rpi4 and a pi cam v2 which I want to have some fun with and was in need of inspiration. I am currently using it as a security camera while I am away on holiday but wish to have a play both with slow motion video and perhaps some object detection using opencv. The latter I have already tried but all the articles I found seem out of date so I will need to spend some more time on.

kbouck• 5 years ago

As webcams were hard to come by during the onset of the pandemic, I resorted to a raspberry pi + HQ cam + lens + raspistill preview + HDMI capture device (instead of usb). I won't defend the total cost (probably more than an acceptable webcam). The HQ cam results were quite good and seemingly high enough framerate (not 660 fps though!). Added benefit is interchangeable lens options including ones that will give the bokeh effect.

matheweis• 5 years ago

This could be a big step toward ultra low latency streaming with an rpi.

I tried to do this (low latency streaming) circa 2016 and ran into two issues, one of which was the high framerate or subframe capture and the other was subframe encoding.

With high framerate capture you can fake the subframe capture by capturing a whole frame and encoding just a subset.

Now to get the h264 drivers to do subframe encoding…

adolph4• 5 years ago

I'm interested in synchronizing sensor data to the camera stamps. For example, strain gauge to measure model rocket engine performance. Does anyone know of a project or approach that would allow for synced high speed sensors?

wiml• 5 years ago

Low tech approach: I wonder if you could put a flashing LED in the camera's view, and sync that with the strain gauge data logging, kind of like an old fashioned move clapper board?

adolph4• 5 years ago

That sounds like an excellent idea, thanks!

kejaed• 5 years ago

An approach I’ve seen used in the past was to add lines to the video frame and write other sensor data in “extra” lines in the frame before writing to disk. You can then parse the data after the fact and know what the sensor data was for each frame.

TomJansen• 5 years ago

Would it also be possible to increase the exposure time to above the 10 seconds which is officially supported by the raspberry pi V2 camera?

Havoc• 5 years ago

That's amazing. I couldn't even get decent 24fps out of it....

kragen3• 5 years ago

The 1007 fps he said he got out of the US$6 V2 camera is over 8 times the 125 fps at which exporting the camera from the US requires an arms dealer license: https://www.bis.doc.gov/index.php/documents/regulations-docs... but that only counts if the shutter speed is under 1 μs.

So these cameras should still be safe from ITAR.

dr_zoidberg3• 5 years ago

I'm a bit confused here. There are a few Sony cameras (the RX series) that have a 960fps capabiilty, with exposure times of under 1/1000th of a second. That's around 8 times the 125 fps limit set by ITAR, yet those cameras are sold pretty much everywhere around the world. What am I missing/not understanding here?

Scaevolus• 5 years ago

Japan doesn't have the same export laws as the US.

kragen3• 5 years ago

1 μs is less than 1000 μs. (And, yes, also they are probably not made in the US.)

Recording 660FPS Video on a $6 Raspberry Pi Camera (2019)

Related Stories