Non-Rectangular Video Cropping with FFMpeg

https://dragonquest64.blogspot.com/2019/11/non-rectangular-video-cropping.html

by lipeltgm • 7 years ago

68 32 7 years ago

dragonquest64.blogspot.com

ArtWomb3• 7 years ago

My new years resolution might just be to gain mastery of ffmpeg. It's incredibly versitile. Combine with python bindings, image magick and opencv and you have a powerful weapon in your arsenal.

But I keep running out of memory. I like that the process just dies. And can usually be solved by sequential batching. But how can I automate via predictive instrumentation?

the84725• 7 years ago

ffmpeg can operate in on streams of frames, you don't have to keep everything in memory or operate on discrete batches.

BubRoss3• 7 years ago

I don't know what you mean by predictive instrumentation or why you are running out of memory, but don't forget the original interprocess communication, the file system.

ArtWomb3• 7 years ago

Thanks for the replies the8472 and BubRoss!

I just mean I'd like to "cap" ffmpeg's memory usage to ensure it never crashes.

I'll provide a simple example. Creating an image slideshow from a directory of huge images. I can downscale and compress to jpeg before processing. But even then, besides batching, it always fails. And this is just raw, without any filters or image processing. Have tried all the i/o techniques from the wiki:

https://trac.ffmpeg.org/wiki/Slideshow

All this local laptop based (4GB). I just want to get a better handle before devoting cloud resources (ffmpeg is standard on gcloud serverless) ;)

pselbert• 7 years ago

Learning ffmpeg can be daunting but it is a highly useful tool to know. The option intricacies and peculiarities make it extremely hard to “master” though. It requires so much experimentation and experience (trial and error) that achieving mastery in a year seems nigh impossible!

rejschaap• 7 years ago

It has some aliasing artefacts around the edge of the circle, you can see it very clearly during the fade-in. When the video is playing you don't really notice it. My guess is that this is caused by the way the mask is created. I can't try it out right now, but it could probably be solved by setting the RGB channels of the mask image to black.

_Gyan_5• 7 years ago

See https://superuser.com/questions/1270950 for a way to do it without needing to create a masking image.

lipeltgm• 7 years ago

Agreed. The 'gex' filter can readily accomplish the masking if the shape can be represented as an X,Y equation. This method does however work for a more comprehensive list of shapes, titled wrt circular since that seems to be a highly desired form of <i>cropping</i>. Thank you for pointing it out and likely would address the aliasing another reader referenced.

the84725• 7 years ago

Note that a few video codecs also support alpha these days, which could be used to preserve the mask and overlay the video over some background.

It even works on the web: https://files.catbox.moe/uu2ze0.webm (needs a non-black background)

jpincheira• 7 years ago

Wow. Pretty cool. Had no idea about this.

izietto• 7 years ago

here with a yellow background (edit: changed to a yellow-to-green circle-shaped radial gradient because why not): https://codepen.io/mdesantis/pen/RwNZEZJ

pbhjpbhj2• 7 years ago

Maybe, https://codepen.io/pbhj/pen/dyPzwEj instead - centre the video and put the gradient on the body to emphasise the matting?

pbhjpbhj2• 7 years ago

Firefox (Linux): I had to open dev tools, remove the bg-image that Firefox shows around media links, and then adjusting the bg-color showed the effect. Otherwise it just looked like a normal video.

vernie• 7 years ago

I think it's more appropriate to call this matting. I came in thinking they'd figured out how to coax FFmpeg into encoding non-rectangular frame geometries.

GJR• 7 years ago

Fill and key is basic video manipulation. As to the aliasing, they also need to apply some filtering...

batt4good• 7 years ago

Is there any way to provide a dynamic mask? For instance, if I wanted to blur a dynamic portion of each fram, like a face or logo?

I've been trying to find higher-performance methods of doing this with live footage that don't require openCV. Seems like FFmpeg doesn't have a great way to do this or I just haven't found it yet?

I guess I'm also looking for frameworks or tools that can handle live video "manipulation" like rendering shapes or graphics. OpenCV is painfully slow for most of this stuff, everything else is seemingly completely proprietary. But nVidia seems to have some incredible tools baked into their latest stream manipulation API's.

rambojazz• 7 years ago

Some day someone will write a simpler replacement for ffmpeg's user interface. A tool that you have to read the manual every time you want to write (or even just understand) a command is not a good tool.

_Gyan_5• 7 years ago

The CLI syntax design reflects the N-inputs --> N-outputs capability and no. of options reflects the number of components e.g. there are 300+ demuxers and 400+ filters available.

This is not to say there isn't scope for improvement. A couple of areas for improvement is the consistency of options naming across filters that implement the same feature within those filters, and a clearer syntax to identify the target of an option i.e. protocol, demuxer, decoder..etc

the84725• 7 years ago

perhaps a declarative config file instead of CLI arguments would help. With a config file you could lay things out in a more tree-like manner, have lists etc.

And it's not just N inputs, M outputs. You also have a processing graph for the intermediate filters. So ideally you want graph nodes represented by some identifier which can then be referenced by other nodes.

bobosha• 7 years ago

Actually, someone already has: https://kkroening.github.io/ffmpeg-python/

Non-Rectangular Video Cropping with FFMpeg

Related Stories