Vulkan video extensions for accelerated H.264 and H.265 encode

https://www.khronos.org/blog/khronos-finalizes-vulkan-video-extensions-for-accelerated-h.264-and-h.265-encode

by mshachkov • 3 years ago

245 104 3 years ago

shmerl13• 3 years ago

In addition, given the high industry demand for AV1 codec support, an AV1 decode extension release is imminent, with an AV1 encode extension development also underway.

That's good.

By the way, mpv already supports Vulkan video decoding:

https://github.com/mpv-player/mpv/issues/11739

Now we need OBS and Firefox supporting Vulkan video too.

dist-epoch• 3 years ago

On Windows/Mac Firefox uses the system hardware video decoding APIs - DXVA and Video Toolbox.

shmerl13• 3 years ago

On Windows Vulkan video would work. But not on macOS (that mpv thread mentions it).

On Linux it should be a better option than VAAPI once drivers will support it. Since Firefox is using ffmpeg anyway, it probably shouldn't be too hard to make it an option.

kllrnohj3• 3 years ago

Now we need OBS and Firefox supporting Vulkan video too.

Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos. Why use Vulkan video extensions? It seems like it just complicates things since MacOS/iOS and Android are unlikely to get these and Vulkan on Windows' is a second-class citizen as well. Not to mention the feature & API availability is lagging very far behind native APIs that they're already using.

It's not a bad extension necessarily, seems like it'd be a great option for games that want to integrate some cutscenes. But if you have an existing solution, it also doesn't seem valuable to migrate to this, either.

jdiff2• 3 years ago

Firefox still has no support for hardware video decode on Linux. There's all sorts of shims for it, but I've never managed to get them functioning. Maybe Vulkan video decode could be the one agnostic API to rule them all this time, especially if it's more cross-platform than the current situation.

shmerl13• 3 years ago

I think a very simple reason is I'd rather rely on Vulkan driver for faster iteration, than on VAAPI. It's historically moving much slower since it first needs Intel to pull weight, then AMD to implement their side for example.

tagrun7• 3 years ago

They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos

One vendor specific API, not "various OS' native APIs".

Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA. (A third-party VA-API to NVDEC translation layer for Linux does exist on GitHub, nvidia-vaapi-driver, but it's not yet reliable as the officially supported VDPAU or NVDEC, and is not included in official linux package repositories.)

Intel has VA-API, AMD has AMF, and NVIDIA has VDPAU which is being replaced by NVDEC/NVENC.

The idea behind Vulkan Video Extensions is to have a vendor independent and cross-platform video API.

geraldhh4• 3 years ago

Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos

they are also plentiful, vendor specific and unstable.

a generic solution would be well appreciated

luthMan• 3 years ago

What about systems where there is now hardware support for AV1? How would Vulkan implement that?

Also, maybe someone knows more about this, but I thought that most software like Firefox, MPV and VLC are using dav1d for AV1 decoding as it seems to be the most performant AV1 decoder (in Software) out there at the moment.

shmerl13• 3 years ago

How would Vulkan implement that?

See the post. They are working on AV1 extensions.

dishsoap• 3 years ago

So to clarify, this still leverages the fixed function hardware video encoders present alongside many GPUs from eg. Intel/NVIDIA/AMD, and would not (for example) be possible to use on just a GPU itself (without said encoding hardware)?

shmerl13• 3 years ago

It is for using GPU video related ASICs (same as VAAPI let's say). If you want to use shaders for some video filters, scaling and such - you don't need those extensions.

mpv for example can use Vulkan for both at the same time.

monocasa• 3 years ago

Interestingly, quite a bit of video decode isn't as parallelizable as you might think and isn't a great fit for GPUs. For instance the initial Huffman decoding of the stream is essentially an intrinsically sequential process.

0x1ceb00da• 3 years ago

Is there a reason we don't have parallelizable video formats?

bonton89• 3 years ago

During the video card shortage I was kind of lamenting that straight encoder cards just don't exist anymore. To get the best encoder acceleration you were stuff fighting with gamers and crytominers over the same video cards even if you didn't need the majority of the hardware for your task.

dlojudice• 3 years ago

vkGetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR

how to exceed the character limit on your super wide screen with just one function name

suprjami• 3 years ago

Eskil Steenberg suggests width of code as a measure of quality. Vulkan wins.

IPTN• 3 years ago

Huh, It only takes up 2/3 of my phone screen width in portrait mode and reads fine at full arms length.

mft_• 3 years ago

Needs more snake case for added width sillyness :)

lostmsu• 3 years ago

C needs an extension for import without namespaces purely based on names:

  import vkGetPhysicalDeviceVideoEncode;

  QualityLevelPropertiesKHR(...);

Or even better:

  import vk*PhysicalDeviceVideoEncode;
  
  Get_QualityLevelPropertiesKHR(...);

temp0826• 3 years ago

This already works decently well in mpv though you need the latest (from git) mesa/libplacebo/ffmpeg to use it.

Vulkan video extensions for accelerated H.264 and H.265 encode

Related Stories