Drawing Graphics on Apple Vision with the Metal Rendering API

https://github.com/gnikoloff/drawing-graphics-on-apple-vision-with-metal-rendering-api

by georginikolov • 2 years ago

142 83 2 years ago

I've put everything I've learned while developing graphics for Apple Vision with the Metal API and Compositor Services in writing. It covers foveation, vertex amplification, stereoscopic rendering, and more techniques that extend beyond this particular device.

fsloth• 2 years ago

Thanks! Looks very interesting and usefull.

nelup20• 2 years ago

Looks great, thanks so much for writing this!

tomovo2• 2 years ago

I'm curious about the autoreleasepool. What's the reason to have it and how much of a difference does it make?

flohofwoe2• 2 years ago

It's a lifetime management system for short-lived objects, same idea as memory arenas but for manually refcounted objects you got out of an API. It keeps the object alive until the end of the autorelease scope without anybody having to explicitly call the release method (the autoreleasepool does this instead). Not sure if the concept still makes sense with ARC though, it's probably just a no-op there - or probably not seeing that Swift seems to have inherited autoreleasepools from ObjC).

tomovo2• 2 years ago

Hi! Yeah I know the concept from non-arc ObjC times but trying to understand the point in this context.

lapcat4• 2 years ago

After finally releasing a Vision Pro version of my app last week, I've sold a total of 7 units. That's by far my worst software launch ever, even worse than my past failures.

Don't bother developing for this platform. The customer base is miniscule.

that_lurker• 2 years ago

You should still get familiar with it in the event that the next one will be cheaper as that will increase the customer base a lot.

closewith• 2 years ago

On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM), production deliveries slashed, and the reported shareholder near-revolts, I think it's clear that the Vision Pro has flopped and there won't be a sequel any time soon.

troupo3• 2 years ago

So, a developer should spend ~$5000 with no chance of ROI on the off chance that at some undefined point in the future this will pay off when they would have to spend another ~$5000 developing for the newer/cheaper/whatever device?

jsheard2• 2 years ago

The rumor mill is saying it will be cheaper... but still not cheap at ~$2000.

https://www.uploadvr.com/non-pro-apple-vision-headset-report...

Meta is having enough trouble attracting developers with hardware that's an order of magnitude cheaper and has two orders of magnitude more units in the wild. They got through the first hurdle of convincing people to try VR but keep falling at the second hurdle of user retention, headsets that are gathering dust in a closet don't make any money for developers.

Apples decision to go all-in on hand tracking compounds the software problem even further because the vast majority of VR software that has already been developed is designed around controllers with IMUs/buttons/sticks/triggers/haptics, and would be very difficult to port over to the Vision Pros much more limited control scheme.

jncfhnb• 2 years ago

Could hit double digits next time

moolcool• 2 years ago

You're still making a big assumption that this is a device anybody actually wants in the first place. How many of the people who bought one of these actually still use them with any regularity?

Cthulhu_• 2 years ago

Familiarity is fine, but don't bet your livelihood on it just yet; have enough knowledge to be able to pivot to it if it does take off. But I'm not seeing it.

jayd162• 2 years ago

I doubt it... The platform seems like a dud without some big changes. If big changes are coming, how much time to market do you really save working on this iteration?

layer8• 2 years ago

The platform has no killer mainstream use-case besides watching movies on a big virtual screen in solitude, with the drawback of having a heavy headset strapped to your face. I doubt that it would fare very well even sub-$1000. If they invested in gaming and allowed 3D controllers, the story might be different.

reaperducer2• 2 years ago

Don't bother developing for this platform. The customer base is miniscule.

A good hockey player don't skate to where the puck is. He skates to where it will be.

brudgers• 2 years ago

50% of hockey players are below average.

bee_rider4• 2 years ago

I mean… I think Apple hasn’t totally missed VR, and is still the only company that seems to have much of a chance of making it work…

But, a good hockey player doesn’t skate in the direction that absolutely nobody else is heading, somewhere that might even be outside the rink (in the sense that we haven’t really shown if a good UI for VR can actually be created, yet, so it might not even be a possible “part of the game,” so to speak).

lapcat4• 2 years ago

A good hockey player don't skate to where the puck is.

I'm not a hockey player.

He skates to where it will be.

The time scale for hockey players is seconds, not years.

This is such a useless cliché.

Most developers have very limited resources. Apple can afford to invest in pipe dreams (reportedly they spent $billions working on the now cancelled car project), but I certainly can't.

madeofpalk• 2 years ago

I'm doubtful this is where the puck will be.

gpm3• 2 years ago

Any chance you can link to your app?

gpm3• 2 years ago

Replying to myself, I believe it is this: https://apps.apple.com/us/app/stopthemadness-pro-spatial/id6...

airstrike2• 2 years ago

In my hallucination, my customer base is myself. If I had extra time right now and one of these devices to play with, I'd build something I wish existed, not something I expect the general population to buy.

If it's good enough for me, it's probably already worth my time—though https://xkcd.com/1319/ comes to mind

Then maybe some subset of the population that is closer in interests to me would also find it useful, in which case the app might get more traction. That'd be nice, but not a necessary requirement for my definition of success

The big issue here is justifying spending thousands of dollars on the product. I'd go for it at $500 or so. Definitely not $2k or $5k.

latexr• 2 years ago

The developer in question makes a living selling Safari extensions, and the Vision Pro app they’re referring to is a version of what’s probably their most well-known. So we’re talking about an established developer with an established user base selling a version of an established app.

Your advice makes sense in general but doesn’t apply to this specific case.

wpm3• 2 years ago

I'd build something I wish existed

Then you also better hope Apple permits you to use the APIs you need to build it.

FumblingBear• 2 years ago

For what it's worth, I'm a big fan of your app and was hoping for a Vision Pro release. I didn't know you released it, but will be buying it today.

I know that doesn't pay the bills, but I appreciate the hard work you do.

azinman2• 2 years ago

What’s the app?

georginikolov2• 2 years ago

Hey, OP here. The game is called RAYQUEST, please check it out here: https://rayquestgame.com/

assassinator42• 2 years ago

I would assume targeting OpenXR would be much more productive? It's unfortunate that Apple insists on being so proprietary.

gpm3• 2 years ago

The app in question appears to be a safari extension... so... unlikely.

spookie• 2 years ago

They are a latecomer to this market. Decisions like this are going to kill any hopes of success...

(Just a guy who has seen all proprietary APIs in this space being left in the dust when a standard arrived)

corysama• 2 years ago

I've always assumed the Vision Pro was an experimental platform that is available publicly solely to enable developers to figure out what the heck can be done with a high-quality MR device.

wpm3• 2 years ago

Nothing can really be done with it until Apple provides more software and APIs for it, at least, aside from some medicore iPad app ports.

And yes I'm talking about most of the "native apps".

wpm3• 2 years ago

Friggin Apple didn't even bother porting all their apps over, not sure why anyone else should.

xivusr• 2 years ago

What's yr app? I'll check it out if its still available

dagmx8• 2 years ago

This is a pretty great reference repo in general for optimized mobile rendering, and most of it other than compositor services, can be generalized to other headsets in terms of approach (not API).

Thanks for putting this up!

neomantra3• 2 years ago

This is a great read, thank you for sharing! I learned a lot from reading it and I'm pretty experienced in this space! I enjoyed checking out your other work too.

OPs article heavily covers Vertex Amplification. I didn't realize VisionOS used Vertex Amplification for this, but it makes sense since the hardware supports it.

For those interested in Mesh Shading (a few good videos released this week [1][2][3]), Vertex Amplification is a key tech for Mesh Shading, where one writes Object/Mesh functions (names in Metal API, called Task/Mesh/Amplification shaders in other APIs). Introduced by Nvidia in 2018 and only really available en masse for the last couple years, Vertex Amplification was the first time a GPU could create vertices (or "destroy" them by not emitting anything), versus the fixed mesh inputs. It's so cool and powerful and a different way of thinking about the pipeline.

This article shows the same concept, but in Vertex Shaders for multiple render targets. While you might not make a Vision app, it could be worthwhile read to further understand this architecture. I've spent a few months in Metal Mesh Shading and hadn't realized this application of it at all.

[1] https://news.ycombinator.com/item?id=41839190

[2] https://www.youtube.com/watch?v=3EMdMD1PsgY

[3] https://www.youtube.com/watch?v=EtX7WnFhxtQ (good explanation and demo)

dagmx8• 2 years ago

Nit: amplification shaders are not the same as mesh shaders (though in many cases they can be seen as abstractions over one)

Your description is of a mesh shaders, but an amplification shader is able to basically reuse vertex data from a vertex shader pass without use of a mesh shader.

cubefox3• 2 years ago

Nit: Mesh shading, or the mesh shader pipeline, has two unique parts: optional task/amplification shaders, and mandatory mesh shaders.

tasoeur• 2 years ago

For people who want to play with metal shaders on Apple Vision Pro I recommend this live coding app: https://shader.vision

Animats• 2 years ago

Metal is the main reason many games are never ported to MacOS.

Metal is roughly equivalent to Vulkan, which is on Windows and Linux. But Apple just had to Think Different. So all the major rendering libraries either have a layer which abstracts over Vulkan and Metal, or they don't support MacOS at all.

That's what WGPU does. If it were not for Apple ignoring standards, much of that would be unnecessary. It adds so much excess baggage that the consensus is to go direct to Vulkan or DX12, and blow off MacOS support.

spease• 2 years ago

Does Metal allow for better performance with Apple hardware than if Vulcan were the “direct” interface, if you optimize with it?

GeekyBear2• 2 years ago

There is no such thing as "the one true gaming graphics API"

Windows/Xbox uses Direct3D, Mac/iDevices use Metal, PlayStation uses GNM/GNMX, Nintendo Switch uses NVN.

Compared to DirectX, Vulkan is barely used at all in gaming, except on Linux and the most recent subset of Android devices.

If it weren't for Proton's DirectX emulation running on top on Vulkan, Vulkan would be largely irrelevant, from a "widely used in real world gaming" perspective.

aseipp• 2 years ago

If Apple actually wanted to bring games to MacOS, they'd give Valve and CodeWeavers a billion dollars and get the Steam Client working, put tons of effort into a d3dvk-for-Metal solution, and have it working with as little overhead as possible. They don't do this because they can't funnel them through the App Store. The graphics API they use for the operating isn't relevant (Vulkan isn't used very much for most games.)

The funny thing is that with the rise of Proton and Steam Deck driving its development -- there's literally no reason anymore for a game developer to ever target any "Desktop PC" platform except x86_64 Portable Executable files, using DirectX. Because Proton will just handle the port with nearly zero effort. Even on ARM64 devices like Snapdragon, the CPU is often not the limiting factor (and binary translation is only in the realm of 15%) so emulation for games is totally viable if the GPU can handle the load.

aYsY4dDQ2NrcNzA• 2 years ago

Apple just had to Think Different.

Metal is 1.7 years older than Vulkan. (June 2014 versus February 2016)

Drawing Graphics on Apple Vision with the Metal Rendering API

Related Stories