rslice

It takes time for older paradigms to be plodded through. I think the community is still sort of dizzy / dazzled from the novelty of the medium, and perhaps to some extent kept back by VR thinking and object/static 3d model type thinking. As developers and users' Overton windows shifts, and they begin to think of these glasses as something totally normal and even quite boring, they might develop the orientation of mind necessary to hone in on use cases that properly _fit_ what this medium can offer.

It took me a year of confusion and throwing darts in the void before I finally began to see what this form of computing might represent. And that gain of clarity happened _thanks_ to deeply synergetic advancements in AI model capabilities and more recently with agentic coding which freed up the bandwidth to operate on higher level interaction and geometric primitives.

AR in isolation is nearly useless, but in conjunction with other domains and the environment it begins to make sense.

In my opinion, the ideal environment to nurture AR development looks a lot like Bret Victor's Seeing spaces. A lab, tools, devices, surfaces, combined with spontaneous, open-ended, and somewhat continual interaction and learning. https://worrydream.com/SeeingSpaces/

I think you need to shake up paradigms a bit in order to get true value from these glasses. You genuinely need to start viewing scene understanding abilities, frontier AI model API calls, and seamless hand tracking as core primitives that you combine thoughtfully and not new tech that you chain together hoping to land on a use case.

Now, the ecosystem and the engine so far were designed for lower stake experiences but to make the purchase of this device worthwhile for average users, they will need Vision Pro app depth of thinking and attention to detail while fitting into the far tighter software and hardware constraints.

So the key for them is to lower the skill ceiling so creatives can tap into more of the juice, which is why they've announced an Unity to Lens Studio bridge and closed loop agentic coding, while making it more attractive for seasoned developers to use their engine and push its capabilities(native c++ sdk).

It takes quite a lot, but from what I've seen over a year developing on their kit and watching their platform evolve, their strategy has sound fundamentals. But they urgently need to tap into a broader and more diverse talent pool, like say Blender and Houdini artists, harware & robotics engineers, ML engineers, even music producers and sort of corral their attention into their platform long enough for truly novel and useful applications to emerge.

I think the biggest constraint so far has been on the computer vision side. With a better processor, the glasses should be able to run real time 6dof object tracking, lower latency hand tracking potentially with better occlusion, and ideally OS-level subtle intent recognition since current air pinch based UI interactions will not make it into mainstream. I'm curious though as to whether they've expanded the ONNX model compatibility. In my experience with the '24 dev kit, this was a serious blocking point in doing any serious custom CV work.

They've also announced a C++ native dev kit, as until now you could only use JS and TS withno node libraries. I think this specific update might have an outsized downstream impact on the ecosystem.

The implementation follows from the 2024 paper “Visual simulation of opal using bond percolation through the weighted Voronoi diagram and the Ewald construction” by Soma Yokota and Issei Fujishiro.

I also wrote an article on opals an more broadly photonic crystals: https://armandsumo.com/posts/opals/

It's not because something can be done that it must be done.

Your substack is a treasure trove. Makes lesswrong articles look mentally rigid.

I can recommend this presentation by Joscha Bach on the subject:

Virtualism as a Perspective on Consciousness, https://www.youtube.com/watch?v=b6oekXIQ-LM

It honors the richness and complexity of conscious experience and provides a refreshing computational perspective on it.

The most beautiful one I've read is from 'The Gentle Seduction' by Marc Stiegler [0] It's a wonderful short read.

[0]http://www.skyhunter.com/marcs/GentleSeduction.html

Perhaps GPT-like models are already capable enough to do math, but they need to store what we call mathematical reasoning as one of many distinct processing pathway and tap into it whenever the context is appropriate.

Easy to say obviously but there's some promising work in this direction[1]

[1]Tracr: Compiled Transformers as a Laboratory for Interpretability, https://arxiv.org/pdf/2301.05062.pdf

deleted

I strongly recommend this presentation: "John Baez: "Symmetric Monoidal Categories A Rosetta Stone"[0]. It is easy to get lost in the ocean of theorems and definitions and overlook some powerful core concepts.

[0]https://www.youtube.com/watch?v=DAGJw7YBy8E

Just tested it on a Samsung Galaxy phone and there does seem to be an issue with longer audio files. I'm working on it, in the meantime I've added settings to reduce computational load particularly for smartphones.

It should be fixed now.

I would love to work on something like that, I have a couple ideas regarding ways to implement it.

However I don't know if I have the time to do it, as there are so many concepts I'm trying to channel through code. Hopefully as I build more computational tools I will increase my bandwidth.

Providing visual tools for phonetic assistance is definitely something I've had in mind for a while and solving the dataset building problem should solve that along the way.

As of now, you can: 1) set the amount_sphere_tube slider to 1 2) decrease the playback_rate to 0.1(in the GUI on the right) 3) on the media controller, go to options(the naming might vary depending on the browser) and set the playback rate to normal or x1. This should give you maximal temporal resolution.

One way features can be further highlighted by generating embeddings from magenta ddsp[0]. Speech sounds are fairly complex though so I don't know how models built on music data would generalize to them. I think tech to do is is there, but for the time being it seems to be scattered around fairly siloed fields. I also tried to use live voice recordings but there are latency issues with the subset of the Web Audio API I'm currently using. However there definitely is value in having a live, spatial feedback of pronunciation.

[0]https://github.com/magenta/magenta-js/tree/master/music#ddsp

Thanks! Glad you're enjoying it!

I agree. As of now, the extra dimensions are redundant.

However they are still somewhat useful. By using distance functions I localize sound sources in space but I also turn them into symbols. This helps me keep track of them and I plan on building a set of rules (grammar) for how these symbols can interact.

Now, building the semantically relevant 'spatial symbols' from data is where the real challenge is, and the first step is to actually gather such data. Unfortunately I don't have access to a photogrammetry setup so all I can do is wait for companies/research institutes to make appropriate data accessible. The alternative is to generate the data synthetically, but you hit a procedural audio generation wall.

lifting with a kernel

I am not familiar enough with the lifting trick to know whether or not it is relevant to this context, which is that of 'embodying' sounds, and not of classifying them. I think it would be silly to think the 3d space would be sufficient to perform sound source separation and/or music transcription. If I were to add those features I would definitely use existing models(neural nets), which properly leverage much higher dimensional spaces.

By building symmetries and lifting low-dimensional data into higher dimensional spaces(Here 3D), you can better leverage the human perceptual system. You also unlock more intuitive modes of interaction, that are closer to the real-world processes that originated the sounds.

Thank you for your thoughtful comment! This is by no means a product, it's more of a way for me to test an idea and share it with the community.

This demo is meant for small audio samples. My initial goal was to use it to visualize and compare drum samples by looking at their 'spatial signature'.

Right now, I'm using arbitrarily defined 'shapes' (sphere and tube) but the goal is to recover those from real-world data. Unfortunately, building the appropriate data-set and the model to go with it is currently out of my reach but I know that's how my brain sort of learned these audio/spatial associations.

In order for this to work on complete tracks, I would need to add source separation, transcription and some form of information compression.

Additionally, I'm working on ways to deal with richer sounds, by laying them out in space or by splitting them into 'voices of unison'. Here's a demo of what it would look like: https://imgur.com/gallery/gkFPXXu

There are two directions I could take this, either making music from interacting with spatial representations or building spatial representations from music. I don't have the bandwidth to do both alone so I would love to work on it with other developers.

Thanks for the feedback, I'm investigating issues on Linux. Try opening the website on a smartphone.

In the repository https://github.com/a-sumo/hyperstep#organic-drums-through-ag..., I explain how one can generate drums by framing them as agent locomotion.

Moreover, music and dancing are two sides of the same coin and in styles such as hip hop and popping you have 'beat-killers'[0] who basically represent music through their motion.

If you build an appropriate spatial language, you can compose music from dancing or you can build a form of generalized dancing. I've explored this concept artistically[1],[2] and I'm currently working on formalizing it and making it computational.

[0] https://www.youtube.com/watch?v=IeNn4XG0QHo [1]https://imgur.com/gallery/pfrvIot [2]https://imgur.com/gallery/MoDVr6v

I am working on using audio worklets to perform the audio processing without blocking the main UI thread, but support on browsers other than Chrome is still limited.

Try loading smaller samples or audio snippets.