rozgo

Declaring war on big models and bloated RL. Im training autonomous robots/vehicles by building tiny world models that are trained in 5-30 mins on a single GPU, with less than 50M params.

World model for AI agents. Doing process mining of missions, operations and logistics to transform them into digital twins. AI agents can then leverage these digital twins as world models for control or prediction.

I develop simulators for digital twins and games. Currently working on a simulator for LLMs to use as world models.

It could be the language. Almost 100% of my code is written by AI, I do supervise as it creates and steer in the right direction. I configure the code agents with examples of all frameworks Im using. My choice of Rust might be disproportionately providing better results, because cargo, the expected code structure, examples, docs, and error messages, are so well thought out in Rust, that the coding agents can really get very far. I work on 2-3 projects at once, cycling through them supervising their work. Most of my work is simulation, physics and complex robotics frameworks. It works for me.

Working on "context engineering" for coding agents. Specifically for complex dev environments and targets, like robotics, digital twins and games. Been able to witness agents go from 100% failure rate to contributing nearly 90% of the plumbing code. I'm helping agents understand how to use simulators and game engines; configure, build and deploy DevOps/MLOps pipelines.

panoc, fast, efficient, good for real-time, embedded, and adaptive control.

I’ll check what I can dig up for Rust examples.

Yang, this is really well done. I work with Rust and robotics all day, and would have never imagine seeing all this in one file. It is a great learning tool. If you complete the MPC controller it can also be a great learning tool for training AI pilots.

We ask it to predict, and in doing so it sometimes creates a model of the world it can use to "think" what comes next. In forward pass.

Yoneda Lemma, an object is entirely determined by its relationships to other objects.

That’s exactly right. We, just another games and simulation studio, are implementing LLMs in several games and engines. For dialog systems, hero journeys, behavior trees and techart pipelines. Most do not even expose a chatbot to the player, LLMs are also used for game systems to chatter among themselves.

6mo+ is spot on. This is true for most dev work that deviates from the CRUD path. Like gaming, simulation, robotics. We might as well use the time to train them in Rust too.

In startups and projects where Rust is a premature optimization this makes sense. But, some startups and projects are all about the competitive advantage created by optimizing from day one. In these cases, choosing Rust and other early optimizations is the main enabler of a unique product.

This works well. Starting with bigger models allows us to explore whats possible. I find it easier to scale down from here.

Built in Rust tells me a certain bar has been reached. Tells me the team went through a lot of effort to do their best. And the mature tooling makes it easy for me to evaluate and confirm my assumptions. So yes, positive marketing in my case.

In our game studio, engineers are creating lots of developer art on their own. But the real productivity booster is coming from artists using language models to generate entire art pipeline scripts. Several Python scripts to automate Blender3D and offline asset post-processing. Many artists are also changing shaders by asking language models to modify existing code.

Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more.

Good job!

Would it be possible to integrate something like bevy_engine as a plugin? For live or low-coding experience and tools.

You should consider surrounding yourself with people that can tell the story. It's common to not be able or willing to tell your own story. Others can carry or lift this weight for you, and take it where it needs to go.

When your AI models operate on a vectorized space, then yes. Pipelines from real-to-vectorspace and synthetic-to-vectorspace can be extremely indistinguishable from each other. Basically a video game. This makes training on synthetic data extremely effective, if not for real inference, at the very least for transfer and curriculum learning.

We’ve been able to build a mobile/web hybrid ride-sharing app using Yew/GraphQL. A robot Lego-like 3D builder, with Yew interop with ThreeJS. And a full 3D web game with Yew hosting a Bevy game. Everything packed with trunk.

It’s still a bit rough around the edges but the Rust benefits more than makes up for it.

Usually not an issue with very interactive apps. We have a similar workflow targeting web and desktop, but Rust + WASM. Using BevyEngine for real-time apps, and Yew for web apps.

No. Pretty messy in-house solution.

20 minutes of video animation for the cost of 5 minutes.

Training simulated robots instead of iterating with real hardware.

Got the chance of running the examples and glance over the code. Also checked the book and docs. As I mentioned in another post about Bevy, it looks incredibly elegant and clean. Almost feels like this could make engine development fun again. And custom engines feasible.

Contrary to popular belief, the need to build your own engine is real for a significant amount of projects. I look forward to being able to quickly put together a few building blocks, customize the renderer and experiment with new systems.

This is absolutely the case for programming languages. From Haskell to Rust. Programming languages shape the way we think and reason about the problem. Sometimes it does this so well, it leaves us starring into the abyss. Facing dead on the problem domain—after all sides quests have been removed. Once you can delegate away all the busy work, you can focus on bigger and better abstractions.

2. "you're kind of like a video game." I think this is a fair characterization. It's obviously not a video game, but there is enormous overlap, once you've successfully modeled your perception in vector space.

This is actually my entire strategy for approaching autonomy, I make as much of the problem space a game; just a serious one. And the most successful optimizations and solutions are the ones that adopt techniques from games. Many academic solutions are formalizations for old-school game "hacks".

In my line of work, training, in many occasions, is about making deep computer vision match ground truth, generated by game tech.

Thinking with your code can be an effective learning tool. You can learn how to offset more and more abstraction layers to your code and focus on higher level capabilities. When your code becomes a thinking tool, you will pay more attention to the languages you use, and start requiring more from them; as they will shape your thinking.

Initially, ML languages did this for me. I now use Rust in a similar fashion. Even when I have to program in languages that require heavy cognitive load, like C++ and JS, I can leverage what I've learn in my preferred languages. Program defensively (maybe slower?), design with types, and still be able to build highly complex software while managing complexity.

Control theory for fun can be really fun. Using control systems for solving problems you are familiar with can help develop an intuition, that you can then map to other problem domains. Years ago a peer at LucasArts taught me how to use PIDs for game cameras, and I've used them everywhere since.

Very simple rules can create lifelike emergent behavior. This is an example of 2 simple motors using PID to try to target a red dot, in 3D, with gravity on: https://www.youtube.com/watch?v=8hfKs02QHR4&list=PLpDgOnZpfh...

I can watch control systems all day.

Apple II, TRS-80. Quake engine. Haskell, F#, Rust. Type theory, domain and test driven design. Philosophy, economy. Linux. Physics engines, PyTorch, GStreamer. Twitter.