mplappert

“Never attribute to malice that which is adequately explained by stupidity.” (or carelessness in this case)

Seems like a hallucination to me; note that the context contains “unmarkBlock” as the function name, which invites a connection to Minecraft. Still shouldn’t happen of course.

The alternative explanation is that the inference engine, which batches several unrelated requests for parallel processing, messed up the unpacking and returned an unrelated user’s query. This one would be very scary as it will leak arbitrary content, but it seems much less likely here.

I agree with this. I recently went back to writing more code by hand again for this reason: the amount of effort / mental toll it takes me to steer the LLM is sometime just not worth it. I might be slower but I enjoy my work much more again.

There’s a lot of confusion here:

- There is no double taxation if you just pay yourself a salary (since it’s a normal business expense). If you want to take money out of the company flexibly, a GmbH is the wrong structure.

- I’ve never heard of anybody doing an UG/GmbH + KG to get started. This is highly unusual. Most people either do just a simple UG or maybe they set up a holding structure with two separate GmbH / UG entities.

- Related to the above: if you go with a simple, standard structure you will incur minimal legal fees. You don’t need a lawyer, you just directly task a notary and tell them you want a standard setup.

- If you don’t want the complexity of a limited liability company, the standard way to reduce liability risk is to get liability insurance. Many, many people do this instead of having a GmbH.

The valid criticism is the a) lack of digital processes and b) sequential processing of steps that could happen in parallel. For example, I sped up my own GmbH process by driving to the register court and paying in cash on-site. For whatever reason that’s much faster and saves about a week.

Yeah I tried to use ROS / Gazebo about 10 years ago (it was still ROS 1) and getting it set up was an immense pain. I remember that creating new modules required writing CMake files. Maybe this is better now, but I decided to skip that.

I also agree on the need for a simple, easy, extensible open source framework. LeRobot IMO is some of this but also contains the dataset + ML code. I think Flask is nice because it's so singularly focused on just one thing with extensibility if you need extras.

So I really like the idea. But being an OSS maintainer these days seems... intense.

I’d recommend looking at the SO-101 then. Much much cheaper.

And hardware is very fun! It’s also very frustrating. But to me worth it.

This all makes a lot sense, thanks for sharing!

For depth I agree on the VLA route but for ACT / DP-style imitation learning from scratch it seems more feasible (since you’re not fighting a pretrained model that was not trained on this modality). Might also increase robustness since you naturally end up with an input that’s invariant to colors / textures. Plan is to try both paths: the from scratch (and then ablate RGB vs RGB-D) and the VLA + fine-tuning one.

Did you ever a late RGB vs RGB-D? I haven’t seen that much in the literature.

Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.

Yeah I can see that. Especially the rotation part is pretty awkward (hence why I currently do the axis parallel to table surface trick and only use the yaw angle to rotate the gripper). The space mouse was super easy to get started with though, so in terms of getting something up and running quickly it has an advantage.

This seems indeed useful. I haven't seen this for robotics but I'm sure people need this for larger deployments (either for a distributed fleet or for an "arm farm"-like setup where there's many robots in one location for data collection / eval). Interesting idea!

Wow very cool project and thanks for the Ariel pointer! Added to my reading list.

And yes, having a nice robot makes life so much easier. Very happy with my choice so far!

Oh very cool! Looks a bit like the TRLC-DK1 (I was looking at this one for a bit).

I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)

I timestamp everything twice: once with the hardware clock (if available, like for the realsense camera) and once within my robot stack once it gets read from the device (using `time.monotonic_ns()`). Both are stored and alignment can happen with either timestamp. I think the 2nd timestamp is actually more meaningful since ultimately I want to reconstruct the state that the policy would've seen; so if one modality is delayed I should actually include that effect during training.

That being said, I might switch to a realsense for the static tabletop camera as well; the realsense wrist is clearly much more reliable than the cheap Logitech C920 that I currently use.

I'm only starting down this road but my sense is that ACT and Diffusion Policy both make it pretty feasible to start on real data only. LeRobot also makes it easier to train these. But that's the next step that I'm working on, so I don't know yet.

On attachments: during this project I really wanted a 3D printer several times. So that's probably next on the shopping list.

The Ufactory arm is actually quite extensible: it exposes digital input/output and you have a standard wrist mount where you can mount different end effectors or attachments.

Cool video! What made you stop if I can ask?

And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rubik's cube via teleop.

I do think simpler hardware like the one described in my post works really though and it's so much easier to do something with it.

Very cool!

Have you seen the recent nvidia thing? They do this at scale for robotics manipulation: https://research.nvidia.com/labs/gear/enpire/

Great points and I very much appreciate the input!

Right now the static camera is probably really bad: It's mounted on my desk, so its very easy to bump into it and move it. So yeah, it's position for sure will change over time. I think I need a better solution, maybe a rail system that's more rigidly attached to the robot arm so that at least the camera stays fixed relative to that point of reference.

That was my take 8 years ago glad to hear it’s still that

For understanding: I think the level is much deeper if I wrote the code vs reading someone else’s. Same applies to coding agents of course which is why I wrote most of it myself and only delegate some tasks (for example codex was great help at setting up telemetry dashboards or writing the custom glfw renderer).

On control: LeRobot will change all the time and I’ll be unaware of what changed. If something suddenly doesn’t work anymore, it’s a pain to find out. I can of course fork or pin but that defeats the purpose a bit.

At the end it’s also partially just preference: I wanted to write this layer myself, and I have opinions about how it should be architected, so I did.

Glad you like it!

Re your questions:

- The driver situation turned out totally fine; I intentionally picked HW with good python sdk support so that was very painless.

- The static camera (the C920) is not super great; it drops frames and sometimes cuts out. We’ll see how that goes but it’s probably the clostest thing I want to swap right now. Another issue is reach of the arm when forcing the worst to be axis parallel with the table; you cannot get very far away. The chess setup demo in the video gives an example: I can just reach the row of pawns and beyond that it’s out of reach.

- I don’t know yet! The 50-100 figure comes from the ACT and diffusion policy papers but it depends on the type of task. For fine tuning my sense is that you only need a few hours worth of demos to get good results with pi0.5 etc. a big reason I’m doing this project is that I want to try all of this myself, so the next posts definitely will talk about that

Thanks Tim and fun seeing you here :)

Looks very cool! I’m not a huge discord user but how about you shoot me an email and we can figure out how to share notes? (I don’t want to post it directly here but it’s easy to find on my personal website, just google my name)

Re why not SO-101: the article has a footnote about this; I actually bought the SO-101 as well! I want to integrate it into the same setup so I can switch depending on task.

Somewhat surprisingly the xarm was actually much faster to arrive; I got it within 2 days of ordering. I don’t have a 3D printer and getting the SO-101 from the vendor I ordered it at took almost 4 weeks. So partially it just came down to what I had access to more quickly.

Second point is reliability: I think the SO-101 is cool but I’d be surprised if it doesn’t break more quickly than the xarm. I wanted something that’s going to last a long time without headaches. And these industrial arms are really mature hardware wise now.

Hope this helps!

It’s partially my fault I currently clip the max speed _and_ I only input soft control changes when teleoperating to avoid crashing into things. The robot itself could definitely move more quickly than what you see in the video.

It would be interesting to explore how RL can be applied on top of my (flawed) human demos to optimize beyond what I’m able to do.

I think there‘s a serious issue with people using AI to do an immense amount of busywork and then never look at it again. Colossal waste.

I very much agree; I think laziness / friction is basically a critically important regularizer for what to build and for what to not build. LLMs remove that friction and it requires more discipline now. (Wrote some of this up a while ago here: https://matthiasplappert.com/blog/2026/laziness-in-the-age-o...)

Where’s the data flywheel exactly?

Yeah those are demos. I think we‘re pretty far away from this becoming a real thing. I wrote up why here: https://matthiasplappert.com/blog/2026/humanoid-robot-in-the...

That seems like quite an extrapolation and an extraordinary statement. This is a single task, in a lab setting. What your describing are extremely open-ended tasks in people’s homes.

What is informing these timelines?