megadragon9

I worked on this project (https://github.com/workofart/harness-training) for the past few months to reframe "Agent-driven Self-improving Harness" to "Harness Training".

The idea is simple, the harness is trained once with a frozen task LLM against a given task environment. Then you can then swap out the task LLM to any model and evaluate the "frozen trained harness" with any task LLM on any new task environment.

Since this was a general problem, I took the chance to create a general PyTorch-like training framework. Right now, you can train with any OpenAI-compatible API for interfacing with the task LLM and train against Terminal-Bench or SWE-Bench tasks, but you can easily extend it to support any task environments.

I wrote a blog post (https://www.henrypan.com/blog/2026-07-18-harness-training) on this journey, including (but not limited to):

- results from using this harness training framework to improve general capabilities across many task LLMs to beat Terminal Bench 2.0 (Terminus Harness) and also transfer learnings towards better task-solving abilities in unseen task environments (e.g. harness trained on SWE-Bench tasks solving Terminal Bench tasks)

- how this framework is built

- learnings on what was missing in my initial version of the project (hint: determinism)

Interesting project. Do you think manual memory management help understand computational graph lifecycle better, or does it distract from backprop itself?

btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.

Model built and trained using a hand-built deep learning library (numpy primitives)

I'm continuing to expand my own deep learning library [1] built with numpy-primitives to support LLM post-training techniques like supervised fine-tuning (SFT) and reinforcement learning with GRPO. It's a good learning experience to work without all the high-level abstractions to "build a wheel" and "use that wheel to build a car".

I'm also looking into coding harness self-improvement [2]. An inner LLM (raw LLM request) + harness solves coding tasks, an outer agent like Claude or Codex that proposes harness changes. I experimented with many things in the past few months that made me realize this self-improvement thing that everyone is talking about is just an experiment design problem. I wrote about it here [3]. I'm continuing to improve the infra around the self-improvement loop, to increase signal-to-noise ratio per experiment. I'm also generalizing the infra to expand beyond terminal bench tasks and to collect some data across different models (harness-bound vs model-bound).

[1] https://github.com/workofart/ml-by-hand

[2] https://github.com/workofart/harness-experiment

[3] https://www.henrypan.com/blog/2026-05-25-self-improvement-ha...

looks like elon web services (EWS) is the master plan all along :D

I don't think the market will swallow the stock offerings until we see early signs of GDP growth attributable to these entities. But until then, I think the cost is higher than the benefit, which "The dead economy theory" essay covered it well [0]

[0] https://www.owenmcgrann.com/p/the-dead-economy-theory

I recently wanted to see whether an AI agent could self-improve a harness to solve terminal bench tasks. It’s possible for an AI agent to propose a meaningful one-time change to the harness, but after experimenting with this for a couple of weeks, I think the continuous self-improvement is mostly an experiment-systems problem. The system needs a way to decide what kind of improvements can safely compound.

Turns out there's a lot of parallels to coding-agent customization (e.g. SKILLS.md etc..) too.

I wrote my experience of building such system here, including the successful and failure attempts during the process, and how I approached the self-improvement loop. It's not intended as a benchmark claim but more of a systems/research writeup.

https://www.henrypan.com/blog/2026-05-25-self-improvement-ha...

I'm continuing to expand my own deep learning library [1] (PyTorch-clone built with Python and Numpy) to support LLM post-training techniques like supervised fine-tuning (SFT) [2] and reinforcement learning with GRPO [3] . It's a good learning experience to work without all the high-level abstractions to "build a wheel" and "use that wheel to build a car". Post-training results are still cooking, since training on my MacBookPro is quite slow with "unoptimized PyTorch" :)

1. https://github.com/workofart/ml-by-hand

2. https://github.com/workofart/ml-by-hand/blob/main/examples/s...

3. https://github.com/workofart/ml-by-hand/blob/main/examples/g...

Interesting to see more demand shaping mechanisms applied to LLM inference. Even though the "batch processing" feature is already available. I guess this "promotion" is to test the hypothesis of sliding along the spectrum towards more "real-time" demand shaping.

Thanks for sharing! It's inspiring to see more people "reinventing for insight" in the age of AI. This reminds me of my similar previous project a year ago when I built an entire PyTorch-style machine learning library [1] from scratch, using nothing but Python and NumPy. I started with a tiny autograd engine, then gradually created layer modules, optimizers, data loaders etc... I simply wanted to learn machine learning from first principles. Along the way I attempted to reproduce classical convnets [2] all the way to a toy GPT-2 [3] using the library I built. It definitely helped me understand how machine learning worked underneath the hood without all the fancy abstractions that PyTorch/TensorFlow provides. I eventually wrote a blog post [4] of this journey.

[1] https://github.com/workofart/ml-by-hand

[2] https://github.com/workofart/ml-by-hand/blob/main/examples/c...

[3] https://github.com/workofart/ml-by-hand/blob/main/examples/g...

[4] https://www.henrypan.com/blog/2025-02-06-ml-by-hand/

Love the educational value of this "nano-sized" project. This reminded me of the from-scratch project I created to learn about deep learning libraries, neural networks all the way to LLMs like GPT-2 using just Numpy and Python [1]. Learning is done by "re-inventing the wheel" yourself, one step at a time :)

[1] https://github.com/workofart/ml-by-hand

Reminds me of this HN discussion (Writing Code Was Never the Bottleneck): https://news.ycombinator.com/item?id=44429789

Thanks for this inspiring essay, I couldn’t agree more that “reinventing for insight” is one of the best ways to learn. I had a similar experience couple months ago when I built an entire PyTorch-style machine learning library [1] from scratch, using nothing but Python and NumPy. I started with a tiny autograd engine, then gradually created layer modules, optimizers, data loaders etc... I simply wanted to learn machine learning from first principles. Along the way I attempted to reproduce classical convnets [2] all the way to a toy GPT-2 [3] using the library I built. It definitely helped me understand how machine learning worked underneath the hood without all the fancy abstractions that PyTorch/TensorFlow provides. Kinda like reinventing the car using the wheel I reinvented :)

[1] https://github.com/workofart/ml-by-hand

[2] https://github.com/workofart/ml-by-hand/blob/main/examples/c...

[3] https://github.com/workofart/ml-by-hand/blob/main/examples/g...

The blog post is quite informative: https://ai.meta.com/blog/llama-4-multimodal-intelligence/

I built a machine learning library [1] (similar to PyTorch's API) entirely from scratch using only Python and NumPy. It was inspired by Andrej Karpathy's Micrograd project [2]. I slowly added more functionality and evolved it into a fully functional ML library that can build and train classical CNNs [3] to even a toy GPT-2 [4].

I wanted to understand how models learn, like literally bridging the gap between mathematical formulas and high-level API calls. I feel like, as a beginner in machine learning, it's important to strip away the abstractions and understand how these libraries work from the ground up before leveraging these "high-level" libraries such as PyTorch and Tensorflow. Oh I also wrote a blog post [5] on the journey.

[1] https://github.com/workofart/ml-by-hand

[2] https://github.com/karpathy/micrograd

[3] https://github.com/workofart/ml-by-hand/blob/main/examples/c...

[4] https://github.com/workofart/ml-by-hand/blob/main/examples/g...

[5] https://www.henrypan.com/blog/2025-02-06-ml-by-hand/

Writing reveals what you don’t know, what you can’t see when an idea is only held in your head. Biases, blind spots, and assumptions you can’t grasp internally.

I completely agree with this. Often, I think I understand something, but when I try to explain it to others, I quickly realize where my understanding is shaky. The gaps become even more apparent when I attempt to write it down because I have to structure my thoughts logically and precisely. Writing goes a step beyond speaking because it forces me to re-read and refine my ideas, whereas spoken words often disappear without deeper reflection. Oh, even this comment that I'm writing now was edited a couple of times before submitting it. The second half of the comment was added after re-reading the first half.

I would say the demand for this kind of tool definitely exists. Good work! From a rough glance it looks pretty similar to another tool that I've been using https://github.com/mufeedvh/code2prompt

uploaded 00's geocities https://geocity.ichi.city/

What does the "Tell Friends" column mean in the table? Was it supposed to be a metric or just a placeholder?