vibe42

I'm using the pi-mono coding agent (open source, free) without any extensions and very simple prompts. The 3.6 27B model (BF16, 250k context) uses 67GB VRAM on an RTX PRO 9000.

It's very capable on almost any coding task I've thrown at it, and very good for easy-to-medium hard scripts, new code bases.

It struggles on some complex tasks in larger code bases, e.g. using to debug and fix bugs in llama.cpp it gets close to working code but often introduces errors. For such tasks its still very useful as a search/explore tool and drafting fixes.

It can indeed cause some models to try too hard to come up stuff, but the next verification prompt does counteract it.

E.g. some findings first classified as moderate priority often get reclassified as low priority even if the finding itself is correct.

The exact phrasing doesn't seem to matter as much as keeping the prompts short, simple and to the point.

However some models seem to do a bit better when adding ", if any" to prompts such as "List potential improvements".

Something I've had good progress with using local models and simple open-source harnesses is to repeat, in a new context, simple verification prompts.

I'd run the following 5-10 times with one model, then again with a 2nd model.

"Verify the correctness and completeness of all security configs/rules in SETUP.md. Consider if anything is missing, and if anything is not needed. Do not modify any files; only write potential findings to report.txt"

"Verify all findings and claims in report.txt."

Replace "SETUP.md" with whatever you're working on.

It's both terrifying and incredible watching what the models get correct and what they get completely wrong.

However, after enough runs they tend to settle on a state they claim does not need any more edits. And that result is generally useful with much fewer errors/hallucinations compared to a single run.

Many crypto wallets use a key derivation function (KDF) to add an amount of computation (and memory usage) per password tried - to mitigate brute force of weak passwords.

The increase in compute (decrease in brute-force cost) combined with price increases in many crypto tokens means brute-forcing old wallets can become worth it years after passwords were forgotten.

And of course even smaller, local AI models can now easily write optimized scripts to brute-force any given KDF function.

I run both MoE and dense models on laptops.

One set of models run on 8GB VRAM / 16GB RAM and another set runs on 24GB VRAM / 64GB RAM. Both are very useful for easy and easy-to-moderate complex code, respectively.

The latest open, small models are incredibly useful even at smaller sizes when configured properly (quant size, sampling params, careful use of context etc).

https://old.reddit.com/r/LocalLLaMA/

Bit of a hype madhouse whenever a new model is released, but it's pretty easy to filter out simple hype from people showing reproducible experiments, specific configs for llama.cpp, github links etc.

Outside Trading.

This. And when possible, first asking the AI to add more granular logging around the code where the problem is - then re-run the code and feed the new log in a new context.

I've used this to debug some moderately complex bugs in golang and godot code and it works really well - the combo of having a new context with the (sometimes overly) granular debug logging and only the required, specific source code.

Keep it simple and run a fresh, new context for each prompt.

I use the pi-mono coding agent with several different new open models running locally.

The simpler and more precise the prompt the better it works. Some examples:

"Review all golang code files in this folder. Look for refactor opportunities that make the code simpler, shorter, easier to understand and easier to maintain, while not changing the logic, correctness or functionality of the code. Do not modify any code; only describe potential refactor changes."

After it lists a bunch of potential changes, it's then enough to write "Implement finding 4. XYZ" and sometimes add "Do not make any other changes" to keep the resulting agent actions focused.

With the pi-mono coding agent (running local, open models) this works very well:

"Do not modify any code; only describe potential changes."

I often add it to the end when prompting to e.g. review code for potential optimizations or refactor changes.

Q4-Q5 quants of this model runs well on gaming laptops with 24GB VRAM and 64GB RAM. Can get one of those for around $3,500.

Interesting pros/cons vs the new Macbook Pros depending on your prefs.

And Linux runs better than ever on such machines.

Both are possible; increasing demand and bubble collapse.

The way this could happen is if model commoditization increases - e.g. some AI labs keep publishing large open models that increasingly close the gap to the closed frontier models.

Also, if consumer hardware keep getting better and models get so good that most people can get most of their usage satisfied by smaller models running on their laptop, they won't pay a ton for large frontier models.

That's a very good point! And an opportunity for game worlds; extrapolate those blind spots by assuming small planets and planets further out from their stars are more common than what's been confirmed so far.

Thanks for the link! Looks like pretty useful tools.

I'm playing with a space game idea of physics simulation somewhere between the fidelity of KSP and Eve Online. More robust ships and easier gameplay than KSP, but much more in-depth physics than Eve.

A bit too early (and too much AI slop code!) to share but can push to github if useful - wrote some scripts to parse the gaia DR3 release: https://gea.esac.esa.int/archive/

Parses all rows of the gdr3/Astrophysical_parameters/ files and filters out all objects within X ly of our solar system.

Same idea there as with the exoplanets; build a statistical distribution from real-world data and use it to generate fictional solar systems.

Their latest open models are pretty competitive with other open models, and some innovation around the smaller sizes (2-4 GB).

They're helping close to the distance to realistic quality inference on phones and other smaller devices.

The pics of the cooling system is pretty good sci-fi / cyberpunk / steampunk inspo.

If the whole AI bubble spectularly collapes, at least we got a lot of cool pics of custom hardware!

Training their own, closed, internal models on their own data sets? Probably a good way to squeeze out some market trading signals.

Something related and fun is parsing a simple CSV file of exoplanets.

https://exoplanetarchive.ipac.caltech.edu/cgi-bin/TblView/np...

Download Table -> All Columns, All Rows.

Tried a few new, open, local AI models by giving them the CSV file and asking them to write a simple python script:

1. Parse all rows and build statistical distribution of mass, radius etc.

2. Use those distributions to generate fictional exoplanets.

Playing with this for a space game idea where star systems are populated with fictional exoplanets, but all their params are from the real statistical distributions of all known exoplanets.

A way to get some harder sci-fi using real world data :)

A parcel of land.

A few robot legs and arms, big battery, off-the-shelf GPU. Solar panels.

Prompt: "Take care of all this land within its limits and grow some veggies."

If Starship puts some kind of fully-fueled, modular stage in LEO, and Orion docks with it, how fast could Orion then fly to and from Mars?

Space Weather still looks calm: https://www.swpc.noaa.gov/

NASA's rendering of the flyby:

https://svs.gsfc.nasa.gov/vis/a000000/a005500/a005536/a2_fly...

Hope we get to see something like this in 4K !

Mild Space Weather: https://www.swpc.noaa.gov/

Moderate geomagnetic storm watch until April 2.

They can move around after they switch from launch to spaceflight config. Apparently they also have some exercise gear for the journey.

Here's a habitable zone in a different star system: https://en.wikipedia.org/wiki/TRAPPIST-1#Habitable_zone

How does this compare to the pi-mono coding agent?

https://github.com/badlogic/pi-mono/tree/main/packages/codin...

With 16 GB VRAM one can run a decent quant (Q4-Q8) of newer, smaller dense models. This leaves room for e.g. 32-256k context size.

This might not be enough to chew through a large code base but for smaller projects it can easily fit enough if not all of the code base to drive a good coding agent.

I don't recommend specific models or model providers due to how much hype and BS there is around benchmarks etc. Easiest is to check the latest generation of open models and look for a dense-type where a decent quant fits within the VRAM.

Some models run fast enough that some of the weights can spill over from VRAM to RAM while maintaining a usable prompt/token gen speed.

Higher-end gaming laptops are still decently priced and work well for local AI inference.

And Linux runs better than ever on them; I'm running debian 13 with almost no driver issues.

For $2k you can get 32 GB DDR5 RAM and 16 GB fast VRAM. Bump the RAM to 64 GB and you're still below $3k.

The main server runs 3x RTX PRO 6000 (288 GB VRAM combined), power limited to 280W each (can crank it up as temps are fine but about to add some more fans first as the cards are stacked).

The second server is 2x Radeon RX 7900 XTX (48 GB VRAM combined). It's a fairly recent gaming PC that's being repurposed. Idea is to power limit those cards too and run some overnight stuff w small/medium sized models.

Intel just released some 32 GB VRAM cards, but sounds like support across AI tooling is a bit rough atm.

Ethereum has a new site for PQ research: https://pq.ethereum.org/