bashbjorn

There is an RSS feed now: https://nobodywho.ooo/feed.xml

It's just an eleventy site: https://github.com/nobodywho-ooo/website

No RSS feed currently, but it's a good idea to add one!

Cool technique, but I'm not sure I'd call it simple.

Doing this means that you can't just tokenize the string output of the chat template as one big string. You might need to tokenize things separately, and combine them after.

You're right, there must be a good and simple way to do it.

Obviously the prefix-with-backslash convention won't do it. The escaping system could be something like inserting a character on the second position in the text repr, and reversing that on output too if it matches an escaped known special token.

Changing the vocab on the fly requires tokenizing things separately, breaking the chat template.

Anecdotally, even claude code has an anneurism sometimes when listing special tokens. Idk exactly what claude's <eos> token is, but I'm fairly sure I've seen it stop generation when it tried to generate it before.

I should also say that I've (clearly) not thought about this deeply. There should be a simpler way to do it.

The model sees one token per marker - but the overlap with ingested actual text is still relevant, because the tokenizer will ingest regular text, where it will turn "<|turn>" into the same token.

For this reason, it can be tricky to work on the runtime for a model with the same model. This really feels like an accidental problem, but I'm not sure if it's really solvable without abandoning the text representations altogether (and the jinja abstraction along with it).

Yeah, TheBloke era of local LLMs were good times. TBF Unsloth are doing a fantastic job of publishing quants of the major models quickly - they just don't have nearly the volume of "weird" models as TheBloke did.

I love mistral, but that model is... not the best. Maybe try out Gemma 4 e4b, it's a similar size to Mistral 7B, and should run great on your 4070 ("E4B" is slightly misleading naming).

whoops, my bad. Just a typo in the markdown. Fixed :)

Oh hey I'm also working on a thing to solve the devx of llama.cpp: https://github.com/nobodywho-ooo/nobodywho

In contrast to Ollama, this is a self-contained library, not a server.

I wrote some quick notes on this blogpost, just to jot down how we think about good open-source citizenship: https://www.nobodywho.ai/posts/notes-on-friends-dont-let-fri...

NobodyWho | Copenhagen, Denmark | Software Engineer, ML Engineer, DevRel | Full-time | ONSITE

NobodyWho is making developer tools for running small language models in local-first applications. Our core principle is to ship the model weights along with the application, and then do efficient inference locally and offline, on any device. We run fast on Linux, MacOS, Windows, Android and iOS.

The main product is an inference library that wraps llama.cpp, written in Rust. We provide bindings for Python, Godot (the game engine), and will be releasing a Flutter plugin soon. It's all licensed under EUPL 1.2. Repo here: https://github.com/nobodywho-ooo/nobodywho/

We're hiring people who are comfortable building highly cross-platform FFI applications in Rust (with C++ dependencies), and people who are deeply familiar with language models and the open standards around them, as well as fine-tuning and evaluating models. We're also looking for a technical DevRel profile.

If any of that sounds relevant to you, feel free to email me: a>at<nobodywho.ooo

I made something silly to this effect last year:

https://gitlab.com/AsbjornOlling/nixllm

You provide the prompt to a nix function, and it deterministically generates the source code using llama.cpp.

I'm working on a plugin[1] that runs local LLMs from the Godot game engine. The optimal model sizes seem to be 2B-7B ish, since those will run fast enough on most computers. We recommend that people try it out with Gemma 2 2B (but it will work with any model that works with llama.cpp)

At those sizes, it's great for generating non-repetitive flavortext for NPCs. No more "I took an arrow to the knee".

Models at around the 2B size aren't really capable enough to act a competent adversary - but they are great for something like bargaining with a shopkeeper, or some other role where natural language can let players do a bit more immersive roleplay.

[1] https://github.com/nobodywho-ooo/nobodywho

What makes you say that these are all Stenbergs creations?

Could it be that these are just projects that use libcurl in some way?

I'm having trouble finding any sources that say that Daniel Stenberg actually worked on spotify, utorrent or openttd directly - just to test three of them.

Neat. I wish that I could test the regex immediately in the browser. Tools like regexr.com are great for convincing myself that my regex works.

Python, Typescript, Nim, Rust, C++, C and Elm.

The MNT Reform[1] is a pretty sexy open hardware laptop with a mechanical keyboard, optional trackball, and a small ARM processor.

A cursory glance at geekbench shows that you can get around 50% the performance of a 2020 Macbook Air M1, if you get the upgraded CPU option.[2][3] I have no idea how useful those benchmarks are, though.

[1] https://shop.mntre.com/products/mnt-reform [2] https://browser.geekbench.com/v4/cpu/16291496 [3] https://browser.geekbench.com/v6/cpu/2504172

I did AoC in Nim one year and was very happy with it! I also became infatuated with Nim that year..

I highly recommend it to explore new programming styles in general. As some others, I use it as an excuse to learn a new language each year.

While I think python is a really great language for advent of code, I'm not sure I'd recommend going for an OOP-heavy style. Although that might just be a matter of personal taste - I think OOP is a poor strategy for most problems.

Shameless plug: My project, Cactus Comments[1] does something similar, although for commenting, instead of support chat.

We ship a web-embeddable Matrix client. The users' browser connects directly with a Matrix server of their choice (or a default one).

[1] https://cactus.chat

Hey, thanks! It's a small web I guess.

Aforementionend friend and Cactus Comments dev here.

We don't support any sort of threading yet, although Cerulean-style threading is definitely somewhere down the road. Although stuff like redactions and emoji reactions are a higher priority right now.

We're also keeping our eyes out for the upcoming spaces stuff. That might be useful for grouping comment sections.

There is one bundled with a standard nim installation, and I use it regularly (when working in nim). It works, but it's not very good.

I expect the developers to know this, since its accessible only though the `$ nim secret` command.

I'm currently studying a CS(-ish) bachelors degree (and I'm currently procrastinating on a group assignment) so I have some experience with this.

Our uni gives us quite a lot of group assignments. The university-wide solution to this problem is to include a mandatory section in each group assignment, where we are supposed describe the specific contributions of individual groupmembers. This is a neat idea in theory, but it doesn't really work. Because we've almost always lied in this section.

You'd think that the more productive student would want to claim their own work, but more often we've prioritized 'sticking together', and aimed for distributing the described workload as evenly as possible. The idea being that the highest grade wouldn't be higher anyway (the report doesn't magically become better), but it might give the grader incentive to give some group members a lower grade.

It's not a zero-sum game. So we might as well sacrifice some pride to keep the group dynamic strong, and to help out weaker students (regardless of whether it's a work-ethic or an intelligence problem).

That being said - we might be an unusual social group. We generally prioritize technical (and social) abilities way higher than grades (which aren't so important here anyway). We've all stuck together since the start of our studies, despite being a very wide span of academic ability - so theres a lot of teaching and learning within the group.

A survey of our uni recently ranked our line of study #1 for "How motivating the social life is to performing academically". So it seems that our approach is working.

EDIT: I should mention that it's by no means the case that the actual workloads are distributed evenly. They are of course very skewed, and often a single group member will do almost no work.

Agreed. Although I'd much sooner recommend just using Mopidy. Mopidy supports spotify and many other music services, and has a number of low-effort ways of controlling playback programmatically.

But for sure the interesting part of this is the method, not the purpose.

This is neat and all, but their widget.js file is 888KB.

For comparison, Bootstrap is 180KB, Elm is 29KB and Vue.js is 100KB.

It might not be much for sites otherwise dealing with large multimedia assets, but for anyone going for lean / fast web development, this seems to be a no-go.