Ambix

There were two popular models: Agat 7 and Agat 9. The second one was great machine, one of the best personal computers produced in USSR in large quantitites.

This [1] link is absolutely golden, thanks!

I've used this meta framework for LLM tuning, it really one of the best out there.

Still my default tool to install on any remote server. My secret weapon :)

Hehe, great post. Just use Hono or Hono X. It's so easier to reason about what's going on for me, mostly backend guy.

Wow, so cool retro designed blog there! For those who kinda liked window system of first Mac generations :)

This so cool! Would love to have some time to train myself with those exercises more

On acoustic with light or medium strings - it's OK. I used to flatten them with nail file from time to time. But it might be much easier on your finger tips just to start with electric and then progress towards acoustic.

I did my own experiments and it looks like (surprisingly) Q4KM models often outperforms Q6 and Q8 quantised models.

For bigger models (in range of 8B - 70B) the Q4KM is very good, there are no any degradation compared to full FP16 models.

LLMs are very bad at interpreting Markdown tables

Which table format is better for LLMs? Do you have some insights there?

I did started learning 6502 assembler back in those days :)

Recently I have been busy writing the emulator in Golang:

https://github.com/gotzmann/6502

Yes, that's what I've seen from a lot of my experiments with fine-tuning. One should be really careful to not "lobotomize" already capable model and achieve better results at the end. It's trickier than seems from multiple of tutorials.

But I believe that most of the data stored in foundation models are just useless for some particular domain. So it's better to forget something, getting really useful info instead.

Try to use less cores. RAM bandwidth is real limiting factor there, so there always some sweet spot between CPU cores and RAM bandwidth for individual system.

For example, I use only 6 cores from 10 on my M1 Pro laptop.

Both 1 and 2 versions are good enough in Russian even for some real use in production environments. But you should tolerate some crazy / weird typos and mis-wording they'll produce. Russian is complicated.

Is it possible to write llama.cpp in Futhark? Like do effective math manipulations on 4-bit vectors within GPU?

OMG you do not need this bloatload for just PG hosting. Just harden SSH, harden PG configs and voila :)

Yep, I'd always prefer freedom and power of hosting my own PG instance upon a some robust VM offering to guiggling with clumsy AWS menus.

Token is basically a number. Embedding is a vector consisting of those numbers (tokens).

Tokens are just integer numbers, showing their position in the big vocabulary - it's that simple :)

And vocabulary is just an array / vector / list - it depends which programming language you use, each has each own terminology for that data structure.

For example LLaMA vocabulary has 32,000 tokens.

You might reuse simple LLaMA tokenizer right in your Go code, look there:

https://github.com/gotzmann/llama.go/blob/8cc54ca81e6bfbce25...

For those who interested, there some new researches in the field [0]. It usually possible to create more compact token representation from given text, but my guess the greedy "optimal" tokenizer might harm the performance of the model?

[0] https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tok...

I think the same. LLMs are actually sort of "multi-linguas", able to transform source of any language to internal representation and then do output in some other language, thanks to so many layers of neurons inside it.

Intel and AMD might be fast on the paper... but the RAM bandwidth matter much more now (thing of gen AI applications) than raw CPU power. So Apple Silicon is really the only CPU doing crazy 200 / 400 / 800 GB/s on the consumer market.

Wow, looks like fresh idea and will be useful for me! Will wait for new releases

From what I know nobody in China cares about English and that's a nation like 2-3x bigger than all population of English-speaking countries :) Can't see a demise of Chinese even after 1,000 years. As well other languages with big enough user base (I'm not speaking about smallish European countries)

Go see yourself :)

https://github.com/ggerganov/llama.cpp/blob/master/examples/...

There's too many schemes right now with 4_0 and 5_1 really popular between LLM geeks.

Not actually, you might see here how bigger models have much worse perplexity with 4bit due to the weight outliers:

https://github.com/saharNooby/rwkv.cpp/issues/12

For LLaMA models - yeah, different story.

Not sure here. The LLaMA models - yes, all weights fit in the small range between -2.0 .. 2.0

And some other models have more crazy numbers with even more crazier outliers within them, like you might have a weight of 12.00 between long array of typical small numbers around 0.00

I've read story about attempt to quantize RWKV model into the 4/5 bits which failed short due to the presence of outlier weights.

The author told somewhere that bigger models had worse perplexity because of this.

No need to convert models, 4bit LLaMA versions for GGML v2 available here:

https://huggingface.co/gotzmann/LLaMA-GGML-v2/tree/main

Looks like ADHD person with high IQ and curiosity (like me). You could cure the problem with medicine or just become better managing it with age. From my experience, eventually you'd become more selective with your ideas and more involved into really finishing them when having long term plan and returning to implementation from time to time. But it might took some years for ambitious projects :)