HNGlance

Top Best New Show

BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model

https://arxiv.org/abs/2309.11568

by jwan584 • 3 years ago

3 2 3 years ago

AR

Related Stories

Loading related stories...

Comments

2

jwan584• 3 years ago

A helpful paper with the full recipe Cerebras uses to train LLMs and their process including: - Extensively deduplicated dataset (SlimPajama) - Hyperparameter search using muP - Variable sequence length training + ALiBi - Aggressive LR decay

version_five• 3 years ago

When models are released like this, it would be great to do it with a PR to ggml/llama.cpp giving support, or use a format that's already supported. Imo if I'm choosing between a 3B and a 7B, I'm using it in an edge or local model and I don't want HF/pytorch. It would be easier to evaluate and rank higher in things to consider if I could easily get it into llama.cpp.

Source preview

arxiv.org

HNGlance

© 2024-2026 HN Glance. Story content belongs to its original sources.

Privacy
Terms