pizzao

A country in decay..

The required compute seems a bit high: "We trained all CIFAR-10 models on 1xB200 GPU, and all ImageNet 64×64 models on 8xB200 GPUs. The largest CIFAR-10 model uses 20 B200 hours to train, and the largest ImageNet 64×64 model uses 640 B200 hours"

20 B200 hours for CIFAR-10 seems like a lot...

I wonder if there is way local small LLMs can complement each other in away that the sum-total yields a much more performant LLM

Can someone explain to me what is their "prompting-scaffolding" to make it work ?