Fine tune LLAMA3 on million scale dataset in consumer GPU using QLora, DeepSpeed

https://medium.com/@sumandas0/fine-tune-llama3-on-million-scale-dataset-in-consumer-gpu-using-qlora-deepspeed-3ae8ad75299a

by mehulashah • 2 years ago

145 26 2 years ago

medium.com

unraveller2• 2 years ago

This is a thorough "how to" but it is missing a "why for" about any of the chosen starting elements.

I don't understand why you would use an old dataset that worked for llama2 and just fine-tune llama3 on it. Isn't it most likely that the new model has covered off everything it missed last time around and now the last dataset is only valuable for the last gen.

blackoil• 2 years ago

Dataset may not be public. All large companies have millions of internal documents. Internal LLM can be trained on them.

bradfox23• 2 years ago

Qlora won't work well to add knowledge via private data.

Parameter efficient methods are not useful for these cases at the 8b scale without a more complex training procedure that periodically merges back adapters. Maybe at the 70B scale.

factorymoo• 2 years ago

This might be an unfair statement but it really feels like all of these blogs don't know why. They copy/paste each other (you often seem the same errors in multiple notebooks/blogs) and I have a feeling no one really deeply understands what they're doing.

jackblemming• 2 years ago

Most of the entire field of machine learning is “try shit and see what works”. So it seems like they’re par for the course.

unraveller2• 2 years ago

Found my answer for why thanks to the issues in latest dolphin fine-tune. They do these types of fine tunes mainly to reduce refusal rates and increase intelligence. They did the knee-jerk rerun of the same old data this time, as I suspected, just for lols to see where open-source is at.

Spoiler alert, fine-tunes won't be better until the data quality is better than meta's instruction fine-tune. Give it some weeks.

Why does [doplin-l3-8B] perform substantially worse in some tests?

Essentially, it's trained like this:

  LLama-3-8B-base_model --> LLama-3-8B-Instruct
  LLama-3-8B-base_model --> dolphin-2.9-llama3-8B

And not like this:

  LLama-3-8B-Instruct --> dolphin-2.9-llama3-8B

https://huggingface.co/cognitivecomputations/dolphin-2.9-lla...

ijk• 2 years ago

One additional problem with people who write breathless tutorials about doing things with AI is that they are more likely than average to have been written with ChatGPT. Which, given the knowledge cutoff for most models, is not where I'd personally turn for data on recent technical developments, but is par for the course for the kind of low-effort copy-paste bloggers doing it for attention.

This particular one seems to be from someone who is documenting their learning process, which is a valuable contribution but, obviously, not a source of great authority on the how's and why's.

sa-code• 2 years ago

Thank you for saying this! The number of people that would need to fine tune vs just using RAG is really small. People that are not familiar with the source often jump to fine tuning as an option

Foobar85682• 2 years ago

I am still unsure where to stand on this fine tuning vs rag. I feel that for live data, rag would be preferable but for daily/weekly updated one, then fine tuning.

Another aspect where I am unsure is multi user for a model e.g. can we have concurrency for a model or the queries have to be queued.

imjonse• 2 years ago

The "why for" is usually learning/gaining experience/FOMO.

TOMDM• 2 years ago

For the human or the LLM?

iAkashPaul• 2 years ago

With unsloth's optimizations you can do llama-3-8b's QLoRA fine-tuning on your 8GB card(mine's a 2070S) with 900MB to spare with BS of 4.

SunlitCat• 2 years ago

Since the crypto (currency) craze of 2017, every time I hear "consumer GPU" somewhere in a story that has nothing to do with gaming, it sends a chill down my spine.

j0hnyl• 2 years ago

RIP your spine for the foreseeable future.

Fine tune LLAMA3 on million scale dataset in consumer GPU using QLora, DeepSpeed

Related Stories