Ask HN: What do you find most difficult about fine tuning AI models?

Those of you who have fine tuned NLP (GPT-X/Llama) or diffusion models, what was the most difficult or annoying part about the process?

I'm trying to help solve problems in this field, so I'm curious about your experiences.