AMÁLIA and the future of European Portuguese LLMs

https://duarteocarmo.com/blog/amalia-and-the-future-of-european-portuguese-llms

by johnbarron • 3 months ago

140 77 3 months ago

duarteocarmo.com

hartator• 2 months ago

What a waste of time and money.

Trying to force a LLM into a specific language makes you missed out on most of the world knowledge.

mistrial9• 2 months ago

makes you missed out on most of the world knowledge

and, who knows what will happen to grammar ?

embedding-shape9• 2 months ago

What LLM isn't forced into a specific language? That'd be a weird language model no one could understand, you need to chose at least one language, ideally the same as the creators speak.

Besides, there is knowledge that is locked behind languages, there are things known in Portuguese that aren't known in other languages, and the same for other languages too. More accessibility to those ideas wouldn't hurt.

Miraste• 2 months ago

To my knowledge, all major LLMs are multilingual. This article could really have used an evaluation of existing models' European Portuguese capabilities.

cess11• 2 months ago

E.g. gemma3:4b can fake simple conversations in several european languages, including portuguese, swedish and finnish.

It's just a database. If you push text in one language into it, it'll likely crap out stuff in that same language, unless the system prompt that also goes in with your query causes it not to.

numpad0• 2 months ago

yeah, they seem all confined to being an American-consultant-Chinese-authoritarian split personality with broad second language capabilities. I suppose they become too incoherent otherwise.

CrimsonRain2• 2 months ago

Europe always has a thing for their languages. They think many languages make them stronger while spending billions in system loss due to communication barriers. It is obvious they will try to do the same with LLMs and call it the next best thing since bread and butter.

I went to JCON EUROPE this year. The amount of "Europe this" "Europe that" "sovereign this, sovereign that" is mind boggling and just a waste of time and money. The regular people know this and thus left the conferences mid way. But somehow the people "in charge" really need to push this. Same thing here.

lmf4lol• 2 months ago

whats your suggestion? we just eradicate all of our culture and languages and go full on english ?

whats wrong with exploring ways to keep national languages alive in the LLM area

swiftcoder5• 2 months ago

It is definitely an interesting problem, because Portugal is a small enough country that the actual total corpus of available texts in (non-Brazilian) Portuguese is potentially problematic.

embedding-shape9• 2 months ago

I don't think so, Portugal the country might be small, with a small population, but there is ~250 million "Lusophones" (native Portuguese speakers), making it the fifth-most spoken native language in the world, I'd hardly call that small :) And before everyone screams; yes, European Portuguese is different from Brazilian Portuguese, but they're still both Portuguese and understand each other, so it's not like the text from one cannot be used to train a model for the other, or vice-versa.

All in all, I don't think that's a major issue here.

KK7NIL3• 2 months ago

The whole point of this project is to have an LLM that speaks European Portuguese, not Brazilian Portuguese.

swiftcoder5• 2 months ago

The authors are pretty clearly trying to draw only from European Portuguese sources - I feel like there's a fairly widespread attitude here that the language is being overwhelmed by the sheer number of Brazilian speakers (which there is obviously at least some truth to).

I don't necessarily personally feel like preserving European Portuguese in amber is a worthwhile goal (anymore than it is productive for Brits to be prickly about the meteoric rise of US English)

madaxe_again2• 2 months ago

Mutually intelligible, yes, but far from perfectly so. I speak both, as a native anglophone, and the difference is not so much “US vs British English” so much as “Guyanese English vs British English”. Like, fundamental points of grammar differ, the spoken rhythm and syllabic stress differs (poetry does not translate well between them), never mind just vocabulary. Continental Portuguese people tend to find it easier to understand brasileiros than vice versa, largely due to mostly one-way cultural exports, but to try to roll both into a single model would create a creole at best.

mghackerlady• 2 months ago

Right, but most of those speak brazilian portuguese. There's so much less european portuguese text that it becomes impossible for a model to not speak brazilian portuguese if not trained in a way that ignores brazilian sources

evandrofisico• 2 months ago

Portugal has a growing Xenophobic attitude towards immigrants, specially Brazilians and this is reflected in linguistic prejudice.

They have concerns of portuguese children learning to "speak brazillian" because there is a lot more of video content being produced in Brasil than in Portugal and stuff like movies, videogames and software in general are avaliable in brazilian localization/adaptation first.

matusp• 2 months ago

Never heard the term Lusophone before. TIL

pjmlp5• 2 months ago

You're missing African Portuguese.

fy20• 2 months ago

European Portuguese is the 13th most populous language in Europe. Not that small, there are many other European languages in use that are much smaller.

https://en.wikipedia.org/wiki/List_of_languages_by_number_of...

depaulagu• 2 months ago

European Portuguese is the 13th most populous language in Europe

that's not impressive

augusto-moura3• 2 months ago

It is pretty small when considering content output. It is only 11 million people, and only a fraction of them will be writing something that could be used on training datasests. If you look at the countries by scientific contribution, for example [1], Portugal is on the 28th position, while Brazil is in 14th by more than double the number of contributions.

Don't get me wrong, it is definitely impressive given Portugal's actual size, but I believe there's a hard limit for population and size that will be difficult to cross

[1] https://en.wikipedia.org/wiki/List_of_countries_by_number_of...

SkeuomorphicBee• 2 months ago

What makes Portugal's situation unique is that it is a small population that is eclipsed in models by the bigger weights of the much bigger population of Brazil.

Yes, there are much smaller European countries, but those are generally the only source of truth for their specific language, so the context of a LLM query in that language steers the LLM towards facts from that country, for example, if I ask a big generic LLM something in Latvian then it most likely will answer something relevant to the context of Latvia. But Portugal, being the much smaller user of its language, have the somewhat unique problem that if I ask a generic model something in Portuguese it will probably answer something related to Brazil instead of Portugal.

Maybe the UK and Spain have somewhat similar struggles, but I suspect that none has it as bad as Portugal in that regard.

bobthepanda• 2 months ago

I would have to imagine this might not actually be as bad as it seems, at the very least there should be a giant corpus of translated EU texts.

pu_pe• 2 months ago

I'm not sure the direction should be to finetune a small local model for each country or language. These models are already not particularly great at information retrieval, so I doubt anyone would use them for questions like the author suggests (ie who was the president between X and Y). Similarly, they are a little too lightweight to be used for translations too.

If the budget is indeed so modest (5.5 million euros!), I would focus completely on preparing datasets and making sure all open cultural artifacts that we can find are well documented in them. That way every model, private or open, that gets trained in the future could better represent the culture and language of your country.

iugtmkbdfil834• 2 months ago

I agree, the research is complex enough as is without having to worry about splitting it babel-like into multiple languages.

dyauspitr• 2 months ago

Yeah I think India is going the better route with Sarvam which is trained from scratch and still relatively cheap.

TheMagicHorsey• 2 months ago

This is the way.

Sovereign SOTA models might also be possible with nation-state involvement. But this is a good stopgap.

dudefeliciano• 2 months ago

who was the president between X and Y

this is the type of question that should never ever be asked to an llm running on some A100 on the other side of the world, local llms are already more than capable to answer these

algoth12• 2 months ago

Wouldnt it be easier to fine tune a model to convert the Brazilian Portuguese corpus into European Portuguese and then use that corpus?

kinow• 2 months ago

That idea is different than what most are talking here in other comments.

The grammar and vocabularies don't match, but I think the worst are the expressions. Both sides have *a lot* of expressions that vary per context and location.

algoth12• 2 months ago

I’m a native Portuguese speaker, whom grow up watching morning Brazilian Portuguese cartoons. I would say it’s doable

simianwords• 2 months ago

Domain specific models will never be a thing. You don't get generalised intelligence with that.

https://simianwords.bearblog.dev/why-domain-specific-llms-wo...

mariopt• 2 months ago

This model is a waste of Public Funds.

There is no public website to use it, be it free or paid, the dataset is not public, the code is not public (The github URL in the article returns 404 ), the claimed model intelligence is so low that is pretty much useless at 32K context and massively inferior to GPT‑4o.

As per tradition in Portugal, some people managed to get 5.5 Million to produce nothing and no one is asking questions.

You want a better idea? Just fine tune the open source Kimi 2.6 with an open source Portuguese dataset, the cost would be under a million and we would be getting something useful.

It would be really nice to know what happened to 5.5 Millions whilst not being able to even provide a functional website to use the model.

dr_dshiv• 2 months ago

It’s a way to suck all the money out of the room in the name of nationalism — and it’s all over Europe. Only idea everyone has had.

vova_hn22• 2 months ago

I'm not arguing with the rest of your points, but...

Just fine tune the open source Kimi 2.6 with an open source Portuguese dataset

I think that tokenizers of all popular models are heavily biased towards English or English and Mandarin.

And I don't think that it is possibple to replace the tokenizer without full retraining.

mcyc• 2 months ago

You are right about most tokenizers being heavily biased towards English, but the situation is not so bad for Portuguese. Here are some results on the Goldfish corpus [1] with a few different tokenizers. This measures #characters in corpus / #subwords in tokenized corpus.

```

Llama3

english, 0.216

portuguese, 0.285

italian, 0.287

greek, 0.592

```

Gemma4

english, 0.219

portuguese, 0.246

italian, 0.249

greek, 0.537

```

Kimi2.6

english, 0.214

portuguese, 0.310

italian, 0.308

greek, 0.716

```

Portuguese is worse than English certainly, but it is on par with Italian (which I think has more overlap with English) and much better than Greek (since it doesn't use the Latin script and is definitely not prioritized in the tokenizer construction).

On your second point, tokenizer transfer allows for extending/modifying a tokenizer without retraining the model from scratch. The simplest version of this is tokenizer extension + continual pretraining, where you just add a bunch more tokens to the vocab for the language/domain that you want to improve and train a little more. It's been done for Japanese [2] and Indic languages, but afaik not Portuguese.

So I think that continual pretraining for a large base model would have probably been fine for this case with huge cost savings. But it is good to have the ability to train your own base models, so I don't think this is such a bad idea.

-----------------------

[1] https://huggingface.co/datasets/goldfish-models/fish-food

[2] https://arxiv.org/abs/2404.17790

upupupandaway• 2 months ago

As a pt-BR speaker from across the pond: https://soberania.ai/

Similar waste.

gverrilla• 2 months ago

Why?

avdelazeri• 2 months ago

Given that their publication says the dataset is freely available on Huggingface that's at least something ig

edwcross• 2 months ago

Interestingly, the mobile version of the website contains a hamburger menu with an "Equipe" (Team) link that returns a 404 error (https://soberania.ai/equipe).

This link is absent from the desktop version.

Isn't it a bit odd that the team responsible for it is nowhere to be credited?

pjmlp5• 2 months ago

As Portuguese living abroad, reading the article made me think exactly the same , even though I wasn't even aware of the project.

mt_• 2 months ago

5 million for a llama-2 finetune, how is that impressive?

r2ob• 2 months ago

"This model is a waste of Public Funds". There is no "public funds", this is a waste of money from the tax payers.

alexaholic• 2 months ago

The Amália model is not yet publicly available. Until it's ready, one can fool around with Anália at https://analia.pt

ncruces• 2 months ago

That name change…

pelf• 2 months ago

I just died.

drivebyhooting• 2 months ago

I’ve noticed that ChatGPT is noticeably dumber in languages other than English. It even will confidently repeat common but wrong superstitions from the target language as if they were fact.

guilhas• 2 months ago

Amazing. Every country should have a foothold on AI, as it will have impact every area of a citizen's life

Quite cheap compared with most public spendings

Europe countries already produce very little. Let's not let the wave pass and end up in a future where Europe is continuously reliant on US and Chinese tech as usual. And their definitions of what truth is

AMÁLIA and the future of European Portuguese LLMs

Related Stories