clauderoux

I have been a pretty consistent user of AI since 2022 (Instruct-GPT), so I don't have a bad opinion about the topic. However, I think the real problem now has become pretty obvious. We are hitting a reality wall, where we simply don't have enough ressources to feed the AI industry. We don't generate enough electrical power nor enough GPU or TPU. For the first time in computer science, the real issue here is the finitude of the physical world. Unless, we start digging asteroids, we are already facing a shortage of raw material and industrial output. In my opinion, the only way to go is small models running on regular hardwares.

LispE has been fully ported to WebAssembly, you can see it in action with some examples.

LispE is a programming language that has been ported to WASM. It's lightweight (about 3.3MB) and provides many features natively, with no external dependencies. You can now try it directly in your browser.

This article nails it. The claim 10x is in my opinion one of these tactics used by large corporations to force engineers into submission. The idea that you could be replaced with an AI is frightening enough to keep people in check, when negotiating your salary. AI is a wonderful tool that I use everyday, and I have been able to implement stuff that I would have considered too cumbersome to even start working on. But, it doesn't make you a 10x more efficient engineer. It gives you an edge when you start a new project, which is already a lot. But don't expect your whole project of 100,000 lines to be handled by the machine. It won't happen any time soon.

Formulas can be written in English and translated by an AI in JS formulas.

I love how so many people are eager to criticize LLM code, when in fact, according to my experience it is pretty superior to anything I have seen produced by human programmers, most of the time. It is documented, the code is explained at each step of its creation, and it is pretty readable when you dig into it. I have 30 years of experience in coding, and I have been playing with these LLM for 3 years. Yeah!!! Of course, sometimes they produce very bad code. But in average, the code they produce is largely on par with my fellow humans. And since, they produce the whole explanation of it, it takes a couple of minutes to understand it. And if you don't understand the main points of the code, the LLM will tell you all about it. When did you have a colleague that was eager to explain his/her code to you??? When did you have a colleague that did produce a code you could understand in a few minute??? I really think these tools are quite useful, no need to wrap yourself into the mantel of expertise and look down on these LLM, because sometimes they will produce a code you don't like.

The answer is very easy. Latinate words (mostly of French origin for the matter), first because of the number of people speaking romance languages (French, Italian, Spanish, Romanian and Portuguese) is the largest in Europe, second, because most European languages have also borrowed a lot from Latin, Greek and French during the last centuries, which means that latinate words are usually the subset which is shared across most languages. Furthermore, English and other Germanic languages have started evolving quite early one from the others (around the 6 century) and cognates might be quite difficult to recognize: through/durch, for instance.

Funny as they always seem to forget the hardware side of search engines. Google was incredibly fast compared to its competitors because they were among the first to store their whole index in RAM rather than on hard drives. They were among the first to install huge data center with computer blades that could be changed in an instant in case of failure. As an early user, I was on board as early as 99, I was amazed by the response speed of Google and its bare style quite dépouillé.

Well. There is measles epidemic right now...

Some people still think that LLM are just word predictors. Technically, it is not. First, transformer architectures don't process words, they process semantic representations stored as vectors or embeddings in a continuous space. What a lot of people don't understand is that in an LLM, we go from discrete values (the tokens) to continuous values (embeddings) that the transformer takes as input. A transformer is a polynomial function that will project into the latent embedding space. It doesn't generate word per se, but a vector that is then compared against the latent embedding space to find the closest matches. The decoding part is usually not deterministic. This huge polynomial function is the reason we can't understand what is going on in a transformer. It doesn't mimick human speech, it builds a huge representation of the world, which it uses to respond to a query. This is not a conceptual graph, it is not a mere semantic representation. It is a distillation of all the data it ingested. And each model is unique as the process itself is split over hundreds of gpu, with no control over which GPU is going to churn out which part of the dataset and in which order.

For those who don't know, "frelon" is French for hornet.

Gradient descent is to AI, what a loop is to understand programming. However, understanding a loop doesn't mean you can program a full video game from scratch. Organizing hundreds of layers in an efficient way is pretty complex, even if the work today has been simplified thanks to PyTorch or Tensorflow, it remains pretty complicated. You have to understand how to organize your data, how to size your batches, how to make your code resilient enough to survive GPU cards crashing. Train a model over hundreds of GPU is really really complicated. New algorithms are proposed all the time, because we have no idea how to handle these interconnected layers in an efficient way, but with cumbersome heuristics. However, salary inflation is never a good thing, because it will create a gap between decent engineers and other people pronounced geniuses. The AI teams will suffer from these decisions... Badly. It will be like these samouraïs who would kill peasants after a battle to increase their head count, because this was how people were rewarded after a battle. Some of these people, in order to justify their salaries, will feel pressure to poach other people's ideas...

The problem of measuring intelligence is that most of these techniques reveal more about what we think intelligence is in a specific cultural context. Which means that individuals who rank higher in these tests are those who conform the most to a society expectancies. High level of literacy, proper nutrition during their childhood, proper education, help foster better IQ. Which of course leads to a better understanding of how your own society works, hence to a better evaluation of probabilities. If you have grown up in a family with money, you have a much better understanding of how to benefit from your investments.

What I do is to build a substring out of the initial strings that is a multiple of 2. If the string I try to search is 9 characters long, then I extract an 8 characters substrings that I transform into an integer over an integer:

Here is the example for 4 characters: //We look for the presence of four characters in a row ``` int32_t cc = search[3]; for (shift = 2; shift >= 0; shift--) { cc <<= 8; cc |= search[shift]; } __m256i firstchar = _mm256_set1_epi16(cc); ```

In this case, I will look for a 4 bytes integers over my sequence: ``` current_bytes = _mm256_cmpeq_epi16(firstchar, current_bytes); q <<= 1; q |= _mm256_movemask_epi8(current_bytes); ```` I'm looking for blocks of 4 characters at a time in my string.

I implemented many different algorithms for searching and splitting strings using SMID methods as well: https://github.com/naver/tamgu/blob/06aedf2f14895925d7b5a8e2... I used a different algorithm than the ones presented here.

Or a variation of Princess Bride mind challenge...

Seriously, most fine-tuning now is done with LoRa adapters. They are much faster and more reliable. In my lab, I don't know anybody who is trying to do any kind of thorough fine-tuning...

As Linus Thorval said in an interview recently, humans don't need AI to make bugs.

If you have a look on the GitHub, the oldest files are 5 years old, way before LLMs...

As I said many times, I have been in the game for 30 years. I started doing AI with rules as early as the beginning of the 90 and I never...never expected to see anything like LLMs in my lifetime. When I read Marcus once again saying that: yes this time LLM have reached their limit, which he has been saying for 2 years in a row, I'm really feeling tired of his tune. The idea that LLM are a dead end, a failing technology is pretty weird. Compared to what??? I use LLM everyday in my work, to write summaries, to make translations, to generate some code or to get explanations about a given code... And I even use them as research sparring partners to see how I could improve my work... Gary Marcus has been involved in the domain for 30 years as well... Where is his technology that would match or surpass the LLM???

Who needs culture when you can buy guns...

"La suite vulgate du Merlin" might translate as: "the vulgate follow up of THE Merlin", which I find quite curious. Modern French would rather say: "DE Merlin". It was as if Merlin was not a proper name but the name of some creature of the Merlin species, or some kind of properties.

There's an error in the text. It says they were Huguenots who left after the Edict of Nantes, which is not true. They left after the revocation of the Edict of Nantes in 1685, which had protected Huguenots until then. This last edict is called the Edict of Fontainebleau. So they left after the Edict of Fontainebleau.

So true... The Democrats would be a right wing political party in most European countries... The Republicans on the other hand are closer to extreme-right parties such as the Rassemblement National in France or the Aternative für Deutchland in Germany. By the way, the only leader that Vance met with was the leader of the AfD in Germany, which is pretty telling of their political orientation... In the UK, Elon Musk, for instance, is closer to Farage than to any other English leaders, which again shows an extreme-right trope from the current Republican administration.

You've never been to France, right? Maybe this was true 20 years ago, but it is no longer the case. Where I work, in France, there are people who've been here for 10 years who still can't speak the language, which I find pretty depressing. Especially that learning French when you already speak English is pretty easy. I met enough people that managed to become fluent in French in six months to know that it is really not that hard when you put a bit of effort in it. What it usually shows is a lack of curiosity for your environment, which is a big red flag about the social skills about the person.

I own a Mazda CX30 since 2021 and the most important buttons are all physical. There is still a screen of course, but it is not a sensitive one. Basically, after a few weeks, you reach most of the buttons through muscle memory. You never need to move your sight out of the road to start your music or anything else... So far, it has been a great experience

This is the same problem that I have with creationists, when they say that life cannot emerge from "rocks". Still, we are a walking bag of electrico-chemical reactions based on the same basic minerals that are found everywhere on the planet. There is no "specific, invisible component" that would make us alive, but an incredibly complex arrangement of minerals and metals. In the same way, what would make the mind impossible to duplicate? Unless you believe that there an inscrutable, extra-subtile element that we cannot see or demonstrate? The soul maybe? So basically, every argument against AGI fells in the category of religious beliefs.

Mazda is pretty great actually. I have a CX30 (3 years), where the screen is not sensitive and everything is controlled with mechanical buttons...

The best of both worlds...

As long as the US has H1B visas, everything will be fine...

I find the narrative that Chinese researchers are stealing AI technology from the US very weird indeed. I have been working in AI for more than 30 years and in the last 15 years almost every major paper on the topic has had at least one Chinese author. In 2018 I went to conference to Florence and 60% of the speakers were Chinese. They sent their students to every lab in the world to train them and to advance in the field as much as they could. And today it is paying off. Also, if you watch the video that was released by OpenAI for the O1 presentation, almost no researchers in the team was American. Most of them were either European or Asian. So yeah!!! The US is number 1 in AI, but without much of local players. It reminds me of the British football championship, which is one of the best in Europe, but where the national team is not much of a powerhouse.