kamranjon

So here is an important question I think.

If LLM outputs aren't copywriteable and you create your own synthetic training set using Fable and share it publicly on huggingface, and someone else uses that training set to fine-tune a model, would this be considered illegal?

I ask because this happens all the time, synthetic datasets have basically become a key aspect of training a model at this point. I even generated a synthetic set from DeepSeek v4 to aid in fine-tuning a classifier just a few weeks ago.

So I just wonder on what grounds any of this makes sense, I wouldn't be surprised if some of these American labs were using open models on their own self hosted infrastructure to generate training data, but by nature of them being open nobody has to know.

I'll make a prediction: I don't think we will ever see any of the evidence of this "distillation" before they end up implementing some type of ban.

"It went from the start of training to launch in under nine weeks..."

This is pretty impressive.

What quant are you using?

Whoa whoa whoa, 118b params, 8b active MOE, long context reasoning, open weights - music to my ears. Hadn't heard of this lab before but I am very excited, will definitely try this out tomorrow - this is a real sweet spot I think in terms of model size and performance.

What I think is really incredible about this is that they are selling a ton of subscriptions, both personal and business subscriptions. So the fact that they are doing advertising on top of this suggests to me that they are in a deep deep hole, much deeper than he probably would have imagined when making this statement. If your revenue is 25 billion and you still need ads, your costs must be running you down at a pace you really didn't anticipate.

Pretty bold move to release this at the peak of the Open Models vs Proprietary Models debate.

Feels like a really easy decision when it's framed this way.

I don't really think the author is misrepresenting the quote in any way, it just seems like many people are reading it as a hard statistic instead of a reference to a properly attributed quote.

Just anecdotally though, my company is not a startup, well established and well known and has already started investigating, purely for dev purposes (not product), using Chinese models - this was spurred by costs rising much faster than expected.

So while I agree that I don't think it's anywhere near the 80% level across the board - I wouldn't be surprised if it starts moving that way.

"It’s obvious to me that there are ecosystem benefits throughout China, from manufacturing to scientific research; every sector can just plug in these models."

I agree with this sentiment and think it's echoed in Fareed Zakarias take here: https://youtu.be/VBblUjLw5lE

China seems to perceive AI as a much more sensible technology than the US and seems to be integrating it in far more industries than the US.

I'm not sure the American mind can understand the distributed benefits afforded to the Chinese economy from opening their AI models, I think it's pretty reductive to assume it's purely a strategy of undercutting American frontier labs.

DeepSeek V4 Flash

You can run Wordpress locally in your own sandbox to test these vulnerabilities, you don’t have to break any laws.

So I am likely coming from a completely different world because I use entirely self hosted models - but is it really common in your workflows to just wait for 12 minutes without any feedback? I am constantly watching my model and following the path it’s taking and cutting it off or steering it if it’s headed down a dead path. I was just curious if this is really common to just have no feedback for 12 minutes?

Unfortunately it is becoming more and more common for US alleys to use the pins with cords systems. Many lanes will actually upgrade to this system and I feel it is a mini tragedy in a way. I am very happy to see this because it gives me hope that there might be alternatives for bowling alleys who are considering the string based systems.

I wouldn’t say I’ve backtracked- I think I’ve been incredibly consistent here. Chinese labs are releasing open weight models, research and analysis. Anthropic is not. They haven’t produced any actual evidence of distillation themselves and what they have presented is tenuous at best.

By evidence I mean logs, I mean IP addresses, I mean timestamps. They claim millions of requests, let’s see literally any of them?

I don’t consider a tweet by Denise Wu, who works at Anthropic, to be reproducible evidence.

I don’t consider “Caveat: fully AI-generated research.” To be someone taking time to analyze anything in great detail.

Because two AI models produce vaguely similar front-end styles when generating similar prompts I also do not consider to be of much value?

I think this is what I mean when I say the U.S. has its head in the sand. The Chinese labs are releasing ~60 page research reports with citations and analyses and evidence and Anthropic is throwing up defensive blog posts with zilch. I’ve seen more detail in a tech blog from Uber than anything I’ve seen from Anthropic.

It’s so funny to me that Anthropic can make claims like this one with zero evidence provided.

DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe.

Chinese labs might not have the funding of labs like Anthropic, but at least they provide the receipts.

The fact that API based distillation is even a conversation right now makes me feel like the U.S. has their heads so far in the sand that it’s not really excusable.

These Chinese labs are producing novel models, publishing their techniques and sharing their open weights and the first topic of conversation is how they stole from U.S. AI labs.

Setting aside the fact that it doesn’t make any feasible sense to do API distillation, these models are outperforming frontier models on a number of benchmarks, and often times run more efficiently by several orders of magnitude.

We have to stop crying distillation, it’s getting embarrassing and at this point feels even a bit delusional.

If you look at this chart here it seems the tiny model has a WER of ~12%… not sure about the micro model:

https://github.com/moonshine-ai/moonshine#when-should-you-ch...

Allen AI is fighting the good fight with their Olmo models - I’m hoping for a new release soon. Also huggingface has released some pretty nice smaller models with open pipelines like Smollm.

The difference is that because they are open weights, it’s trivial to decensor the models with abliteration and you’ll generally see decensored versions within days of new releases - something you just can’t do with proprietary models from frontier labs.

I also think Allen AI with their Olmo models are doing great stuff by releasing full training data and pipelines and checkpoints which I’m not sure any other US company is really coming close to transparency-wise.

“QuickJS is to this PSP what V8 is to Node: the host hands it a strike API surface and a ui API surface, and the same openstrike.js bundle boots against them on every target.”

This comparison seems backwards? Also I have no idea what it means - it seems like really strange phrasing and I can’t seem to make sense of it. Does anyone understand what was trying to be conveyed here?

Hearing a world leader talk about open source is huge. I think it highlights pretty starkly the difference in approach between the U.S. and China and will likely in time be seen as a pretty historic statement.

Where did you hear about the deepseek release? Would love to follow the same source.

“Alongside Inkling we are sharing a preview of Inkling-Small, a 276B-parameter Mixture-of-Experts model (12B active, vs. 41B for Inkling) with a different performance/latency trade-off.”

Buried at the end there is the details I was most interested in - a possible competitor for DeepSeek V4 Flash? Excitedly awaiting the release of the weights for this one.

70% seems a little extreme? Here is a helpful mobile coverage map: https://www.fcc.gov/BroadbandData/MobileMaps/mobile-map

A few years ago I actually traveled all around the US and worked remotely with a 4g unlimited data sim card and was pretty amazed at the coverage, only very few places was I left without coverage. I'm guessing it's even better now?

I am traveling in Europe currently and got a Saily SIM card - 5g coverage is really good and seems to be expanding fast - a small village I visited 2 years ago that had no cell coverage at all I was getting 250mbps download, faster than than the wired internet at most of the places I was visiting. This prompted me to actually look into Starlink speeds and apparently 5g is generally faster than Starlink...

This made me wonder if Starlink is a much more niche product than I thought - I actually thought it was significantly faster than cell data - but it seems it's just for folks in super remote areas at this point?

Doesn’t this suggest you aren’t properly running the model?

Curious why you find that’s a useful test since it seems to be solely measuring training data memorization, something you’d expect to degrade from quantization.

When new models are released (I realized this is qwen 3.6 but the quant is novel) - it takes a few days for the kinks to get worked out - you’ll likely have better luck if you give it a few days and try again.

Yea I don’t use sub-agent style workflows. I often use planning patterns and generate markdown for really complex tasks, but never got into the sub-agent thing. I am likely closer to AI-assisted, though I am heavily using pi coding agent and my editor is basically just for viewing files and changes now.