joshhart

According to the article, the major effect is they are catching and prosecuting cheaters with a secondary effect that students are over-relying on them for homework or practice, and are unprepared for exams.

The article also talks about the fact that the SAT was banned. I really am curious what percentage of the effect is from no SAT vs LLMs. I guess they haven't had the SAT for a while though so it's gotta be the LLMs?

Fireworks will serve them for $1.74 / $0.14 / $3.48. That's input / cached input / output. https://fireworks.ai/models/deepseek-ai/deepseek-v4-pro . Call it about a third the price of Sonnet.

Not nearly as cheap as the Chinese infra but still pretty cheap.

If you have good ideas that have a nice return on investment and leverage existing skills, sure. If you don’t have good opportunity laying around, best for the business to switch to maintenance mode, which means cutting staff. Or maybe cut staff, then use equity to buy growth via acquisition. It really depends on the business. Block’s growth has slowed so perhaps this would have happened anyway and AI is just what’s getting the blame.

I thought this wasn't viable due to cooling requirements - how do you cool massive amounts of compute when the only option is to radiate it into space - nothing to convect it with?

Also, the incredible amount of grift here with the left hand paying the right is scarcely believable. Same story as Tesla buying Solarcity. Board of directors should be ashamed IMO.

Vitamin D toxicity is absolutely real, causes hypercalcemia, and can occur even at the 4,000 IU dose. I would really recommend you be getting regular bloodwork done if you go beyond that. Here’s a fun podcast on a case study. https://www.barbellmedicine.com/podcast/episodes/episode-381...

Huh. The standard in your case is to measure waist circumference if BMI is high. Did no doctor do that? As long as you are below 40” or 37” if Asian you are considered good to go.

This is super awesome, but how in the world did they come up with a name "Nano Banana Pro"? It sounds like an April Fools joke.

I believe it's intended to convince the audience they are experts, that this type of thing is dangerous to a business, and they are the ones doing the most to prevent it. There is no explicit statement to this effect, but I get the sense they are saying that other vendors, and especially open models that haven't done the work to curate the data as much, are vulnerable to attacks that might hurt your business.

Also a recruiting and branding effort.

All of this is educated guesses, but that's my feeling. I do think the post could have been clearer about describing the practical dangers of poisoning. Is it to spew misinformation? Is it to cause a corporate LLM powered application to leak data it shouldn't? Not really sure here.

So the inference speed at low to medium usage is memory bandwidth bound, not compute bound. By “forecasting” into the future you do not increase the memory bandwidth pressure much but you use more compute. The compute is checking each potential token in parallel for several tokens forward. That compute is essentially free though because it’s not the limiting resource. Hope this makes sense, tried to keep it simple.

A single node with GPUs has a lot of FLOPs and very high memory bandwidth. When only processing a few requests at a time, the GPUs are mostly waiting on the model weights to stream from the GPU ram to the processing units. When batching requests together, they can stream a group of weights and score many requests in parallel with that group of weights. That allows them to have great efficiency.

Some of the other main tricks - compress the model to 8 bit floating point formats or even lower. This reduces the amount of data that has to stream to the compute unit, also newer GPUs can do math in 8-bit or 4-bit floating point. Mixture of expert models are another trick where for a given token, a router in the model decides which subset of the parameters are used so not all weights have to be streamed. Another one is speculative decoding, which uses a smaller model to generate many possible tokens in the future and, in parallel, checks whether some of those matched what the full model would have produced.

Add all of these up and you get efficiency! Source - was director of the inference team at Databricks

This is pretty exciting. Now an organization could produce an open weights mixture of experts model that has 8-15b active parameters but could still be 500b+ parameters and it could be run locally with INT4 quantization with very fast performance. DeepSeek R1 is a similar model but over 30b active parameters which makes it a little slow.

I do not have a good sense of how well quality scales with narrow MoEs but even if we get something like Llama 3.3 70b in quality at only 8b active parameters people could do a ton locally.

Hi,

Yes you can. The community creates quantized variants of these that can run on consumer GPUs. A 4-bit quantization of LLAMA 70b works pretty well on Macbook pros, the neural engine with unified CPU memory is quite solid for these. GPUs is a bit tougher because consumer GPU RAM is still kinda small.

You can also fine-tune them. There are lot of frameworks like unsloth that make this easier. https://github.com/unslothai/unsloth . Fine-tuning can be pretty tricky to get right, you need to be aware of things like learning rates, but there are good resources on the internet where a lot of hobbyists have gotten things working. You do not need a PhD in ML to accomplish this. You will, however, need data that you can represent textually.

Source: Director of Engineering for model serving at Databricks.

The benchmarks compare it favorably to GPT-4-turbo but not GPT-4o. The latest versions of GPT-4o are much higher in quality than GPT-4-turbo. The HN title here does not reflect what the article is saying.

That said the conclusion that it's a good model for cheap is true. I just would be hesitant to say it's a great model.

Hi, I run the model serving team at Databricks. Usually you run regex filters, LLAMA Guard, etc on chunks at a time so you are still streaming but it's in batches of tokens rather than single tokens at a time. Hope that helps!

You could of course use us and get that out of the box if you have access to Databricks.

I spent 12 years at LinkedIn. Sadly, it's not even close to the engineering org it used to be. The era where Kevin Scott led engineering was a really good one in comparison.

At Databricks we have an LLM that is fine-tuned to do the problem you raise -

https://www.databricks.com/blog/announcing-public-preview-ai...

Many customers like it a lot. Although perhaps in your case if there are many pricing details it may not be quite accurate.

Makes sense, CPUs and memory sizes aren’t growing that fast anymore. But I’m sure they are spending a ton on TPUs/GPUs, the article is clear on very high capex

If you are making many requests in batch this works ok because you can shuffle the next layer in while the current one is processing a set of matrix multiplies. This takes it from being a memory bound problem to a flops bound problem. This really only works if you care about throughput and not latency.

LinkedIn was already very FCF positive. They tightly managed margins to get to net income positive (account for dilution and so on) but it took maybe 2 years after the acquisition.

This was cancelled over a year ago - which the articles notes and is old news. It was clear the effort would have needed a very significant push that would have required a large halt in product development and management wasn't willing to stomach it due to high growth in 2020/2021. Which made sense. But LinkedIn revenue growth has heavily slowed with the pullback in tech hiring and they had the space to do it and consider it optimization time.

Also as part of Blueshift the plan was to do batch processing first but LinkedIn had a culture belief in colocation of batch compute & storage, which is against the disaggregated storage paradigm we see now. IMO this led to some dragging of feet.

Source: Worked at LinkedIn 12 years, am a director at Databricks now.

I left LinkedIn 1.5 years ago. I was there 12 years. I saw the revenue & profitability growth that occurred post acquisition. I am very very confident LinkedIn would be worth north of $100B on public markets today and Microsoft made the acquisition for $26B. You might argue that in the subsequent 6 years post acquisition that wasn't enough growth and they should have bought back shares instead but it was completely a debt financed acquisition and very high ROI for Microsoft.

Will be a long time before a large number of these are available in clouds though, not until summer at least.

Trainium and Inferentia sadly aren't going this way, they have their own approach https://github.com/aws-neuron/transformers-neuronx . I think the best case scenario is some middleware like Triton https://openai.com/research/triton becomes the standard and we can build adapters to all of these backends. It is disappointing though.

This is a big cluster, definitely large enough to pretrain 100B+ parameter LLMs in months. Source - I work at Databricks in the ML platform.

I don’t know much about AV processing, that’s highly customized to only a few customers but I’d expect it to also have very large computational requirements to do video processing and reinforcement learning.

I think these are good points, I am hugely in favor of expanding food stamps and child tax credits for this reason. One estimate is that every $1 spend on food stamps expands GDP by $1.50, so this is really good for the overall economy. I have heard of much higher estimates, but cannot easily find the source. We definitely need to help poor families break the poverty cycle. Schools should have free breakfast, lunch, dinner, and after school programs as well to deal with lack of food at home and 2 parents working full-time.

https://www.ers.usda.gov/amber-waves/2019/july/quantifying-t...

The article suggests several causes: 1. Bias in population - these students all have families where at least one parent has a stable job, which isn't true elsewhere. There could also be other factors, for instance maybe people who enter the military are more motivated on average and that genetically or through parenting is passed on to their kids. 2. Better funding 3. Frequent feedback to teachers and more methodical planning 4. Excellent racial & socioeconomic integration

Is there a way to tease out the contribution in each area through controlling for variables. I suspect #1 is the largest by far, but I think this could be statistically controlled for partially by looking at children of parents who attend non-military schools. Curious for thoughts from HN.

Former director at LinkedIn. There might be that many internal URL endpoints (I am skeptical though). There are far fewer services than that though, each service supports many APIs.

You are totally right - both more and better matters. There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind - https://arxiv.org/abs/2306.11644

From a modeling perspective, I am impressed with the effects of training on 2T tokens rather than 1T. Seems like this was able to get LLAMA v2 7b param models equivalent to LLAMA v1's 13b performance, and the 13b similar to 30b. I wonder how far this can be scaled up - if it can, we can get powerful models on consumer GPUs that are easy to fine tune with QLORA. A RTX 4090 can serve an 8-bit quantized 13b parameter model or a 4-bit quantized 30b parameter model.

Disclaimer - I work on Databricks' ML Platform and open LLMs are good for our business since we help customers fine-tune and serve.

One of the “words” is a stop token, which represents the ending of text. So you can say the thing that maximizes coherence is to stop right then.