jamesblonde

Yes, coding agents have reduced the skills/knowledge required to operate workloads on virtualized hardware. K8S and its ecosystem has changed so that it now provides 90% of what you need from the public cloud providers. Big changes that make 8-15X savings by running your own workloads. I think it will be the big players who move first, as they have most to save and have the resources to make it happen.

It certainly feels like Netflix is now a k8s shop. And it probably only a matter of time until they start repatriating workloads to optimize for costs. Then the world will sit up and notice.

It took me 5 years to get the boat - a 22ft daycruiser with toilet. That was 15 years ago, haven't looked back. Got a daycruiser from the UK. Drove there, bought a trailer there, drove it home. Arbitrage during the financial crisis - half the price of the same boat in Sweden at the time.

The name of the agent is 'finn' - is that a reference to Intercom's Fin agent?

I lead a European owned and operated Data/AI company, Hopsworks. We are the only competitor to Databricks/Snowflake/etc based-in and from Europe. We can still compete, as we have a deep research/industry background (ex-MySQL and KTH folks). Crossing the chasm is harder from here. Even if you build a great product (we are best at real-time AI) - we had a paper at SIGMOD 2024 where we showed higher thoughput/lower latency by a factor of 4-40X Databricks, AWS Sagemaker, and GCP vertex - we lack the echo chamber. (Try the mental exercise where Databricks' peers acknowledge massive over-performance through a peer-reviewed paper and imagine how much noise it would generate). Still, we can replace our competitor at their largest European customer, Zalando, for real-time AI. But it's a much harder slog than it should be due to the 10X lower round sizes (due to 10X smaller VC fund sizes). European pension funds place way more money in US VC funds than in EU VC funds - that is self-defeating.

How could they release something that doesn't support atomic rename and has no prospect of supporting atomic rename? Lots of workloads will crash and burn on this layer.

Files can be immutable if you have mutable metadata - but S3 does not have mutable metadata, so you can't rename a directory without a full copy of all its contents.

Immutable files can be solved by chunking them, allowing files to be opened and appended to - we do this in HopsFS. However, random writes are typically not supported in scaleout metadata file systems - but rarely used by POSIX clients, thankfully.

We have advanced to building S3 stores with claude code - impressive:

https://github.com/fractalbits-labs/fractalbits-main/graphs/...

S3 Files was launched today without support for atomic rename. This is not something you can bolt on. Can you imagine running Claude Code on your S3 Files and it just wants to do a little house cleaning, renaming a directory and suddenly a full copy is needed for every file in that directory?

The hardest part in building a distributed filesystem is atomic rename. It's always rename. Scalable metadata file systems, like Collosus/Tectonic/ADLSv2/HopsFS, are either designed around how to make rename work at scale* or how work around it at higher levels in the stack.

* https://www.hopsworks.ai/post/scalable-metadata-the-new-bree...

I say this quite a lot to data scientists who are now building agents:

1. think of the context data as training data for your requests (the LLM performs in-context learning based on your provided context data)

2. think of evals as test data to evaluate the performance of your agents. Collect them from agent traces and label them manually. If you want to "train" a LLM to act as a judge to label traces, then again, you will need lots of good quality examples (training data) as the LLM-as-a-Judge does in-context learning as well.

From my book - https://www.amazon.com/Building-Machine-Learning-Systems-Fea...

There has been a lot of proposals to dam up massive unpopulated sea-facing valleys in Mayo and Donegal and use pumped hydro with seawater. Was a bit topic 15 years ago, but never happened. All that happened was the silvermines pump hydro plant that seems behind schedule.

Prof Igor Shvets was behind this: https://en.wikipedia.org/wiki/Spirit_of_Ireland

The reference in the text to Anthropic’s “Towards Understanding Sycophancy in Language Models” is related to RLHF (reinforcement learning with human feedback).

Claude code uses primarily different "pathways" in Anthropic LLMs that were not post-trained with RLHF, but rather with RLVF (reinforcement learning with verifiable rewards).

So, his point about code being produced to please the user isn't valid from where I am sitting.

Here is an article I wrote this week with a section on Feldera - how it uses its incremental compute engine to compute "rolling aggregates" (the most important real-time feature for detecting changes in user behavior/pricing/anamalies).

https://www.hopsworks.ai/post/rolling-aggregations-for-real-...

Another headline would be: Founder saves 10X operational costs by investing time (and with Claude's help) to migrate off AWS.

They need to fix the addresses. In Stockholm, all of the companies are placed in the old town. At Hopsworks, we are in Sodermalm (hipster) - we are not old school money.

That, too, and materialize. Feldera is my favourite, though.

I gave a talk at PyData Berlin on how to build your own TikTok recommendation algorithm. The TikTok personalized recommendation engine is the world's most valuable AI. It's TikTok's differentiation. It updates recommendations within 1 second of you clicking - at human perceivable latency. If your AI recommender has poor feature freshness, it will be perceived as slow, not intelligent - no matter how good the recommendations are.

TikTok's recommender is partly built on European Technology (Apache Flink for real-time feature computation), along with Kafka, and distributed model training infrastructure. The Monolith paper is misleading that the 'online training' is key. It is not. It is that your clicks are made available as features for predicitons in less than 1 second. You need a per-event stream processing architecture for this (like Flink - Feldera would be my modern choice as an incremental streaming engine).

* https://www.youtube.com/watch?v=skZ1HcF7AsM

* Monolith paper - https://arxiv.org/pdf/2209.07663

it's whitewashing. The US Cloud Act means they can still take the data.

I got turned off in the first paragraph with the misuse of the term "back pressure". "back pressure" is a term from data engineering to specifically indicate a feedback signal that indicates a service is overloaded and that clients should adapt their behavior.

Backpressure != feedback (the more general term). And in the agentic world, we use the term 'context' to describe information used to help LLMs make decisions, where the context data is not part of the LLM's training data. Then, we have verifiable tasks (what he is really talking about), where RL is used in post-training in a harness environment to use feedback signals to learn about type systems, programming language syntax/semantics, etc.

If they are exploding categorical variables using OHE and storing the columns - that is the wrong thing to do. You should only ever store untransformed feature data in tables. You apply the feature transformations, like OHE, on reading from the tables, as those transformations are parameterized by the data you read (the training data subset you select).

Reference: https://www.hopsworks.ai/post/a-taxonomy-for-data-transforma...

And the Wallenberg family.

Subsidiarity has been a key building block of the EU and has failed the EU for unexpected reasons. Subsidiarity was pursued for accountability and to make the EU less centralized - decisions should be made at the lowest, most local level possible, with central authorities only stepping in when a task cannot be effectively handled locally. However, it means that here in Sweden govt bodies are all individually moving to Azure, because each one makes that local decision in their best interest. The same thing has happened all over the EU - and very few govt bodies would ever take the risk of investing in using EU cloud or data platforms. We need public procurement to help kickstart life into the Eurostack.

[flagged]

I will be servers as well. Eurostack cloud providers. We are involved in one of these - a large car company doing the same.

They control Europe's digital infrastructure and are able to increase rent to usurous levels (tarrifs!) because Europe is dependent on their digital services. Without digital sovereignty, Europe has no sovereignty and will quickly become a modern colony from which wealth will be extracted.

I agree that this is an anti-pattern for training. In training, you are often I/O bound over S3 - high b/w networking doesn't fix it (.saftensor files are typically 4GB in size). You need NVMe and high b/w networking along with a distributed file system.

We do this with tiered storage over S3 using HopsFS that has a HDFS API with a FUSE client, so training can just read data (from HopsFS datanode's NVMe cache) as if it is local, but it is pulled from NVMe disks over the network. In contrast, writes go straight to S3 vis HopsFS write-through NVMe cache.

I have one (top of the line!). Here's how bad the engineers were. For the last 6 months, the device emits 10 audible beeps every 6 hours. I do a lot of customer meetings and public speaking. People would sometimes ask - "what is that noise"? I would say "No idea, but if you wait 8 seconds, it will stop"!

Also, my heart rate would sometimes drop below 40 bpm. Then it would start pacing, which i didn't want and was extremely uncomfortable.

p.s., the reason the battery ran out was because i found a treatment for my condition that works really well through talking globally to experts (i am a computer scientist). I wrote a case study paper about my condition to help others, co-authored by my doctors. https://www.slideshare.net/slideshow/arvc-and-flecainide-cas... 16 years later, the device is still in place, but I will have it removed early next year.

Anything sovereign AI or whatever is gone immediately when the mods wake up. Got an EU cloud article? Publish it at 11am CET, it's disappears around 12.30.

See, Peter Thiel is smart. There are enough idiots who will buy his shtick - it's not just maga who get pointed in the direction he wants society to go (serfdom).

Cloudflare tried to build their own feature store, and get a grade F.

I wrote a book on feature stores by O'Reilly. The bad query they wrote in Clickhouse could have been caused by another more error - duplicate rows in materialized feature data. For example, in Hopsworks it prevents duplicate rows by building on primary key uniqueness enforcement in Apache Hudi. In contrast, Delta lake and Iceberg do not enforce primary key constraints, and neither does Clickhouse. So they could have the same bug again due to a bug in feature ingestion - and given they hacked together their feature store, it is not beyond the bounds of possibility.

Reference: https://www.oreilly.com/library/view/building-machine-learni...