adrianco

I pointed my Claude/Retort evaluator at this blog post and it ran an experiment to test what it found. We got better cache hit refill but it didn’t improve the overall coding results on my M5Pro/64GB system. Results in experiment-24 at GitHub.com/adrianco/retort

I’ve talked to the authors of this, it’s a very interesting project. GPU memory space used to be the limitation but the latest generations of GPUs have enormous shared memory capacity and need something like SiriusDB to manipulate and prepare data in-place before the AI algorithms get to work.

I distilled these ideas over subsequent years into several talks on “Failing Over without Falling Over”. Investing anything in resilience without testing that it actually works is a waste of resources. Thats the underlying lesson. https://github.com/adrianco/slides/blob/master/FailingWithou...

All sounds like a good plan, but there’s no easy way to enforce the lack of cycles. I’ve seen helper functions that call a service to look something up, called from a library that is running on the service itself. So a service calls itself. There was probably four or five different developers code abstractions stacked in that loop.

It’s a bubble. Buy low and sell high. Right now, holding cash is the safe option. You can predict that the bubble will burst, but not when. However with the US being run by someone who is poking everything with sharp implements I think it bursts sooner.

Its hard to understand why so many California houses are built of wood. After a fire all that’s left is a few chimneys. However I happen to live in California in a house made of concrete bricks and we don’t get a higher valuation per square foot or a discount for our fire insurance. There should be more incentives to build fireproof houses.

I think there are two things that change. One is that Amazon would be controlling output scheduling of part of the the power station rather than the grid operator. The other is that this would be behind the meter for Amazon like rooftop solar, so would be accounted for differently in the annual report.

This is super interesting and useful. I tried reading the code to understand how GPU workloads worked last year and it was easy to get lost in all the options and pluggable layers.

Someone came up with the name for this: “Disagree and commute”.

I wonder if the other Boars Head plants also have issues. Avoid the entire brand? What brands have better hygiene?

AWS tends to optimize for whatever its customers want, rather than what its partner ecosystem wants. However we did spend some time trying to help open source partners figure out how best to work with AWS and to leverage the marketplace etc. to be successful by leveraging the platform rather than fighting with it. Mongo is a good example of how that can work.

There were several proposals made to Elastic at the time, but their attitude was that they controlled the project and didn’t want AWS to make big contributions to the open source distribution that would reduce their differentiation. They were also mixing licenses in the code base and deliberately making it hard for AWS to use.

I was also assisting in the deal with Grafana, which I thought was a good deal on both sides, setting up a framework for AWS and Grafana to work together over a longer timeframe. Ash Manzani who negotiated the deal for AWS later joined Grafana to run Corp Dev for them.

AWS at the time had AGPL on its list of licenses that couldn’t be used. There were other clouds in China especially ignoring the AGPL provisions and I think SSPL was used to try and be more explicit.

Here’s the initial AWS response to the license change that they made in 2018, which I helped write. At the time we didn’t think a new license made sense, as AGPL is sufficient to block AWS from using the code, but the core of the issue was that AWS wanted to contribute security features to the open source project and Elastic wanted to keep security as an enterprise feature, so rejected all the approaches AWS made at the time. https://aws.amazon.com/blogs/opensource/keeping-open-source-...

I tried to do this a few months ago (for videos not shorts) and found that YouTube will only let you download transcripts for your own uploads. You have to authenticate to the API. Also the transcripts aren’t that great. I ended up downloading the video and running whisper on it.

I was a fly on the wall as this work was being done and it was super interesting to see the discussions. I was also surprised that Jepsen didn’t find critical bugs. Clarifying the docs and unusual (intentional) behaviors was a very useful outcome. It was a very worthwhile confidence building exercise given that we’re running a bank on Datomic…

Not a big surprise. Garman has been working up to this for a while. It will be interesting to see how the VPs reshuffle and whether anyone else who thought they might get the CEO job leaves in a huff…

When we migrated Netflix to AWS in 2009-2011 we setup a separate archive account on AWS for backups and also made an extra copy on GCP as our “off prem” equivalent. We also did a weekly restore from archive to refresh the test account data and make sure backups were working. I’ve documented that pattern many times, some people have even implemented it…

This is a water-cooled system, so AWS is going to have to figure out a new datacenter architecture to install and run it. It’s a big investment and commitment from AWS and I expect that part of the deal was that NVIDIA would be a guaranteed customer at scale for the system, in return for things like AWS getting preferential early access to the technology.

I use OVMS on my 2010 Tesla Roadster and it’s likely a good option for people who don’t mind a bit of DIY to get it installed and to buy a SIM card. It supports most of the functionality of the car so I assume it can do that on a Leaf as well. Leaf is on the supported car list.

Thanks! That’s cool. I’m glad I found time to write down what I remembered and the names of many of the people I met along the way. I have notes but haven’t had time to tell the story of the Netflix years yet…

These were simpler RISC implementations in those days, the compiler optimizer stage was in charge of deciding whether to set the branch bit or not, and the prefetch would do what the bit said, and stall if it got it wrong.

for example the mmap read test used /opt/iozone/bin/iozone -w -s 2G -r 4k -i 2 -B -t 60 -F test0 test1 test2 test3 test4 test5 test6 test7 test8 test9 test10 test11 test12 test13 test14 test15 test16 test17 test18 test19 test20 test21 test22 test23 test24 test25 test26 test27 test28 test29 test30 test31 test32 test33 test34 test35 test36 test37 test38 test39 test40 test41 test42 test43 test44 test45 test46 test47 test48 test49 test50 test51 test52 test53 test54 test55 test56 test57 test58 test59

The point you should take away from this benchmark is that a 3 year old Centos 5 kernel with no tuning, no filesystem tuning and default mdadm options did 100K IOPS or 1GByte/sec pretty much regardless of the benchmark options in use. It just needed enough concurrency to load up the disk. There are many other options on iozone for using different variants of posix IO and threading. I showed the basic options and the mmap option. So with careful tuning you could do IOPS more efficiently, but what I found was very accessible out-of-the-box performance.

"Technical debt" is a nice way of saying it had bugs. It was mostly a configuration problem, if it had been setup better we would have had no outage or a much shorter one. The work to test all our zone level resilience (Chaos Gorilla) was underway but hadn't got far enough to uncover this bug.

Monitoring is done with two systems, one in-house in-band that we might open source one day (was called Epic, currently called Atlas). The other is AppDynamics running as a SaaS application with no dependencies on AWS. There is some useful overlap for when one or the other breaks, we merge the alerts from both (plus Gomez etc) but they have very different strengths as tools.

I ran one of the recommendation algorithm teams for a few years before we did the cloud migration. The techblog summaries of the algorithms are pretty good. The implementation is lots of fine grain services and data sources, changing continuously. Hard to stick a fork in it and call it done for long enough to document how it works.

Visit slideshare.net/netflix and read my architecture slides, there's plenty of detail about how Netflix works available if you have a few hours to look through it.

Some stuff is in real time, some is pre-calculated. There is an enormous amount of research and testing going on in this space all the time, its complex and it's evolving fast.

Netflix is designed to run on two out of three availability zones in a region. There are tens of TB of customer data triple replicated in that region, which has off-region archive but we don't live replicate the data intensive data sources. We also have the Europe region which does live replicate things like membership (since all members are global members of Netflix).

In this case we had some bugs, we should have had a two minute increase in error rate as a third of the clients retried, then the dead instances would have been out of traffic. That's what happened in the previous power outage, where fewer instances went down, and it didn't trigger this bug.