squarecog

Here are a couple of talks about orchestration and choreography, workflows vs sagas, etc.

Both very technical, both with very little "sell" despite being given by a Camunda co-founder and a Temporal principal eng.

https://www.youtube.com/watch?v=zt9DFMkjkEA "Balancing Choreography and Orchestration" by Bernd Rücker

https://www.youtube.com/watch?v=EaBVzjtSK6A "Building event-driven, reactive applications with Temporal: Workflows vs Sagas" by Dominik Turnow

LLMs are Foundational Models, but not all Foundational Models are LLMs.

If you are interested in this space, it does probably make sense to look up what Stable Diffusion (StabilityAI) and Claude (Anthropic) are. Not recognizing them is fine, the pace is pretty incredible in this space and there's lots to learn, but it doesn't make the point you think you are making.

LinkedDataHub, a "RDF-native notebook", is not to be confused with LinkedIn DataHub, which is a metadata store/crawler/ui for your data systems: https://datahubproject.io/.

I should've added, there's an obvious example for the "SaaS control plane" separation, which is equivalent: "stop processing job X that is destabilizing the cluster" should be processed without needing to fight for resources with job X. Same for ACL changes, user deactivations, etc etc. It's generally a good idea to have your control stuff not be subject to whatever instabilities you might be controlling against.

These terms usually show up in the context of networking protocols. Cloudflare has a very quick explainer: https://www.cloudflare.com/learning/network-layer/what-is-th.... To make it even shorter: a control plane is where all the coordination that controls activity (data) happens. The data plane is where the data actually moves around.

Explainers seem to not cover _why_ you would want to separate these "planes". There are several reasons, and I'm no authority, but for starters: * control messages will have different expectations around them: their amount and frequency, delivery guarantees, urgency with which they are processed. Treating this traffic separately means you can engineer appropriately for data and control traffic. * last thing you want is the control message "stop processing traffic from IP x.x.x.x port y" to be stuck behind traffic from said IP/port...

In this context, the meaning is somewhat different. They are referring to administrative traffic vs "actual work" traffic. Auth, billing/accounting, configuration updates, that sort of thing. If you are running a SaaS, and your customer is very security conscious and wants none of their precious data to ever leave their VPC, you have 2 options: deploy your software into their VPC completely, making it hard to do a variety of things like upgrades, and increasing complexity; or you can separate control actions from your "worker nodes" and storage, and only deploy the latter into the VPC. You can then work on your control panels, monitor usage, continuously evolve various admin panels and config options, etc, using normal SaaS approaches while the security conscious customer knows that their core data is not leaving their virtual walls and only "bob ran a thing and stored results" goes to the vendor.

This post is about abstracting out common bits of how one implements that, and allowing SaaS offerings to provide that sort of separation easier.

Now? No idea. At the time I left in 2016, all Storm usage was replaced by Heron, that was the point of making it API-compatible.

I was in charge of the Twitter data platform team at the time we developed Heron and deprecated Storm. The Mesos component of your retelling is not quite right. Take a look at this comment I wrote around the time we started talking about Heron, addressing the same misconception: https://news.ycombinator.com/item?id=10056479

Donated to Apache in 2018. https://blog.twitter.com/engineering/en_us/topics/open-sourc... (and open-sourced in 2016, having started development in 2014).

Why is this odd?

If he started two years later and there was not a trace of the Prize work at the company, that would be an indicator that the competition was not important. If he started and could still see knock-on effects from the competition, that's an indicator that it was important.

Plus, he didn't just start at Netflix. He "took over the small team that was working and maintaining the rating prediction algorithm that included the first year Progress Prize solution."

Yeah, that sounds like he has some authority on the matter.

So basically all research projects have to redirect funds to cover part of the "open access fees"

Yes, the cost of publishing results, as well as review, editing, etc, should be incorporated into the research budget (researchers can of course choose other means of sharing their findings than through publication). In an open access model, the publishing costs are offset by money saved in not paying for access to other articles the research project needs. I would hope that opting out means that projects have to pay fees on articles they'd otherwise get for free through this agreement, rather than riding on everyone else doing their bit.

One would imagine the fee schedule is set in the agreement UCB reached with Springer (presumably the agreement will be published at some point...)

While HDFS is indeed used for exporting old data and storing some partition mapping metadata, it's clear from the blog post that MetricsDB is much more reliant on BlobStore as well as MetricsDB-specific services.

The servers checkpoint in-memory data every two hours to durable storage, Blobstore. We are using Blobstore as durable storage so that our process can be run on our shared compute platform with lower management overhead.

I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang.

There's a ton of projects Apache Foundation hosts that fit your description but it's a mistake, I think, to confuse individual projects with Apache in general. Bad enough that people confuse the license with the foundation.

Can you explain which part, in your opinion, is facepalm worthy?

In your example conversation, the developer starts off by demonstrating that they are not good at thinking through how long things take them, by giving an impossibly optimistic estimate. Even the PM, whose job is not to figure out how to build stuff or how long building stuff takes, knows the estimate is wrong, and pushes back on it, pointing out dependencies the Dev appears to not have considered.

The dev comes back with another estimate, also provided on the spot, without actually thinking through all the tasks, problems, and potential ways to streamline / accelerate. The PM, having just had the dev's lack of thorough thinking with regards to complexity estimation amply demonstrated, asks the Dev to think more carefully, a third time.

Are we facepalming at the dev who can't be bothered to actually consider what it will take to get the thing done, or the PM who for some reason accepts the 3rd estimate, even though the dev appears to just be randomly generating numbers? Both?

I helped build Twitter's data platform, 2010-2016.

There isn't an "analysis server" and analyzing user activity is not done on a "user database backup" at Twitter's scale, though indeed that's a common way that would be done for smaller businesses.

By the way, if by user db you literally mean the db with user accounts, that's not the right data source -- you want the user _activity_ db to count active users, and for high-scale applications, those are different things. Presumably user activity updates are orders of magnitude more frequent than user object updates. You don't want to thrash your user db by constantly updating some "last seen at" field. Put that stuff somewhere else.

That said, it's true that counting is simple, it's just a Hadoop / Spark / distributed computing platform of choice job. Filter, distinct, count. It's not even hard in real-time if you have enough ram or are ok with approximate counts with bounded error, thanks to Storm, Heron, Flink, etc.

Defining what exactly constitutes an active user and catching edge cases such as this Digits thing is where things get tricky; the number of weird scenarios that cause under/overcount for what seem like reasonable and straightforward definitions would surprise you.

@baddox nailed it.

(ex-twitter engineer) I left before this project got started, and do not have any insider info on how they did it. Given what I know about the number of places tweet length assumptions were built into, it must have been a large, cross-team effort. It likely required thoughtful problem-solving.

Take, for example, search. An early iteration of Twitter search relied on this limit to pack term positions into 8 bits (Source: https://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal...). 280 > 256, so if this was still around, the whole approach had to be rebuilt, and the indexes recreated. That's ... non-trivial. And that's just one subsystem.

hdf5, Feather, Arrow, protobufs, json, xml -- all solve the problem of binary representation of data on disk. They all leave the question of how to map said data to a specific problem domain up to the developer.

Projects like ONNX define said mapping for a specific domain (in ONNX's case, by agreeing on a proto schema for ML models, and its interpretation).

To use a simplistic metaphor: protobufs are the .docx format; onnx is a resume template you can fill out in Word.

Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise, effectively, by looking at the p-value and stopping when it hits 95% confidence, what you are doing is ignoring all negative results and accepting the first positive result you see -- clearly, that's bad science. I'm simplifying the math here, but this is the general notion.

You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).

You may be interested in what Wes McKinney has to say on the topic: http://wesmckinney.com/blog/outlook-for-2017/ (you can read/write using pyarrow)

Yes: https://twitter.com/aneeshs/status/773547228694589440

"We use GraphJet for serving contextual, personalized content recommendations in real time, such as "You May Like", emails, notifications etc"

Hi, I manage the data science team at Zymergen, and I can tell you we are working on these topics (and hiring). Shoot me an email if you'd like to chat -- dmitriy at zymergen.com

Note how the creator of Apache Crunch is on the Dataflow committer list.

co-author of blog post here.

We covered some of this, at a high level, here: https://blog.twitter.com/2015/the-what-and-why-of-product-ex...

Certainly not everything needs to be A/B tested -- depending on the nature of the change and the kind of insight one is looking to get, MAB may be better, if it applies, or a number of other approaches can be more beneficial, including not running an experiment at all. After all, Twitter itself came from a side project at the podcasting company Odeo. If Twitter was a/b tested, I'm sure it wouldn't move any podcasting metrics in a meaningful way :).

I'm obviously biased -- but I've seen really important insights gained from A/B tests, and wouldn't dismiss them too easily.

Co-author of the Twitter post here.

I don't necessarily disagree with you! Though I hope you are not calling Twitter a fly-by-night marketing startup :).

As a company, we've found A/B testing very valuable, but it certainly has limitations, and pitfalls that one must be fairly experienced to avoid. You can see some descriptions of what we do to achieve that in other posts in that series. But it's fair to say that this takes some work (like the work described in the "second control" post) to avoid badness, and it's fair to question if other methods can get us to the same end result -- ability to evaluate impact of proposed product changes -- cheaper, or more accurately.

Hi, author of the post here.

A few brief points:

1. The post talks about most experiments not moving core metrics in a large way. This does not mean the experiments don't ship, it just means core metrics are very hard to move. So "few experiments actually succeed to turn into new features" is not entirely correct -- quite a few of them do ship because they improve a particular feature, or because the team believes they are the right thing to do as long as they don't do any harm, or even an acceptably small amount of harm.

2. If the majority of the experiments worked, we wouldn't need to experiment, would we? :-).

3. Core metrics being hard to move is not a Twitter-specific thing. If you find them easy to (positively) move, you are either so tiny that any change is a large change, or you are measuring something incorrectly. LinkedIn, Bing, and Amazon experimenters contributed to a paper, linked from the post, which states as "rule of thumb #2" that "changes rarely have a big positive result on key metrics." Anecdotally, the same is true for Google, EBay, and Pinterest.

4. Analysis paralysis, according to wikipedia, is "an anti-pattern, the state of over-analyzing (or over-thinking) a situation so that a decision or action is never taken, in effect paralyzing the outcome". Deciding to not ship something is not analysis paralysis. It's a valid outcome. If everything you try works, you are probably not trying hard enough. Now, if you ran an experiment and kept saying there is not enough data to make a decision, and kept analyzing the data and collecting more inputs -- that would be analysis paralysis.

S3 has a number of challenges of its own, for example, file access is quite expensive (performance-wise). The Netflix engineering team has a ton of experience running data processing "in the cloud", and have their share of war stories.

The scaling challenges the Twitter blog post addressed (disclaimer: I work at Twitter, was one of the first Hadoop people at the company, am now doing slightly different things) happen at fairly extreme scale ranges. The NN design leaves much to be desired, but it does work just fine in the vast majority of cases. The scaling challenges we are talking about involve thousands of nodes and hundreds of petabytes of data. This is not what one normally designs for, even in a "big data" system. Take this into consideration when exploring alternatives. Do you have a solid way of managing a few hundred petabytes and a few hundred million objects in S3? Does GlusterFS work at that scale?

[disclaimer: I managed the data platform group, of which the storm/heron team was part, when heron was developed, and I'm still a Twitter employee; however, naturally, this is all my opinion and not an official position of my employer, etc etc].

Hi Andrew,

The Heron rewrite had more to do with what were at the time gross operational inefficiencies of Storm at very high scales, and problems diagnosing failures and bottlenecks. This is Storm 0.9 -- in the years since, I think Storm community in general and the Yahoo folks in particular have been working hard on addressing some of those issues, and a recent blog post from them indicated that some of the stuff we fixed in Heron is on their roadmap. Note that the Heron paper was published a year or so after the first Heron topology went into production inside Twitter. We had a very real problem that we needed to fix very quickly, and writing Heron was faster than making Storm work. Some of that was due to OSS challenges, some due to Storm's architecture fundamentals, some due to the specific people and backgrounds we had on the real-time compute team at that point.

The scales I am talking about are hundreds of nodes, not dozens, and an order of magnitude more messages per second. I am not surprised it works perfectly well for your use case (it worked fine while we were only putting tweets into it in 2013, as well -- that was about the size you are quoting, iirc).

Mesos not only existed when Storm was written, Storm ran inside Twitter on Mesos since before Storm was open-sourced. Mesos went into Apache incubator in 2011, while Storm did so in 2013. I'm not sure why you got the impression any of this was related to Mesos; it's true that we simplified a lot of operational complexity for Storm+Mesos by not writing our own Mesos scheduler, like Storm did, and just using Apache Aurora -- a decision that also meant we were able to use Aurora/Mesos clusters shared with other processes, which was nice; but that wasn't the prime motivation by a long shot.

The API was kept as a trade-off, to make migration of internal customers seamless. There are problems with the API, particularly around back pressure, but at the same time it was a straightforward API that got wide internal adoption, both in raw form and through Summingbird. So it made sense to evolve that part incrementally and deliver our internal customers the performance, observability, and reliability wins first, without having them rewrite a line of code, and tweak APIs over time to address the above-mentioned issues.

I'm biased of course, but I think our track record with open source contributions is still pretty good -- Scalding, Parquet, Aurora, Mesos, Finagle, Zipkin, and many more major projects with very wide adoption in the industry, which we still very actively contribute to and continue to evolve. At the same time, there's definitely a cost to open-sourcing projects, and those decisions get made on a case by case basis.

Good post explaining table stakes for reasoning about whether your site search works (and measuring whether changes make it work better).

If anyone is interested in having this but not rolling their own, www.switfype.com (ycombinator 2012) provide a pretty powerful site (and mobile app, etc) search solution, which includes these kinds of analytics and more (disclaimer: the founders were my roommates :-)).

Well, first, a lot of this stuff is actually written in Scala, which is like Java's younger and more fun cousin. Still runs on the JVM but has many fun language aspects I won't get into because there's already a billion posts online about that.

But second, not quite a "platform" but I've been enjoying Frank McSherry's blog posts in which he builds up a complete distributed dataflow engine in Rust: http://www.frankmcsherry.org/

As pointed out by another commenter, Cassandra is not a column store. Its storage format is quite inefficient for compression and for large scans of subsets of columns, which is what columnar stores are designed for (Cassandra and HBase optimize for different use cases, which are much closer to OLTP).

You can find a brief description of why and when you want actual columnar formats in the introduction section of this blog post: https://blog.twitter.com/2013/dremel-made-simple-with-parque... .

You can read more about columnar databases here: https://en.wikipedia.org/wiki/Column-oriented_DBMS

A columnar storage format for a database engine that does not have operations specifically tuned for vectorization and taking advantage of data being cheaply available a column chunk at a time instead of a row chunk at a time is only half the battle -- you can get massive perf wins on top of the work CitusDB did by doing things like operating on compressed data / delaying decompression, avoiding deserialization for column values based on a filtering scan of another column, pipelining operations on column values to fit better into your cpu architecture, etc.

But just having the IO wins is nice, too.

If you are linked to 100 people on your LinkedIn account, and 10 of them have the same set of people in the address book they imported into LinkedIn, it's not hard for LI to figure out you might also know people in that overlapped set, and recommend them. You don't need to import your address book for this to work -- just enough of your connections, who have connections in common with you, need to do it.