HN user

bothra90

55 karma

[ my public key: https://keybase.io/bothra90; my proof: https://keybase.io/bothra90/sigs/GwHhfZNqVUv0D07d2-mE7ARAXuPfuEGgp0fmCxx7orc ]

Posts21
Comments20
View on HN
fennel.ai 3y ago

How we built a vector search engine in 200 lines of Rust

bothra90
2pts0
lethain.com 3y ago

Micromanagement (sometimes) good?

bothra90
2pts0
benn.substack.com 4y ago

The case for consolidation

bothra90
39pts11
blog.fennel.ai 4y ago

ML behind recommendation systems at FAANG

bothra90
5pts0
benn.substack.com 4y ago

The Case for Consolidation

bothra90
3pts0
doordash.engineering 4y ago

Sibyl – Prediction Service at DoorDash (2020)

bothra90
27pts36
medium.com 4y ago

Big Data Serving – The Hardest Problem in Computing

bothra90
1pts0
doordash.engineering 4y ago

Sibyl – Prediction Service at DoorDash

bothra90
3pts1
github.com 4y ago

Merlion – A new time-series ML framework from Salesforce

bothra90
7pts0
dreamsongs.com 5y ago

The Rise of Worse Is Better – Richard P. Gabriel (1991)

bothra90
2pts0
www.draconianoverlord.com 5y ago

Dislike for Pull Requests

bothra90
1pts1
medium.com 5y ago

Model Portability with OctoML/Apache TVM

bothra90
2pts0
www.notion.so 6y ago

Handling overload with batching and rate-limiting

bothra90
1pts0
aws.amazon.com 6y ago

Avoiding Fallback in Distributed Systems (2020)

bothra90
2pts0
www.notion.so 6y ago

Zookeeper Pitfalls

bothra90
3pts1
www.darpa.mil 6y ago

A different way of thinking about resilience [DARPA]

bothra90
2pts0
safesmart.co.uk 6y ago

Open vs. Closed Protocols (2015)

bothra90
3pts0
www.notion.so 6y ago

Why I loved reading the io_uring paper

bothra90
2pts0
www.notion.so 6y ago

Not All Queues Compose

bothra90
3pts2
www.youtube.com 10y ago

Thinking above the code

bothra90
2pts0
www.slacker.com 13y ago

New Slacker Radio

bothra90
1pts0

Exciting work! Some questions:

1. Any plans on open-sourcing this?

2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases.

3. Related to 2, how does WarpStream compare to tiered storage in Pulsar?

[Edit 1] Added (3)

I really wish more design docs / project reports looked like this article - talking about the whys, goals / non-goals, options considered, prototypes built, challenges, and solutions, learnings and next steps. Kudos!

Although GitHub supports the commit-based review the author is arguing for (they found it too after writing the article), I don't think it completely solves the problem - we need to be able review, test, and merge each commit individually without needing to merge the entire PR. Since GitHub doesn't have first-class support for dependent PRs, we are still "stuck" with a broken workflow. An example workaround: https://wchargin.github.io/posts/managing-dependent-pull-req.... Are there any better solutions?

Zookeeper Pitfalls 6 years ago

I've wanted to write this blog for around 2 years now, but was only now able to get it out. Feedback is welcome.

The fact that PMs are empowered to be "CEOs" of their product probably doesn't help. In my mind, transparency in the 2-way communication between PMs and engineers is key. Engineers need to be motivated by the "why", but PMs also need to understand why an engineer might want to, to quote from the article, "spend months refactoring an app to make it “right”".

The design choice of hiding the entire network behind a single end-point is interesting. As mentioned in the blog, it forces pushing down complex routing logic into the networking module. I'm curious about what other options were considered and why this approach was picked.

I've wondered the same thing about the effect of slow-accelerating trucks on stop-and-go bay area freeway traffic during rush hours. I feel that the effects are fairly non-linear as more of these vehicles are introduced. Some cities in India don't allow heavy vehicle traffic on congested roads during peak commute hours, though I'm not sure if there's been a study of traffic patterns that validates/simulates these scenarios.

The author is arguing that we can always rewrite the txn logic such that (at least in the deterministic case) if one worker decides to commit, all others will do so too. So a worker can proceed independently without other workers needing to finish their share of work and entering the commit protocol. The article doesn't go into exactly how this could work in the case of non-deterministic txns, but the linked papers may have an answer.