fullung

Albert here :) Decision basically came down to the following: we needed to build a 1 million events/sec log processing pipeline, and had only 5 servers (more were on the way, but would take many months to arrive at the data center). So v1 of this pipeline was 5 servers, each running Kafka 0.8, a service to receive events from the edge and put it into Kafka, and a consumer to aggregate the data. To squeeze all of this onto this hardware footprint, I spent about a week looking for a format that optimized for deserialization speed, since we had a few thousand edge servers serializing data, but only 5 deserializing. Capnproto was a good fit :)

Would love to see results from incorporating https://github.com/segmentio/encoding/tree/master/json!

Segment | Engineering | San Francisco, CA & Vancouver, BC | Full-Time | Onsite + Remote for senior engineers

Segment is building the customer data platform for everyone. We transform data and route to over 200 different integrations, adding new ones every day. We're processing billions of events daily and maintain the analytics infrastructure for companies like DigitalOcean, Instacart, New Relic, and Docker. Our goal is to help companies learn from how their users interact with the products to build even better products. We also like to share our work and what we learn, here are some recent examples:

- https://segment.com/blog/the-tools-of-today-arent-the-tools-... - https://segment.com/blog/show-hn-to-series-d/ - https://segment.com/blog/we-test-in-production-you-should-to... - https://segment.com/blog/secure-access-to-100-aws-accounts - https://segment.com/blog/why-status-codes-matter-in-data-del... - https://segment.com/blog/goodbye-microservices - https://segment.com/blog/2018-in-review/ - https://open.segment.com/

We have a modern stack consisting of Go (golang), AWS ECS/Kubernetes, Docker, JavaScript, React, GraphQL, Kafka, and others! If any of this sounds interesting, we'd love to hear from you! Check out our open positions at https://segment.com/engineering#jobs If you have specific questions, feel free to reach out to marta (at) segment.com (but please do apply on the site).

On this topic, shout-out to ykman oath (if you have a YubiKey).

Segment | Engineering | San Francisco, CA | Full-Time | Onsite

Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like DigitalOcean, Atlassian, New Relic, and Docker.

Our goal is to help companies learn from how their users interact with the products to build even better products. We also like to share our work and what we learn, here are some recent examples:

https://segment.com/blog/the-right-way-to-manage-secrets/

https://segment.com/blog/exactly-once-delivery/

https://segment.com/blog/a-brief-history-of-the-uuid/

https://open.segment.com

If any of this sounds interesting, we’d love to hear from you! Check out some of our open positions at https://segment.com/jobs or email tido.carriero@segment.com with some info about your background and we’ll kick it off from there.

Not rolling our own exactly: running on ECS now and busy evaluating Kubernetes. These are still quite blunt instruments though.

At some point, as a company grows, it's not just about running some containers. You want batch jobs, complex placement constraints, scheduled tasks, etc., etc.

If you haven't watched through https://youtu.be/4OLlKGT7aVQ, I can recommend it. aspyker does a great job of enumerating many of their complex use-cases.

The Titus team at Netflix is building this on top of EC2/Mesos and ECS.

The rest of us still need to do a bit of work to find/adapt/build our own.

I'll happily create a redirect once you build that for us. :)

IAM Roles for Tasks is one of the better examples on AWS: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/t...

This thread has some of the motivation for why Titus is exciting in my opinion: https://news.ycombinator.com/item?id=14526452

Kubernetes, Mesos/DC/OS and the various other options are all valid solutions if you need to run in many diverse environments. Titus runs (ran?) on top of Mesos before they built an ECS integration.

That being said, there's interesting benefits to the approach of not trying to abstract the IaaS and deeply integrating with one or two clouds instead. You get to leverage much more of the cloud(s) you've decided to adopt if you don't immediately try to abstract them away.

This site was mostly created as a joke after I watched a couple of @aspyker's great presentations about Titus and realized that we have a huge amount of exciting work to do to get our own container platform on AWS to the same level.

Netflix should do what makes sense for them. I understand that doing a good job of releasing a very complex piece of software as open source is hard and takes a lot of effort.

DC/OS is an open source solution: https://dcos.io/