calvinfo

A couple of things that really help...

Matrices isn't editable (sort of by design) and works best for columns that are all of the same type. It then uses arrow for fast in-memory analytics.

Google sheets has to serve a much broader array of use cases, so I think they can only do so much to improve the performance. It can't always rely on having consistent rows and columns.

Thanks!

Domain was an opportune buy off Flippa. It wasn't really being used for anything and the seller wanted to get rid of it.

What are you using for the charts? 100k points on a scatter is pretty impressive with SVG.

Visx/SVG right now. You're right that 100k points gets slow on a scatter plot, for those charts in particular some sampling occurs. At some point I'd like to investigate other options, but this works for the MVP.

The colder it gets, the lower the efficiency. In a more extreme climate like Fairbanks, the efficiency pretty much drops down to resistive heating (1.0).

If you're set on electrifying there, it might make more sense to investigate a ground-source heat pump which leverages the ambient underground temperature for heat exchange.

Thanks for giving it a try!

We've tried to make this a little clearer in the "receipt view", but the major difference is in terms of cost of the hardware.

Furnaces, heat pumps, and A/C units all have a 15-20 year useful lifetime.

While the annual savings will be only a bit less, the hardware cost will be _significantly_ less because you only need a one unit for a heat pump vs two for an A/C + furnace combo.

Does that make more sense? We'll try and clarify in the UI

Author here. I'm sad to hear this and want to understand how we can do better.

Do you mind following up over email? I'm calvin at segment.

I don't know a ton about the embedded/IOT world, but it's an area that has been coming up more on my radar recently. This project seems like a super interesting solution to the problems of deployment, updates, and monitoring.

Could you talk a bit more about the focus on linux? How much of the ecosystem is running a full OS vs a small embedded program?

Disclaimer: I’m a cofounder of Segment [1], we build a product to help with these problems.

Given what you’ve shared here, it sounds less like your problems are related to scaling for data volume, and more related to all of the complexity that comes with a data pipeline. Instead of adding a bunch of new components, it sounds like you need just a few.

My concrete advice:

- Standardize and document the collection point for your data. Create a tracking plan which documents how data is generated. Have an API or Libraries which enforce the schema you want. If the sources of data are inconsistent, it’s going to be hard to link them together over time. - Load all of the raw (but formatted) data onto S3 into a consistent format. This can be your long term base to start building a pipeline. And the source for loading data into a warehouse. - Load that data into BigQuery (or potentially Postgres) for interactive querying of the raw data. For your dataset, the cost will be totally insignificant and results should give your analysts a way to explore your data from the consistent base. - Have a set of airflow jobs which take that raw data and create normalized views in your database. Internally we call these “Golden” reports, and they are a more approachable means of querying your data for the questions you might ask all the time. The key is that these are built off the same raw data as the interactive queries.

We use Segment to manage all of the top three bullets (collect consistently, load into S3, load into a warehouse). Then we use airflow to create the golden reports that analysts query via Mode and Tableau. As other commenters have mentioned, there are a number of tools to do this (Stitch, Glue, Dataflow), but the key is getting consistency and a shared understanding of how data flows through your system.

This is a pattern we’ve started to see hundreds of customers converge on: a single collection API that pipes to object storage that is loaded into a warehouse for interactive queries. Custom pipelines are built with spark and Hadoop on this dataset, coordinated with airflow.

[1] https://segment.com

We originally moved to Boston to be close to universities. It seemed like a better location for convincing our professors to use the edtech product we'd started with.

We made the move back to SF to be closer to our customers (since we had shifted to building an analytics product by then). It was much easier to walk them through the product in-person.

I wouldn't say this is strictly necessary today, but SF does have a nice density of startups if you are building a developer tool.

Thank you! I'd agree with the other commenters that Peter's talk is the best place to go for hearing the in-depth story: https://blog.ycombinator.com/peter-reinhardt-on-finding-prod...

I'd say more generally that finding product-market-fit felt much more like a 'pull' than a 'push' motion. We had been trying for 8 months to convince even a single user to rely on the product we were building earlier, and it just wasn't sticking.

When we launched today's product, we started seeing a lot more pull from customers. We solved one problem that other products did not, which prompted a bunch more requests from customers.

Thanks, fixed!

I can't share the exact number, but it's high (above 33%) across all customers on our paid plans.

For companies on our business tier, it's even higher (above 66%). Most of these companies are doing more advanced data analysis, and it's hard to replace the utility of a warehouse for analysts at a certain point.

Ah sorry about that, this is a great point.

I generated these via the seaborn palette in matplotlib[1], but agree that these could be way clearer. I'll see what I can do about changing the line type on them to make them a little bit more differentiated.

[1] https://seaborn.pydata.org/

Thanks, glad you liked it!

1) Honestly, there are a _lot_ more than just three startups here who seem to be growing actively. In general, I tend to bias towards great product quality and design, which puts Vitally, Split, ClearBrain, and Kustomer far up my list (and also a part of the reason we've highlighted them). That said, there are a ton of other companies in the space doing really interesting things.

2) No, at the heart of it, our revenue comes from our happy customers. We don't plan to sell or share our user’s data with other companies the way data brokers do. We just help our customers make better use of their own data for their own businesses. Occasionally, we’ll create trend reports like this that show aggregate trends to help our customers make better decisions.

I'm Calvin, one of the Segment co-founders.

We're excited to share some of our data around the evolution of the growth + saas ecosystems.

There are so many tools out there, that it's often incredibly hard to pick one.

We wanted to share the trends about what we're seeing to make it easier for companies to choose between the thousands of tools on the market. And we wanted to highlight a few of the new startups that we think are poised to grow significantly.

We welcome your feedback here! Deeper analysis to come.

Thank you!

How does a director recover from a failure? From my understanding it would require fetching all job IDs from jobs that are in an active state via the job transactions table (which sounds expensive) and then loading the associated meta-data from the jobs table? Is that correct?

This is correct, if a director crashes for any reason, it needs to scan the database on boot. It is a more expensive operation, which is part of the reason that we try and cap the number of total entries in a given database.

Do you assume that the director will have archived all non-completed jobs when deleting the database? Do you try to gracefully shut down the director first then? From my understanding it seems you perform a "drop table" statement on a given database and then regenerate the tables, but this would require being sure that all the jobs have been processed or archived.

Great question, we glossed over this aspect this a bit in the post itself.

Before a given database is transitioned to the 'spare' state, and its tables dropped, a single Drainer process is responsible for moving any non-completed jobs from that database to another active Director. The Drainer will not successfully exit and transition the database to 'spare' until it is certain it has processed all the non-completed jobs. We never drop any tables which have non-terminal jobs. Similar to the Directors, the Drainer will acquire a lock in consul to ensure only a single process is draining at a time.

We're hoping to go into a bit more depth on how the drainers work and these jobs move around in an upcoming post on Centrifuge's two-phase commit semantics. Ensuring that your data has moved to another system does require fairly complex transactional semantics, so we're hoping to go into depth about how this works.

Hey HN, author of the post here. A number of Segment engineers are hanging around today, and we are happy to answer questions in the comments. Thanks in advance for any feedback and thoughtful discussion!

This comment might be in jest, but we've actually been working on a few different ways to help optimize [1] and manage our AWS spend [2].

That said, there are quite a few places left for us to optimize. We're hoping to share more of the techniques and architecture we've used to get better performance in upcoming blog posts.

[1] https://segment.com/blog/the-million-dollar-eng-problem/ [2] https://segment.com/blog/spotting-a-million-dollars-in-your-...

Hey, I’m one of the Segment founders. I helped instrument a lot of these metrics and built the status page Peter linked to above. I’m hoping I can help shed some light on some of those numbers.

The 44ms you’re referring to is the time it takes our API to respond to an incoming call and ingest an event. It’s certainly not the wall clock time, but it is a good measure of the overall health of the system. Your feedback about its prominence is definitely good–it’s our goal to be transparent, not misleading. We’ll change the area where it’s displayed shortly.

If you’re looking for the ‘end-to-end’ wall clock time, you can find that a little further down the page. For every event coming through our pipeline, we average the time from ingestion to successful delivery and display those metrics on a per-destination basis.

You can see right now that the Google Analytics end-to-end delivery latency is ~400ms within the past hour, and is pretty consistently near that number. We’ve also developed internal systems which break down exactly how many events from a given source were delivered to a given destination, and what the latency distribution for those events was.

The ordering problem you mention is indeed tricky. Like TCP, if we wanted to keep a loose ordering, we’d have to keep a window and ensure the partner API would then re-order messages appropriately. If you want a total ordering, this window of delivery _has_ to be 1 for a given user. It does have some pretty serious implications on the throughput of the system, so we’ve been working with both Mixpanel and Intercom (we’re users ourselves) to try and solve the issue just with timestamps. Ideally, partners would be able to re-order events received based upon time, which is what we do inside of customers’ data warehouses.

As far as the deliverability issues you mentioned, I’m terribly sorry to hear that we failed you here. We’ve hit some scaling bottlenecks that we’ve been working hard to fix–and we do our best to keep the status page updated whenever we have production incidents.

All that said, reliability is our top focus as a company. Teams present their SLA metrics on a weekly basis at all hands, and it’s a key part of our monthly board reporting. We’ll be surfacing these metrics and event traces inside the webapp so as a customer can see exactly where your data is, and what has been delivered. And we’re in the process of building an entirely revamped pipeline that will provide better deliverability guarantees. We plan on sharing the architecture on the blog once it’s all rolled out. Giving you transparency into where your data is stored and how it is processed is exactly what we want to achieve as a company.

If there’s anything I missed–please reach out! I’m calvin at segment.

Segment – Engineering | SF (local)

Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like DigitalOcean, Zenefits, and Atlassian.

Where we're at:

    - built with Go, Node, Kafka, Containers, NSQ, Redis, AWS
    - just over 100 people, and growing actively
    - we <3 open source [1]

Apply: https://segment.com/jobs/

[1] https://github.com/segmentio

Segment – Engineering | SF (local)

Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like DigitalOcean, Zenefits, and Atlassian.

Where we're at:

  - built with Go, Node, Containers, NSQ, Redis, AWS
  - just over 100 people, and growing actively
  - we <3 open source [1]

[1] https://github.com/segmentio

Segment – Engineering | SF (local)

Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like Rdio, Bonobos, and Atlassian.

Where we're at:

- built with Go, Node, Containers, NSQ, Redis, AWS

- a little under 100 people, and growing actively

- we <3 open source [1]

If that sounds interesting, we'd love to hear from you. Apply: https://segment.com/jobs/

[1] https://github.com/segmentio

Totally hear you.

We're planning on open-sourcing some pieces of our Terraform config and service toolkit in the next few months. We're definitely excited to share our internal tooling with the rest of the community.

Segment | San Francisco ONSITE | Engineering and Business roles

Our vision is to build the infrastructure that allows businesses to collect, organize, and understand their data. We launched on HN two years ago with analytics.js, and now tens of thousands of developers send us billions of requests a month. We’re looking for experienced engineers to join our friendly team of 50 in Potrero Hill.

You'd be a good fit if you have expertise in Node, Go, AWS, Docker/containers, Terraform, devops, infrastructure, product management, security or technical sales.

A few things we’re working on:

  * Automating the monitoring of hundreds of integrations with third party services
  * Standardizing the semantics of dozens of different types of events that 
    are sent between integrations
  * Scalable and accessible storage of billions of events from customers 
    and partners that can be queried in SQL

Segment – Engineering | SF (local)

Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like Rdio, Bonobos, and Atlassian.

Where we're at:

- built with Go, Node, NSQ, Redis, AWS

- a little under 50 people, and growing actively

- we <3 open source [1]

If that sounds interesting, we'd love to hear from you.

Apply: https://segment.com/jobs/

[1] https://github.com/segmentio

Segment – Engineering | SF (local)

Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like Rdio, Bonobos, and Atlassian.

Where we're at:

- built with Go, Node, NSQ, Redis, AWS - a little over 40 people, and growing actively - we <3 open source [1]

If that sounds interesting, we'd love to hear from you.

Apply: https://segment.com/jobs/

[1] https://github.com/segmentio

You're right, there is a discrepancy and I didn't think to put it in the article.

I couldn't find the doctor's charges on-hand, but the total cost for the facilities/drugs/equipment was $48,851.82. The insurance ended up paying $14,742.88 of that post-adjustments.

Good point, updated the title to be a bit more clear.

Yeah, we are! (calvin|friends)@segment.io if you want to get in touch.

First off, great article! If you can get away with it, using Gearman is definitely a more robust solution. We'll try and see if we can write adapters for dedicated job queues like it in the future.

We're mainly trying to make the setup process really simple - no need to start up extra job servers or worker computers. Ideally they can add in our small bits of code and then start tracking analytics data without much configuration.

Author here. You're right that if you end up dealing with many requests per second, it will end up being resource intensive. In that case, the file or even a dedicated queue are better options.

We actually queue calls to our API and send only a single request to our servers. Even if you make multiple API calls over the course of a request, there will still be one fork per request.

I'd actually wanted to include UDP, PHP Extensions, and persistent sockets as part of the article, they just ended being a little out of scope. We might try and support them in the future, UDP is definitely an interesting idea for analytics applications.

We may also end up writing a custom in-memory queue which uses a persistent socket. For a vanilla PHP install, these seemed like the best available options.