Sinjo

Fixed! I grabbed that example from the bug tracker issue where the change was discussed and that wasn’t based on a version where the backtick change had been made.

Ooh, I hadn't heard of rbs-inline before. I was always a bit put off by the type definitions having to live in separate files.

Oh nice! I haven't used Sorbet yet, but this is the second time I've heard good things about it outside of Stripe (where it originated). I'll have to give it a proper look.

By coincidence I just ordered my first PC Engines computer to use as a router last week.

Glad I got in while they're still being made. It seems like an underserved niche.

Honestly state machines are fantastic in Rails too. My last company built Statesman (https://github.com/gocardless/statesman/) and being able to lean on it to prevent you getting into invalid states is fantastic. You also get the bonus of tracking the history of states your resources went through (which is especially useful when you're dealing with payments).

At some point you'll have to think about query performance on the state transition table, but it'll go further than you think and is firmly in the realm of problems of success.

Ah that makes sense! Having current in opposite directions in the two wires in the cable cancels it out. Thanks for explaining.

I guess the warnings about not running loads of current through coiled cables are more to do with heat dissapation then?

Doesn't coiling cables while passing increasing levels of power through them (which is exactly what's happening with new USB-C PD specs) start giving off more and more heat?

EDIT: Nope! See the replies.

We stopped using HDDs and CRTs, and started using SSDs and LCDs.

Pretty much number 1 on my Postgres wishlist.

Thanks for the extra info, and the insight into how you're using Patroni. Always helpful to hear about someone using it for real, especially someone who's come from Pacemaker. :)

It's a little lost in another comment thread (https://news.ycombinator.com/item?id=15862584), but I'm definitely excited about solutions like Patroni and Stolon that have come along more recently.

Oh for sure!

My claim is that you need to hire some expensive people if you want that performance, not that there aren't reasons to run your own database instances!

Right!

A thing that scares me is anyone saying they're running their own HA cluster (not single instance) for cost reasons. Infra people are not cheaper than the hosted solutions (Amazon RDS, Google Cloud SQL, Heroku Postgres).

1. Unfortunately the logs don't give any detail there. Most likely something arrived down the replication connection that the process couldn't handle, and it crashed.

2. Our understanding now is that INF is the strongest preference, whereas -INF is a veto. It would be very cool to have this confirmed 100% by someone who works on Pacemaker!

To be honest we've not looked into Citus in any depth.

My early impression of it (can't speak for the rest of the team) was that it was mostly aimed at sharding analytics workloads, but parts of the docs (e.g. https://docs.citusdata.com/en/v7.1/admin_guide/cluster_manag...) make it sound like it handles OLTP workloads too.

Maybe I've been ignoring it for bad reasons!

EDIT: Managing Postgres clusters is something that a lot of people are working on. Thought I'd mention two projects that have me excited right now:

  - Patroni https://github.com/zalando/patroni
  - Stolon https://github.com/sorintlab/stolon

We plan to open source it as soon as we can. Tiny bit more work to do, then review from a couple of people in the team, then we make it public. :)

Or any of the layers they run above the block device could be designed to detect it, or even correct it.

GoCardless (YC S11) | London | SRE, Data, Backend and Frontend Engineers | Onsite | Full-time | Visa

GoCardless is building a payments network for the internet. Since 2011 we've been focused on simplifying Direct Debit for small and medium companies (who previously had no access to it) and we're now expanding to serve the largest companies (think newspapers, utilities) and connect with existing payment systems in countries all over the world. We already support the UK and Europe and are aiming to expand to more countries over the next year.

As an engineering team at GoCardless we care most about stable, reliable, understandable code. We rely on testing and code review and a culture of frequent constructive feedback. We define and manage our own roadmap and run projects in whatever way works best for us.

Our stack: Rails, React, Postgres, Elasticsearch, Docker, Chef. We also have a bit of Go and Python knocking around.

We love learning new things and contributing back to the community. We open source everything we can[1] and regularly host meetups and hackathons at our wheelchair-accessable office in Angel. We have a weekly bookclub within the team and give internal (and external) talks about things that interest us.

Interview process: an intro call, one technical video interview, then a couple of onsite interviews (coding exercise and some chats - no whiteboards!)

For more info and to apply: https://gocardless.com/jobs. If you've got any questions, drop me an email (it's in my profile).

[1] Notable examples are Statesman (https://github.com/gocardless/statesman) and Coach (https://github.com/gocardless/coach)

That's good to know. Thanks for the explanation!

Not currently useful to me, but I'm sure this hits the sweet spot for a bunch of people.

It's been a long time since I looked into this: is there now a way to configure a cluster of Redis instances such that you won't lose messages on node failure? If not, all the nice at-least-once delivery (or "effectively once" when you add message dedupe) you get with something like Kafka/Kinesis/GCP PubSub is gone.

If not, either people's messages don't matter /that/ much (which is fine, just not great for most of my usecases at the moment) or everyone's in for another round of "oh shit, where did the data go?"

Edit: Just in case we end up in CP vs AP datastore wars, please go read https://martin.kleppmann.com/2015/05/11/please-stop-calling-...

At-least-once delivery requires neither CAP consistency (linearisability) nor CAP availability (any non-failed node must return a response in a non-infinite time), but is a very useful property!

GoCardless (YC S11) | London | SRE, Data, Backend and Frontend Engineers | Onsite | Full-time | Visa

GoCardless is building a payments network for the internet. Since 2011 we've been focused on simplifying Direct Debit for small and medium companies (who previously had no access to it) and we're now expanding to serve the largest companies (think newspapers, utilities) and connect with existing payment systems in countries all over the world. We already support the UK and Europe and are aiming to expand to more countries over the next year.

As an engineering team at GoCardless we care most about stable, reliable, understandable code. We rely on testing and code review and a culture of frequent constructive feedback. We define and manage our own roadmap and run projects in whatever way works best for us.

Our stack: Rails, Angular, Postgres, Elasticsearch, Docker, Chef. We also have a bit of Go and Python knocking around.

We love learning new things and contributing back to the community. We open source everything we can[1] and regularly host meetups and hackathons at our office in Angel. We have a weekly bookclub within the team and give internal (and external) talks about things that interest us.

Interview process: an intro call, one coding challenge, then a couple of onsite interviews (pair programming and some chats - no whiteboards!)

For more info and to apply: https://gocardless.com/jobs. If you've got any questions, my contact methods are in my profile.

[1] Notable examples are Statesman (https://github.com/gocardless/statesman) and Coach (https://github.com/gocardless/coach)

GoCardless (YC S11) | London | Frontend, Backend, SRE | Onsite | Full-time

GoCardless is building a payments network for the internet. Since 2011 we've been focused on simplifying Direct Debit for small and medium companies (who previously had no access to it) and we're now expanding to serve the largest companies (think newspapers, utilities) and connecting with existing payment systems in countries all over the world. We already support the UK and Europe and are aiming to expand to more countries over the next year.

As an engineering team at GoCardless we care most about stable, reliable, understandable code. We rely on testing and code review and a culture of frequent constructive feedback. We define and manage our own roadmap and run projects in whatever way works best for us.

Our stack: Rails, Angular, Postgres, Elasticsearch, Docker, Chef. We also have a bit of Go and Python knocking around.

We love learning new things and contributing back to the community. We open source everything we can and regularly host meetups and hackathons at our office in Angel. We have a weekly bookclub within the team and give internal (and external) talks about things that interest us.

Interview process: a couple of phone screens, one take home test, then a couple of onsite interviews (pair programming and some chats - no whiteboards!)

For more info and to apply: https://gocardless.com/jobs. If you have any questions, drop us a line (jobs at gocardless dot com).

Or if you prefer slides: https://speakerdeck.com/sinjo/lessons-learned-the-hard-way-p...

Thanks! Glad you enjoyed it.

The main thing that narrowed it down was thinking about the request path from the app server to Elasticsearch. That pointed us at the load balancers (we tried a request directly from app -> Elasticsearch in production to verify they were the problem).

You're right that HAProxy wasn't the first place we looked once we were there, though. If I remember right, we started by diffing `sysctl` output to see if Ubuntu had tweaked something between versions.

That was a fabricated example, rather than something plucked from our codebase. It's the sort of query we avoid running on our production database.

I agree with your approach though. Our data analysis runs elsewhere.

You're right on the approach. One thing to be careful of is that there's no option to add a not-null constraint with the `NOT VALID` flag. That means it'll hold the AccessExclusive lock while it checks every row in the table meets the constraint.

You can work around this by adding a `CHECK` constraint which does the same thing, but it's a little inconvenient. There was some work being done to add this feature to not-null constraints (http://www.postgresql.org/message-id/20140517155857.GD7857@e...), but unfortunately it looks like it's gone stale.

We can't count on a user (either in the browser or using the API) waiting around that long. Even some of our own services time out more aggressively than that.

Those situations can be smoothed over with retries, but we try pretty hard to avoid delays like this in the first place.

Spot on. We ran the migrations against production-sized data beforehand, and everything was fine. We did it again after the downtime, and got the same results.

Most of our queries are fast (tens of milliseconds or lower), which is how we got away without knowing about this for so long. Unsurprisingly, we've been making a bigger effort to eliminate any slow queries we do find lately. ;)

allow_mult has been enabled by default since at least 2.0 - http://docs.basho.com/riak/latest/dev/using/conflict-resolut...

I wrote a little follow up to the article, covering some of the points mentioned here and on reddit. http://blog.sinjakli.co.uk/2011/02/15/email-address-validati...