Webhook Failure Scenarios

https://hermanradtke.com/webhook-failure-scenarios/

by luu • 3 years ago

100 45 3 years ago

leetrout2• 3 years ago

That needs better naming in my opinion.

Just sender and receiver would be enough IMO.

Maybe it is just me but it is unusual to see origin used in this context even if it is technically correct I only ever see it used with CDNs and load balancing.

benburleson• 3 years ago

I’ve always referred to those client/server.

leetrout2• 3 years ago

I think webhooks are even more misleading because they are server to server.

hermanradtke3• 3 years ago

I was frustrated with my naming too.

technically correct I only ever see it used with CDNs and load balancing

That is why I used it!

I responded in https://news.ycombinator.com/item?id=37525971 that maybe I should change it. Sender/receiver is better.

JimDabell• 3 years ago

Some interesting previous discussions here:

- Best Practices for Using Webhooks (stripe.com): https://news.ycombinator.com/item?id=32521159

- Collection of best practices for providing and consuming webhooks (webhooks.fyi): https://news.ycombinator.com/item?id=32517645

- Give me /events, not webhooks (sequin.io): https://news.ycombinator.com/item?id=27823109

losvedir• 3 years ago

Webhooks are fun to think about. A couple more issues off the top of my head:

* Ordering. Since network requests can take variable amounts of time, how do you ensure those two "foo.updated" events are processed in order, or that the receiver can tell their order? Especially something to consider if the webhooks will retry a few times on intermittent failures.

* Consistency. Always a concern in distributed microservice land, but maybe more acute when generating webhooks right as things are updated: if the receiver uses the webhook to make an API request right back into the system, will the API have the same view of the data?

* DDoS. How do you make sure the webhook destination URL is owned by the subscriber? If your system can generate high volumes of webhook traffic, that could be a problem.

* Infinite loops. A silly one, but the user could conceivably point the webhook at a URL of the system that sends the webhook, in such a way that it will cause a new webhookable event to be generated.

sarchertech3• 3 years ago

Ordering is something that everyone always forgets. The only way to actually guarantee they are sent and received in order is to use a lock on whatever subset of messages you care about being ordered. Then only process one of those at a time.

At my last company I built an ordered transaction outbox to handle this.

Each message has a partition key. When the worker picks up a new message to process it locks the partition key, so that it’s the only worker processing any messages with that partition key.

Different messages use different things for their partition key. For some messages where we only care about the order within a given order (it’s an order management system) we use the order_id. For other messages where we care about global ordering we use the tenant_id etc…

You could just use tenant_id for every message, but by using the most granular partition that you can, you can get a lot more parallelism.

foobiekr4• 3 years ago

Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in order. Once you start down that path, you have to be able to answer what you do when the the destination is blocked (for any reason: network congestion or outage, service outage, etc.). This is the path where your internal queue (whether physical or virtual - that is, a cursor) starts requiring potential days of retention and the ability to stream from any point in that history. Been there, done that. You can implement that retention in all sorts of ways (rows in a database, messages in kafka, etc.) each of which has interesting cost dynamics and corner cases.

You are almost better off with a state compressed reporter that has the semantics of always reporting a _sensible_ series of events to the external endpoint where _dependent_ events have sensible relative order ("x was added, x was deleted") that converges on current state no matter how slow the outbound endpoint is and without regard to how many of them there are. No key locking and no buffering, O(1) storage per walker - just a walk on conveying state. This requires careful design of both the system _and_ the schema (boolean state needs to be accompanied with a counter, for example, to convey collapsed/consolidated history of the state changes in this model).

That's a lot more work, but it is possible, this kind of thing dates to the 80s or earlier.

The only in-order/no-drops case that is really common is logs (which is a poor way of conveying event/state tracking) and webhooks are a very poor choice for that kind of thing.

Also, in all cases, receivers should have to accept that they may receive the same notification more than once. Obviosuly there are ways around this, but they are expensive at scale.

dalore• 3 years ago

Could you not timestamp your message and then let the client do the ordering. Granted they might get messages out of order, but I never promised them their webhooks would be in order.

paulddraper4• 3 years ago

At my last company I built an ordered transaction outbox to handle this.

Almost all message queue/event streaming systems support FIFO (SQS, RabbitMQ, ActiveMQ, Kafka, Kinesis).

If you use one of these, it's easy to solve ordering.

tasn2• 3 years ago

Ordering is even more compelx than that! I previously wrote a post about it as it comes up a lot with Svix.

https://www.svix.com/blog/guaranteeing-webhook-ordering/

Semaphor• 3 years ago

I've only ever worked with two webhook systems (Eventbrite and PayPal) and both have those issues handled. I'd say you should just look how others solved it.

sergiotapia• 3 years ago

At the most basic level, what I like to do is create a webhook response controller, the controller enqueues a job and immediately returns an HTTP 200.

The job is saved in my postgres database, using Oban (Elixir) so I have observability, accurate bug stacktraces are saved in the job record's error field, and I can see how many times the job has been attempted.

naikrovek• 3 years ago

right out of the gate: why not use the terms "sender" and "receiver" instead of "client" and "origin"?

the author uses "client" to refer to the application sending the webhook, and "origin" to refer to the application receiving the webhook. this is all backwards to me.

I often say that naming things is hard, but it isn't so hard that this needs be the result.

hermanradtke3• 3 years ago

Fair call out. I responded to similar feedback here: https://news.ycombinator.com/item?id=37525992

JuanPosadas2• 3 years ago

FWIW I wrote a program that used webhooks and RSS feeds and needed to prefix both with their direction, I ended up choosing inbound/outbound.

tasn2• 3 years ago

There are so many more failure scenarios!

I wrote a post about common TLS errors recently: https://www.svix.com/blog/ssl-tls-incomplete-certificate-cha...

nonane• 3 years ago

Any suggestions to alternative to webhooks? I feel a ‘pull’ based model with cursors and long polling would be simpler and more reliable than webhooks.

10000truths• 3 years ago

WebSockets.

JuanPosadas2• 3 years ago

Simple: cron job, one per minute, check for recently created (and/or updated) entries.

andenacitelli2• 3 years ago

Webhooks tend to make a lot more sense in event-driven applications . They introduce additional complexity when it comes to all sorts of edge cases - agree that a pull system is probably much easier to do error handling for, my only concern would probably be performance.

pyrolistical• 3 years ago

The way I solve all these problems is.

Prerequisite: require receiver to be able to process duplicate webhook being resent.

Algorithm outline:

1. Have webhook body stored by sender

2. Attempt to send body to receiver

3. If 2x status returned mark webhook as sent. Done

4. Otherwise for any other status or timeout increment attempt for webhook and try again later until maximum attempts (which then mark webhook as failed)

This covers the tricky case where success response gets dropped in network by having the sender time out and retry later. The receiver gets a duplicate

foobiekr4• 3 years ago

You should describe how you are backpressuring internally while this is going on and also how you send an indicator to the receiver that they encountered a gap.

bennyp101• 3 years ago

Those terms are very confusing - to me orign would be the SENDER ... where the message ORIGNates from ...

But yes, those are common scenarios, and most services that offer calling a webhook have retries baked in for such things. It is the responsibility of the consumer to verify if they have already received a message or not.

You should not be consuming a webhook 'live' and attempting to work on it the moment you receive it - that way lies dragons!

I have dedicated endpoints for services to send to, and they are verified and consumed as quickly as possible and put into a queue to be processed after. That way I can at least see what was sent, and if we missed something.

hermanradtke3• 3 years ago

to me orign would be the SENDER

My history with CDNs is that origin refers to the server sitting behind the proxy. I do agree that

where the message ORIGNates from

makes sense too.

paulddraper4• 3 years ago

A CDN origin refers to the server where the content ORIGINates from.

chunk_waffle2• 3 years ago

For a server sending webhooks to endpoints entered in by users, take care that:

a: The FQDN does not resolve to an RFC1918 address (You don't want to be POSTing payloads to endpoints within your internal network.)

b: If you respond to redirect responses (easier to just not do so for other reasons as well) also make sure those don't resolve to internal addresses too!

foobiekr4• 3 years ago

The attacker-controlled DNS record pointing to an internal private address or an explicit redirect is a classic, especially if they can control the event template being used and the service relies entirely on edge filtering... Too much template control is a risk.

I mean, there's a lot of things you should do when dealing with this that most people don't pay attention to:

https://datatracker.ietf.org/doc/html/rfc2606 https://datatracker.ietf.org/doc/html/rfc3927 https://datatracker.ietf.org/doc/html/rfc4193 https://datatracker.ietf.org/doc/html/rfc6761

... and so on. At least in Go some of the handy checks are simplified by IP.Is(Private|Loopback|Multicast|InterfaceLocalMulticast()|LinkLocal*etc.)

nicolaslem• 3 years ago

It's been years since I've looked into this problem but to tackle it properly one shouldn't just resolve the domain and check that the IP is acceptable. The HTTP client library needs to be involved by providing a way to run code just before creating the socket, which very few do.

chunk_waffle2• 3 years ago

Oh interesting... would love to hear more about that. I guess what could happen is:

You would:

1. Resolve the FQDN

2. check the IP

3. Make the request

When the request is actually made, the FQDN is resolved again and a different IP is returned?

caseysoftware• 3 years ago

Very cool article.

I'm one of the creators of https://webhooks.fyi/ and will do a PR to add this one. Thanks!

andenacitelli2• 3 years ago

Cool site. Seems like a great learning resource, nice work. (No affiliation, just a sucker for good docs)

Webhook Failure Scenarios

Related Stories