Matthias247

When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with.

The report actually seems to confirm this - it was indeed a crash on ingesting the bad config. However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".

The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.

As far as I remember from building these things with others within the async rust ecosystem (hey Eliza!) was that there was a certain tradeoff: if you wouldn’t be able to select on references, you couldn’t run into this issue. However you also wouldn’t be able run use select! in a while loop and try to acquire the same lock (or read from the same channel) without losing your position in the queue.

I fully agree that this and the cancellation issues discussed before can lead to surprising issues even to seasoned Rust experts. But I’m not sure what really can be improved under the main operating model of async rust (every future can be dropped).

But compared to working with callbacks the amount of surprising things is still rather low :)

Some other material that has been written by me on that topic:

- Proposal from 2020 about async functions which are forced to run to completion (and thereby would use graceful cancellation if necessary). Quite old, but I still feel that no better idea has come up so far. https://github.com/Matthias247/rfcs/pull/1

- Proposal for unified cancellation between sync and async Rust ("A case for CancellationTokens" - https://gist.github.com/Matthias247/354941ebcc4d2270d07ff0c6...)

- Exploration of an implementation of the above: https://github.com/Matthias247/min_cancel_token

I've played that one a lot (hundreds of games on Evil difficulty). So far there's no guessing required after the initial click. Everything can be deducted. Even though some deductions are really hard (you have to go through mine blocks from multiple starting points and multiple hops in order to determine the next safe block).

Clarification question:

When a NOTIFY query is issued during a transaction, it acquires a global lock on the entire database (ref) during the commit phase of the transaction, effectively serializing all commits.

It only serializes commits where NOTIFY was issued as part of the transaction, right? Transactions which did not call NOTIFY should not be affected?

I think the only guessing situation in the game is the starting point (first click). after that, everything can be solved by deduction.

How does it work technically?

Does Whatsapp expose these messages via an API? If yes, then it seems like this is not only on Google.

If no: Are they reading data from raw UI widgets? Are they intercepting input controls? Are they intercepting network traffic? That seems unlikely, given its probably end to end encrypted and the decryption happens within the scope of the Whatsapp process.

I'm wondering too.

In general this seems to work only if there's a single offline client that accepts the writes.

With limitations to the data scheme (e.g. have distinct tables per client), it might work with multiple clients. However those would need to be documented and I couldn't see anything in this blog post.

Besides whats called out by others: It would be nice to have the usual action of "press both mouse buttons" on a number to open all non flagged fields around it. That reduces the amount of required clicks a bit.

But maybe its intentional to avoid one player scoring too many points at once?

If you want to stream data inside a HTTP body (of any protocol), then the ReadableStream/WritableStream APIs would be the appropriate APIs (https://developer.mozilla.org/en-US/docs/Web/API/Streams_API) - however at least in the past they have not been fully standardized and implemented by browsers. Not sure what the latest state is.

WebTransport is a bit different - it offers raw QUIC streams that are running concurrently with the requests/streams that carry the HTTP/3 requests on shared underlying HTTP/3 connections and it also offers a datagram API.

Duplex streams are not really a HTTP/2-only feature. You can do the same bidirectional streaming with HTTP/1.1 too. The flow is always: 1. The client sends a header set. 2. It can then start to stream data in the form of an unlimited-length byte-stream to the server. 3. The server starts to send a header set back to the client. 4. The server can then start to stream data in the form of an unlimited-length byte-stream to the client.

There is not even a fixed order between 2) and 3). The server can start sending headers or body data before the client sent any body byte.

What is correct is that a lot of servers and clients (including javascript in browsers) don't support this and make stricter assumptions regarding how HTTP requests are used - e.g. that the request bytes are fully sent before the response happens. I think ReadableStream/WritableStream APIs on browsers were supposed to change that, but I haven't followed the progress in the last few years.

NGINX falls into the same category. It's HTTP/2 support (and gRPC support) had been built with a very limited use-case in mind. That's also why various CDNs and service meshes use different kinds of HTTP proxies - so that various streaming workloads don't break in case way the protocol is used is not strictly request->response.

If this is mainly useful for working with plain/uninterpreted byte arrays, then I wonder why we can't just do `[u8; N]::with_noinit()` method instead of doing the multi-line plus unsafe things listed in the article.

Is the main point that things like `slice_freeze_mut` could also be used for slices of e.g. `struct Coordinate { x: u32, y: u32, z: u32 }`?

It would obviously not work for f64 things, since there also not all bit-patterns are valid.

They still have windows that you can open

Are there any numbers/data for the quality of healthcare in the provinces over time?

At least for housing I see the average home prices in each category (condos, townhomes, detached) still higher in Vancouver than Toronto (when googling a bit, I found https://wowa.ca/reports/canada-housing-market with some data. But there's probably lots of real estate related sites with more).

io_uring reduces the overhead of system calls - but it doesn't do anything to reduce the overhead of the actual networking stack.

If your send/receive calls spend most CPU time in going through routing/fragmentation/filter/BPF/etc path in the networking stack, then uring (or other APIs which just reduce the system call overhead, like SendMmsg/Recvmmsg for UDP) might only make a small difference. Source: Lots of profiling while implementing QUIC libraries.

An alternative to DPDK that allows to bypass the kernel networking stack would be AF_XDP.

I agree that its a very subpar solution for programming, unless you value a lot of screen estate and can deal with tiny text.

4k/5k 27" screens with appropriate font scaling make text so much easier to read and are thereby much easier on the eyes. And they still have plenty screen estate (having 2-3 files open next to each other is no problem).

I actually didn't understand it being phrased as a limitation. It could also be a feature - maybe one would prefer to look at logs for multiple services within a single query?

Anyhow, the nice thing about the system is that one can get anything that is preferable as long as the logs are annotated correctly (with pod and container id).

I've been working in that domain for a long time - including being one of the main architects for HTTP/3+QUIC in a public CDN offering. And I'll agree with everyone that this is very niche question, and a great answer seems out of reach for most "senior engineers".

Translating a UDP packet into a HTTP request and back is reasonably easy. Yes, maybe one can do that in a coding interview with some pseudo code. But scaling it and making it reliable is yet another dimension.

Any candidate would need to understand that a single UDP socket itself would probably already a bottleneck for just running this on single machine, and figure out how to increase concurrency there. That's not easy - the amount of engineers knowing how SO_REUSEPORT works and when it doesn't work is low.

After that you start to dig into how you can actually spread load between hosts. Would an answer like I "I hope my cloud provider solves this for me" be good enough? Probably not. If it actually is, do candidates have to both understand the cloud providers native APIs and Terraform (mentioned in the blog post). Seems pretty unnecessary, terraform is just one tool out of the myriad of tools which can be used to configure cloud services. Not everyone will have used it before. Or would it even expect candidates to do a long writeup about the pro's and con's of client-side load balancing?

Are applicants required to talk about upstream connection pooling? Describe and implement a CDN like multi-tier architecture?

Last but far from least is that the requested architecture is very easy to misuse for denial of service and amplification attacks. Just being able to describe how to monitor and mitigate that is already a very very hard task, that very few specialists have worked on so far.

It's very fuzzy what would be good enough if this is a "homework task". At least in a synchronous interview the interviewer could give feedback that they are satisfied. So I think in a synchronous interview the question might be ok - but there will probably just be time to either talk about coding or about system architecture.

Why? If the pod is defined to spawn multiple containers, and each container runs the same application, then this seems true to me? Unless you would add an additional filter on the container name.

yes it can, if you tag your log stream correctly - either by having the stream externally tagged via attributes, or internally by following certain conventions in the log line.

You can also do something like

select client_ip from requests where elapsed_ms > 10000

which is incredibly powerful

It's not easy to cancel any future. It's easy to *pretend* to cancel any future. E.g. if you cancel (drop) anything that uses spawn_blocking, it will just continue to run in the background without you being aware of it. If you cancel any async fs operation that is implemented in terms of a threadpool, it will also continue to run.

This all can lead to very hard to understand bugs - e.g. "why does my service fail because a file is still in use, while I'm sure nothing uses the file anymore"

Among others: having to transmit 1200-1500byte packets individually to the kernel, which it will all route, filter (iptables, nftables, ebpf) individually instead of just acting on much bigger data chunks for TCP. With GSO it gets a bit better, but it’s still far off from what can be done for TCP.

Then there’s the userspace work for assembling and encrypting all these tiny packets individually, and looking up the right datastructures (connections, streams).

And there’s challenges load balancing the load of multiple Quic connections or streams across CPU cores. If only one core dequeues UDP datagrams for all connections on an endpoint then those will be bottlenecked by that core - whereas for TCP the kernel and drivers can already do more work with multiple receive queues and threads. And while one can run multiple sockets and threads with port reuse, it poses other challenges if a packet for a certain connection gets routed to the wrong thread due to connection migration. Theres also solutions for that - eg in the form of sophisticated eBPF programs. But they require a lot of work and are hard to apply for regular users that just want to use QUIC as a library.

Actually the benchmarks just measure the first part (cpu efficiency) since it’s a localhost benchmark. The gap will be most likely due to missing GSO if it’s not implemented. Its such a huge difference, and pretty much the only thing which can prevent QUIC from being totally inefficient.

We need to distinguish between performance (throughput over a congested/lossy connection) and efficiency (cpu and memory usage). Quic can achieve higher performance, but will always be less efficient. The linked benchmark actually just measures efficiency since it’s about sending data over loopback on the same host

NICs assume stuff for TCP (segmentation offload) that they can’t do for UDP, or can only do in a very limited fashion (GSO).

TLS offloads are very niche. There’s barely anyone using them in production, and the benchmarks are very likely without

And Renesas continues to produce the existing chips/architectures (like SuperH), but also adopted ARM due to customer demand.

I don't think you can. The library would block the libill worker thread, which would also block all other libmill coroutines from running. The libmill channels need to yield in a way where they don't block the underlying threads.

libmill (https://github.com/sustrik/libmill) and libdill (https://github.com/sustrik/libdill) should be similar and probably mentioned.

As far as I understand the differences between CspChan and libmill might be that libmill also implements lightweight tasks (coroutines) and everything that goes with it (IO multiplexing, async timers, etc), while CspChan uses OS threads?

HTTP Streams (and Server Sent Events - which is a predefined body format that is understood among web browsers) are supported just fine by most infrastructure - probably even better than websockets.

The advantage of them is that "they are just HTTP requests", so as long as your proxy actually can forward request and response bodies in a streaming fashion it will work. There's no need to understand the contents. It will not work if the proxy is implemented by waiting for the whole HTTP response to complete before forwarding the body. But that wouldn't work very well for a whole lot of other use-cases like file transfers, where buffering the whole body is not practical.

A caveat is that most proxies won't allow for indefinite body body streaming, since they want to avoid all the TCP connections being tied up, so it's likely that the streaming would be interrupted at some time. In that case a reconnect logic would be necessary. But same applies to websockets.

It seems extracted from levels, so it's about whatever people report there.