HN user

vijayp

585 karma

EIR @ Benchmark. Formerly: Director of Engineering & NYC Site lead for Twitter Engineering; @MitroCo; @Google. I also angel invest in startups

my public key: https://keybase.io/vijayp; my proof: https://keybase.io/vijayp/sigs/LySS57Jo7L6nq51XlNRqtA_vSjisVMjSxx8dWitzCuw

www.vijayp.ca Twitter: @vijayp

Posts6
Comments47
View on HN

Congrats to the team for tracking this down, it was a great write-up!

Twitter has some great networking/kernel engineers. When I was working at Twitter a few years back we isolated and fixed another insidious kernel bug; a large group was critical to making it happen (including Cong, who worked on this bug): https://tech.vijayp.ca/linux-kernel-bug-delivers-corrupt-tcp...

I'm always shocked at how the kernel seems to mostly work, with such meagre test coverage. I guess testing in production does kind of work at scale?

The design is terrible. I never use the touchpad for anything regular keys wouldn't do better. The keyboard on my 2017 MBP13 failed three times in the year I've owned it. The first time, the Apple Store person "cleaned" the keyboard, fixed things for a while. A couple of weeks later, the problem came back. I insisted on a better solution, and they replaced the keyboard.

Five months later, a different key failed, and the Apple Store people sent it to Memphis. This time, it came back with one of the "new and improved" keyboards. They also replaced the screen because of 'delamination' which I hadn't noticed. The total amount they spent to repair this computer is approaching the cost of the machine at this point! So far, the new keyboard hasn't failed yet but I'm not holding my breath…

If he actually flew out of YVR, he was probably in a us preclearance area -- us customs are located in some Canadian airports so flights can go directly into us domestic terminals.

Since those screenings are on Canadian territory, us agents do not have police powers or the right to detain people. They can deny entry but people can leave at any time unless they have violated Canadian law. (https://en.m.wikipedia.org/wiki/United_States_border_preclea...)

If this was the case, he could have simply left the airport at any time.

That's a good point. Though I think it would be challenging to have a phone number that no one knows which you also carry around with you.

It's possible if you use something like Google Voice for most of your regular calls, but you still need to make sure that the telco can't tie your name to your number…

Yeah, I'm also scared of LastPass being a SPOF. While LastPass does do a good job, no one is perfect, and the cost of having my account compromised is really high

I'm now leaning towards encrypting backup codes with a passphrase and putting the encrypted blobs in LastPass. I haven't actually done this but as long as I don't forget the second passphrase, that might work…

Oops I probably should have been clearer.

Hardware verification IS performed. For various reasons, the nic never itself drips packets that are corrupt, packets are instead marked by HW as either verified or unverified. When a packet is marked as unverified, the kernel should verify and potentially reject the packet before delivery to the application. The bug in the veth driver causes the kernel to treat packets marked unverified as "verified"

yes, the code is as follows in the broken veth:

if (skb->ip_summed == CHECKSUM_NONE && rcv->features & NETIF_F_RXCSUM)

checksum offloading is encapsulated in the rcv-features bitmap, so disabling it will hide this bug.

You can do something like this within your container to disable it (from memory, might be slightly off): $ ethtool --offload VETH_DEVICE_NAME rx off tx off $ ethtool -K VETH_DEVICE_NAME gso off

A password manager really needs to be a high-availability service -- it should work even (especially) when AWS is down. Since our service (intentionally) does not cache secret data on the client, running a proxy is not substantially easier than running our service. Plus we'd have to write this proxy :)

Yeah, our largest costs are: - a primary server running on AWS - a read-only replica running on google compute engine. Other smaller costs include networking, DNS, and various tax/administrative/regulatory fees