HN user

cronos

467 karma
Posts1
Comments32
View on HN

On macOS we have 3 ways to run Tailscale: https://tailscale.com/kb/1065/macos-variants Two of them have a GUI component and use the Keychain to store their state.

The third one is just the open-source tailscaled binary that you have to compile yourself, and it doesn't talk to the Keychain. It stores a plaintext file on disk like the Linux variant without state encryption. Unlike the GUI variants, this one is not a Swift program that can easily talk to the Keychain API.

IIUC, it's a bit more nuanced: TPM stores hashes of various things like firmware in PCRs, and when creating keys in the TPM you can optionally bind the key to specific PCR values. But you also don't have to (and Tailscale doesn't), in which case keys survive firmware updates for example.

Not even that. An attacker with local root can just extract the wireguard keys from process memory, or use the TPM to decrypt the state file like Tailscale would.

The only scenario where it helps is a local attacker who can read the state file on disk, but is not full root. Kinda unlikely on Linux, but could happen on Windows.

There are two new-ish features in Tailscale that use TPMs: node state encryption (https://tailscale.com/kb/1596/secure-node-state-storage) and hardware attestation keys.

Hardware key attestation is a yet-unfinished feature that we're building. The idea is to generate a signing key inside of the TPM and use it to send signatures to our control plane and other nodes, proving that it's the same node still. (The difference from node state encryption is that an attacker can still steal the node credentials from memory while they are decrypted at runtime).

We started by always generating hardware attestation keys on first start or loading them from the TPM if they were already generated (which seemed safe enough to do by default). That loading part was causing startup failures in some cases.

To be honest, I didn't get to the bottom of all the reports in that github issue, but this is likely why for some users setting `--encrypt-state=false` didn't help.

Windows uses TPM for Bitlocker. A very common scenario where TPMs get reset is BIOS updates (when a TPM is implemented in firmware). AFAIK, Windows cheats here because it also manages BIOS updates. When an update happens, it takes extra steps to preserve the Bitlocker encryption key in plaintext, and re-seals it to the TPM after the update completes.

Apart from Windows, there are many setups that fail in fun ways: Kubernetes pods that migrate from one VM with a TPM to another one, hypervisors that mount a virtual TPM to VMs, containers or VM images that do Tailscale registration on one machine and then get replicated to others, etc.

Tailscale already did some attempts at cleverness when deciding whether to enable features using a TPM (e.g. probing for TPM health/version on startup, disabling node state encryption on Kubernetes pods), but there was still a long tail of edge cases.

I'm one of the Tailscale engineers who built node state encryption initially (@awly on Github), and who made the call to turn it off by default in 1.92.5.

Another comment in this thread guessed right - this feature is too support intensive. Our original thinking was that a TPM being reset or replaced is always sign of tampering and should result in the client refusing to start or connect. But turns out there are many situations where TPMs are not reliable for non-malicious reasons. Some examples: * https://github.com/tailscale/tailscale/issues/17654 * https://github.com/tailscale/tailscale/issues/18288 * https://github.com/tailscale/tailscale/issues/18302 * plus a number of support tickets

TPMs are a great tool for organizations that have good control of their devices. But the very heterogeneous fleet of devices that Tailscale users have is very difficult to support out of the box. So for now we leave it to security-conscious users and admins to enable, while avoiding unexpected breakage for the broader user base.

We should've provided more of this context in the changelog, apologies!

The tailscale client generates WireGuard key pairs, but only sends public keys to the control plane. The private keys remain on the device only. With only the public keys, tailscale control plane cannot snoop on your traffic.

Want to donate to help Ukraine but there are too many options? A group of volunteers (including myself) put together this aggregator site with structured organization info and some filtering options.

You can skim through ~100 orgs quickly and narrow down your list of options. We also made sure to only put trustworthy orgs there, manually researching each one.

Let's help Ukraine!

PKCS#11 is a C API. It does not describe the wire format for talking to the actual hardware.

To use PKCS#11 for a particular device, you need a module (shared library) to translate between the C API and the actual hardware. This module is usually vendor-specific.

If I develop software with PKCS#11 support, I'm basically asking every user to find a PKCS#11 module from their device vendor and install it in the right place.

With U2F at least the hardware wire format is standardized: https://fidoalliance.org/specs/fido-u2f-v1.2-ps-20170411/fid...

A "ton" of work may have been an exaggeration.

Maintaining a CA (and dealing with cert rotation) is some work.

Other things are indeed just a flag or config option (like jumphosts). But it takes work for a sysadmin/devops to educate all engineers in the company and make sure everyone uses the correct setup and doesn't end up dropping authorized_keys around random servers.

It's not that difficult technically as it is socially.

There are a few differences between an OpenSSH jump host and Teleport: - you have to actively manage authorized_keys for every person using openssh; Teleport manages a PKI and can be backed by your existing SSO - it hard to restrict any given user to a subset of hosts (e.g. only allow select few to access prod database); Teleport has RBAC - hosts with Teleport also get SSH certificates, so you don't need to trust-on-first-use (which everyone has been conditioned to ignore)

Yes, but it's rarely how companies use OpenSSH because it takes a ton of work to set up. Teleport gives you all this functionality by default out of the box. So it's not inventing anything new, just provides better UX.

The code examples are quite non-idiomatic and have lots of issues, to be honest.

- constant strings (defined as vars) as return values instead of the standard error type

- not formatted

- synthetic request's response body not closed - a memory leak

- log.Fatal (which results in os.Exit(1) immediately killing the program) for all the non-critical errors

- timestamp as a string instead of time.Time in the struct

- not re-using http.DefaultClient

- 3rd party package with a custom DSL instead of a simple time.Ticker

Sorry for the pedantry, but if such code examples live on the internet, newcomers to Go will learn from it (if not blindly copy-paste) and it will spread even further indirectly giving bad reputation to the language.

P.S. i do like this idea of continuous integration testing in production, will probably start using it at our company.

Probably no. Author says that he uses IDs as keys. I can assume that they are 32 or 64 bit long. So even if they are 32 bits, you would need a 4GB slice to hold any potential key. Or make a dynamic array (well, slice) that would give even more overhead than a map (and would closely resemble what map does internally better anyways). So only if those keys were really small, that would be a possible alternative.