A new generation of Tailscale access controls

by ingve • 1 year ago

226 66 1 year ago

When do I have to update all those policies to use the new syntax? ... You never have to update your existing rules to use the new grants syntax. We will support our original ACL syntax forever. ... When we say that Tailscale is a stable platform, we aren’t kidding around.

Nice business decision. Just like their reaction to Headscale, I'm reminded Tailscale 'gets it' when interacting with the dev community and why their popularity will continue to grow.

gessha• 1 year ago

How did they react to Headscale? I must’ve missed the announcement.

aspenmayer• 1 year ago

I'm not sure what they meant, but Tailscale links to Headscale and says this:

Headscale is an open source alternative to the Tailscale coordination server and can be self-hosted for a single tailnet. Headscale is a re-implemented version of the Tailscale coordination server, developed independently and completely separate from Tailscale. Headscale is a project that complements Tailscale — with its own independent community of users and developers. Tailscale does not set Headscale’s product direction or manage the community, and neither prohibits nor requires employees from contributing to Headscale.
Our opinion is that Headscale provides a valuable complement to Tailscale: It helps personal users better understand both how Tailscale works and how to run a coordination server at home. As such, Tailscale works with Headscale maintainers when making changes to Tailscale clients that might affect how the Headscale coordination server works, to ensure ongoing compatibility.

https://tailscale.com/opensource#encouraging-headscale

xeonmc• 1 year ago

See https://tailscale.dev/blog/headscale-funnel

matthewaveryusa• 1 year ago

They let it be. The decision is many-fold correct:

1) hobbiests that are obsessed with self-hosting can use it -- tailscale will never see a penny from them anyways

2) some businesses will consider headscale as an escape hatch de-risking if tailscale were to be purchased out by an extortionist (like broadcom's acquisition of VMware)

3) It signals they are doing business in a position of strength: businesses don't buy software, they buy solutions to problems

Lammy3• 1 year ago

How did they react to Headscale? I must’ve missed the announcement.

They hired the dev: https://tailscale.com/blog/opensource#the-open-source-coordi...

Fun fact: Tailscale have an almost-assuredly-accurate count of Headscale users due to how few people disable the Tailscale client's telemetry which by default sends them real-time events about everything you do on a Headscale network. See KB1011: https://tailscale.com/kb/1011/log-mesh-traffic

“Each Tailscale agent in your distributed network streams its logs to a central log server (at log.tailscale.io). This includes real-time events for open and close events for every inter-machine connection (TCP or UDP) on your network.”

It's possible to opt out of this spying on Unix/Windows/Mac by starting Tailscale with `--no-logs-no-support` or `TS_NO_LOGS_NO_SUPPORT=true` environment variable (see https://tailscale.com/kb/1011/log-mesh-traffic#opting-out-of...), but it is not currently possible to opt out in the Android/iOS clients: https://github.com/tailscale/tailscale/issues/13174

This is why they had the data telling them they should hire the Headscale dev to nip that competition in the bud before it had any chance to usurp their business by overtaking them in mindshare using their own client. Definitely a sound business decision, but it really really irked me when I realized I was feeding Tailscale detailed usage data for everything I did on my supposedly “““private””” Headscale network despite having no business relationship with them.

For an example of how invasive this is for the average user, this person discovered Tailscale trying to collect ~18000 data points per week about their network usage based on the number of blocked DNS requests for `log.tailscale.com`: https://github.com/tailscale/tailscale/issues/15326

The privacy-adversarial behavior made me switch from Headscale to NetBird for my personal network now that it's available in FreeBSD Ports: https://www.freshports.org/security/netbird/

neodymiumphish2• 1 year ago

I believe they also hired the developer of headscale!

csomar2• 1 year ago

My guess is they probably don't care because their pricing is really low. If you want to self-host, for any serious company, you need a very resilient setup (multiple servers) because you can't afford to have your "authz" down. If your headscale is down, so is everything in your organization. $6/month/user is then relatively cheap. The savings on self-hosting will be minimal.

OJFord3• 1 year ago

I reckon this (and that it can be used in conjunction with grants) implies they're translating the old config to the new format if it exists on the fly. Otherwise, if it involved running two things one in maintenance mode (and somehow ensuring they both take effect and don't conflict) it'd be ballsy to promise 'forever'.

Although not sure why that's preferable to converting it once-off, so it's still not something a user has to worry about, but then the old way is done with.

yurishimo• 1 year ago

Stripe has been doing it for god knows how long now and it seems to be working for them. With a robust test suite, I can’t think of too much that can/would go wrong. Honestly, I wish more software would approach things this way instead of introducing a major api version every 5 years. Just continuously ship the latest thing and translate everything that came before. Hopefully if people are also using “official” client libraries, they will eventually upgrade their dependencies and their API version will also get bumped. If there is a particularly heavy usage client jumping forward 100 versions and using too many of your resources, hopefully your org can dedicate some resources to help them upgrade which theoretically benefits both of you.

Now that I think of it Adyen did this to us a few years back. We’ll probably stay in the v5 branch until they politely poke us to upgrade again in another 5 years…

tczMUFlmoNk• 1 year ago

Although not sure why that's preferable to converting it once-off, so it's still not something a user has to worry about, but then the old way is done with.

This way also supports users who have scripts or workflows that produce the old form of ACLs. If you had a single cut-off date, those scripts would presumably need to be updated.

csomar2• 1 year ago

They probably have a parser that parses to some other format. They only need to maintain the parser and they probably won't support new features on it.

samlinville-ts3• 1 year ago

Post author here!

We are indeed converting the old syntax to grants behind the scenes. In fact, we have been quietly doing this for months :)

As tczMUFlmoNk mentioned further down, we wanted to allow both syntaxes in perpetuity to account for scripts, Terraform configs, version-controlled policy files, and other automation that may rely on the original syntax.

We will provide a button on the JSON editor in our admin console to do a one-click conversion for anyone who would like to make a hard switch.

aatd86• 1 year ago

There seems to be a missing comma in the first grant example. If I am not beffudled. At the end of the dst line.

In fact, comma usage seems inconsistent for the last element of some of the nested objects.

To be fair, I don't remember what's the json rule either and would need a quick lookup.

stavros• 1 year ago

Yeah, the example that's missing the commas is wrong, if I'm not mistaken about their json parser. The optional comma at the end is fine with JSON5, which I think they're using, as that allows optional commas, comments, things like that.

jrockway• 1 year ago

They use hujson: https://github.com/tailscale/hujson

But it doesn't accept things like the example that omits an intermediate comma:

  {
      "foo": 42
      "bar": 123
  }

  >  hujson: line 3, column 5: invalid character '"' after object value (expecting ',' or '}')

orthoxerox• 1 year ago

They should've gone with Relaxed JSON, which is more relaxed than JSON5, which is more relaxed than HuJSON/JWCC.

a_t483• 1 year ago

Vaguely related - there’s still no easy way to get at the email/user name of a user using tailscale ssh right? This is one of the things I really liked about teleport, you could use it to properly attribute git commits on shared machines, without any special setup on the user side.

samlinville-ts3• 1 year ago

Post author here!

Are you saying you need an easy way to match up a Tailscale user’s email to a user on the host? (Ie, sam@tailscale.com matches to the username sam)

If that’s the case, take a look at the local-part section here: https://tailscale.com/kb/1193/tailscale-ssh#users

If that’s not what you meant, could you clarify a bit further? I’d love to understand how to close this gap for you.

a_t483• 1 year ago

No, for disambiguating between multiple users logging into one shared account. Basically, I'd need for tailscale to inject an env variable into the session. Teleport does this with something like SSH_TELEPORT_USER (I don't have access to a teleport instance right now to check the exact name, sorry). It's not really for security purposes (anyone can spoof any env variable they like, anyhow), just for convenience for helper scripts. In robotics it's super common to have just one or two users per machine, and give shared logins to those machines.

While I have you here, it would also be neat if there were a way to supplement regular SSH's auth with tailscale SSH's keys. Specifically for situations where the target machine has lost internet connectivity but you can still get onto the same network as it. More robotics problems. :) Can understand if this would never "work" for tailscale's auth model.

SOLAR_FIELDS4• 1 year ago

When would I want this over a more generic solution like linking to an idp using something like keycloak? The upside is that its network level blocking, but it still requires modifying the application itself same as connecting an idp, correct? The downside of this approach is that it only works for internal access where your end users are on the tailnet

mousetree2• 1 year ago

It’s at the network level. It will block clients from connecting to the destinations on the network. You don’t need to modify any applications at the destination.

SOLAR_FIELDS4• 1 year ago

Am I misinterpreting like, the entire body of the article?

For private services inside your tailnet, like internal tools, implementing auth on every app is overkill. That’s why we developed tsnet, a Go library that embeds Tailscale directly in your applications. This lets you see and react to the identity of every user who makes a request to your app. So, authentication is handled…what about authorization?

The article then proceeds to show how to modify an application using tsnet to embed in the application. Which sounds a whole lot like modifying the application. The point might be that it’s more straightforward to do this then gate behind something like keycloak, but fundamentally it still requires modifying the application

miki123211• 1 year ago

If you write internal apps, this is a lot easier to support.

You don't have to think about redirects, sessions, cookies, syncing your users and their roles in TS and Keycloak, maintaining two separate policy files and all that.

SOLAR_FIELDS4• 1 year ago

The tradeoff you get is lock in to tailscale and no portability. Though I assume that is true about some of the networking features tailscale offers as well.

ElectricalUnion3• 1 year ago

When would I want this over a more generic solution like linking to an idp using something like keycloak

I assume everyone using Tailscale are opting for "simple code and adminstration" instead of "let me pierce NAT and handle Wireguard tunelling myself". On that point of view, "Identity management" is part of "stuff" that Tailscale helps you in the first place. So why not use it as the "Identity management" solution as well?

When you use `tailscale serve`, you get the headers Tailscale-User-Login, Tailscale-User-Name and Tailscale-User-Profile-Pic that identifies who connected thru the tailscale endpoint. https://tailscale.com/kb/1312/serve#identity-headers

On a separete side note, if you had an exposed OpenID Connect service in the first place, you can use it as the Tailscale tailnet auth SSO: https://tailscale.com/kb/1240/sso-custom-oidc

red0point• 1 year ago

Doesn‘t tailscale force you to bring your own SSO / IdP already?

So it‘s not part of what Tailscale brings, it just adds another layer of indirection between the SSO / IdP you have already and the app, plus requires some custom library integration work, further enhancing lock-in.

codethief• 1 year ago

Speaking of ACLs and grants, I wish Tailnet Lock[0] extended to those as well, i.e. allowed me to sign my tailnet policy file, so that if the coordination server and one of my nodes ever get compromised, the attacker can't just modify the policy file to access arbitrary services in my tailnet from the compromised node, including – in the worst case – SSH[1].

[0] https://tailscale.com/kb/1226/tailnet-lock

[1] https://tailscale.com/kb/1193/tailscale-ssh

ttul• 1 year ago

We were reviewing our SaaS spending recently and I noticed that Tailscale is now our biggest spend. Their approach is apparently working.

bb882• 1 year ago

Tailscale at least on Linux is a mess of surprise configuration and netfilter rules.

It's not for everyone, but if you're into control of your netfilter tables, it's certainly not for you.

haiku20772• 1 year ago

It's a lot more sensible if you use vanilla iptables. And it's far less of a mess than Docker, at least.

bb882• 1 year ago

Right, but then go combine the two and see what happens.

globular-toast• 1 year ago

I recently discovered this is how Calico works and it can indeed be really surprising when it goes wrong. At least in a k8s context you have the option of just burning the node and starting again rather than trying to debug it, though.

dataangel• 1 year ago

Okay but when can we be connected to more than one tailnet at once and have tail scale auto reconnect directly after resume from sleep? That's the quality of life stuff users really care about

ghthor• 1 year ago

You talking about macOS I presume, as I’ve noticed this as well. After every sleep I have to manually toggle tailscale, yet it happily reports being online before I do this…

tpetry• 1 year ago

I would love for Tailscale to work on a "paranoid mode". At the moment, you can be sure that Tailscale can't add machines to your network with Tailnet Lock where you have to authorize a new machine from a trusted node and not from the UI (which Tailscale could also do without me clicking it). That's great feature compared to their competitors which basically have a backdoor into your network.

But changing ACLs and many other changes can be done from the WebUI without needed authorization from a host. So I have to authorize the change in the UI - but Tailscale could really also do it without my consent. Would be great to have a mode that _ANY_ change to the Tailscale network must be authorized from a trusted node with a detailled changeset. So I could be sure that Tailscale can't tamper with my network.

Sure, I could run Headscale and completely work without the Tailscale servers. But I prefer to pay them the small monthly fee to not have to manage this central service of my fleet.

nikisweeting• 1 year ago

Will there eventually be a UI to manage grants so we don't have to write them by hand?

bradfitz• 1 year ago

Yes.

A new generation of Tailscale access controls

Related Stories