Making an SSH client the hard way

by darthShadow • 4 years ago

299 102 4 years ago

dekhn9• 4 years ago

Most of the products from tailscale just seem to be "look at the inner platforms we can build that replace the outer platforms".

Having an SSH client in your browser join your VPN violates all the principles of modern computing.

bradfitz3• 4 years ago

Having an SSH client in your browser join your VPN violates all the principles of modern computing.

I think that's a compliment? You're welcome? :)

convolvatron• 4 years ago

absolutely. the lines weren't bad, but they were off in places and now they are set in stone for no good reason.

edit: _especially_ when it comes to security

dekhn9• 4 years ago

It's not a compliment. Or rather, within my understanding of how things should be architected, it's not. I certainly wouldn't claim that my own beliefs about network architecture should trump others, and I work in a different domain from most of the people with your use case. Whether the disruptive work you're doing is good for the world in the long run is still a very open question in my mind.

I used to think that everything in the world should move to the browser (in my case, that would be high performance molecular graphics and microscope control) and ChromeOS was the logical extension. Web Assembly to handle the existing C++ codebases, a collection of standard web tech to handle the user interface. Big fan of SSH extension because it meant that machines that only ran a browser could be useful command line programming terminals. But didn't like that SSH extension was written in a dead-end container technology (NaCl) and the CHrome folks sort of messed up multiple ways for the extension to integrate better.

But after working with web tech for enough time I came to conclude that putting things in the browser like this is an antipattern, in particular increasing the surface complexity of the software space while not actually replacing existing systems (openssh continues to exist, OS-level VPNs continue to exist, even after you port the VPN and client to web assembly and run it in a browser).

In my mental model, it makes more sense to put the browser in a VM and then wire the VM's networking to a VPN handled by the OS, rather than putting what is more or less a significant fraction of a virtual machine manager's capabilities into the browser. If you're going to do that, why not go whole-hog and add a VM to chrome so that it can run linux with a full networking stack and then host an SSH client inside that? So you can run linux in your chrome in your linux.

My compliment is: I am impressed at how well you parlayed several technical projects into a thought leadership position, but we have fundamentally different architectural principles and work in different domains. Your work disrupts mine, but mine doesn't disrupt yours. My enterprise actually disallows me from visiting your company's website on my work computer because users installing their own VPNs is considered a security risk (fwiw, I bought into BeyondCorp, which eschews VPNs, a long time ago, and would prefer my enterprise eliminate VPNs, as they don't really protect our users).

xaduha• 4 years ago

From the article

Web-based SSH clients aren’t new. Nearly every VPS and cloud provider already lets you connect to your VMs from the web — so how is this different?

This is clearly isn't for everyone, but if you need or already use something like that, then I think this has a chance to be more secure than some other options.

dekhn9• 4 years ago

I really liked Chrome SSH extension but now I've returned to using the ssh command line client on all three platforms.

The issue is wiring up the browser-hosted application with a custom network inside the browser. It's a truly interesting but highly disruptive concept and I'm curious how it will play out. Perhaps in the future every program will statically compile its own TCP stack and talk over RAW sockets, but... that's sort of throwing away everything BSD and Linux and Windows achieved over the last few decades in terms of OS abstractions.

vngzs2• 4 years ago

This is really cool and fun, but is this a safe way to run SSH clients?

If, say, the adblock Chrome extension you're using gets bought by a malware operator and backdoored[0], now it also has SSH and VPN access.

[0] https://www.wired.co.uk/article/fake-chrome-extensions-malwa...

dgentry2• 4 years ago

The Tailscale VPN client, the same one which runs on other devices, is compiled to WASM. It handles all of the key exchanges to connect to the tailnet. The SSH session is running as a WASM Tailscale client.

The browser, opening connections from within the browser engine, doesn't have the keys for SSH or VPN access.

ikiris• 4 years ago

citation needed

shp0ngle• 4 years ago

It doesn’t have the keys, but it can inject any javascript and do whatever the user can do.

vngzs2• 4 years ago

Sounds like about "as good as this gets" if you happen to want to do this in browsers. Good job.

Asmod4n• 4 years ago

And then the addon intercepts the loading of the Wasm code, injects it’s own payload into it and has access to the keys.

fulafel• 4 years ago

The wasm code runs in the browser, hence the keys live in browser memory, so the browser has access to the keys.

Scaevolus• 4 years ago

If you have an actively malwared extension, there's probably an easier way to exploit any given target-- the simplest being recording passwords.

There's not really a "safe" website when the browser is malicious.

gunapologist994• 4 years ago

This is turning a remote server into a previously "safe website" and expanding the threat model to include remote web browser attacks.

vbezhenar• 4 years ago

I believe that replacing vscode.exe that you just downloaded is entirely possible with extension. So if you’ve got compromised browser, it’s gotta be hard to stay safe elsewhere. Lots of attack vectors. Better don’t get infected.

amluto5• 4 years ago

I can't shake the feeling that Tailscale's SSH authentication mechanism is at the wrong layer of the stack. It appears to work by looking at the (source, dest) IP address pair and mapping that to a Tailscale identity. But this may mean that any user or anyone who can initiate TCP connections from an authenticated user's IP can authenticate to the destination over Tailscale SSH.

If Tailscale's client was a userspace construct bound to a specific user SSH program, maybe fine. But Tailscale's client is a regular VPN client. What happens if you connect to the Tailscale VPN, open a malicious but sandboxed app of some sort, and that app connects to the target on TCP port 22.

For all that it's a seriously unfinished product, Cloudflare's SSH offering seems better thought out. Perhaps Tailscale should find a way to issue a short-lived certificate and use that in addition?

(It looks like regular sshd could almost be convinced to handle this. If the SSH_CONNECTION environment variable were passed to the AuthorizedPrincipalsCommand helper or if the source and destination were available as '%' tokens, then AuthorizedPrincipalsCommand could do the Tailscale tuple lookup and use it as a second factor in addition to a short-lived certificate (or regular SSH key or whatever). I bet openssh would accept a patch for this.)

mihaip2• 4 years ago

Tailscale SSH's check mode (https://tailscale.com/kb/1193/tailscale-ssh/#configure-tails...) is meant to address the issue of "rogue process starts an SSH connection". For truly sensitive applications, you can set the check period to be "1s" to always require it.

amluto5• 4 years ago

Hmm. If the problem is that Tailscale SSH doesn’t strongly associate the person authenticating with the connection being authenticated, asking the person to reauthenticate seems like a pretty weak solution.

fisian• 4 years ago

I agree.

I feel like there should be a lightweight way to use SSH certificates, so you could use it independently or on top of Tailscale. Like a server (CA), client and daemon on your machines that should be reachable via SSH that handles short lived certs and authentication of clients. But I'm not aware of anything like that.

I have used Teleport before but it seemed not that great for machines not publicly reachable, because then all the traffic goes through a proxy.

amluto5• 4 years ago

Smallstep and cloudflared do this. Sadly, both of them seem to use essentially identical client-side hacks. Smallstep is a small company that I wouldn’t trust with the keys to the kingdom, and Cloudflare seems to treat their SSH product as something thrown over the fence with nothing resembling support.

Gravitational’s Teleport seems pretty good, but it’s heavyweight and doesn’t have any pricing appropriate for small businesses.

pquerna• 4 years ago

This is how Okta's Advanced Server Access works: https://www.okta.com/products/advanced-server-access/

matthewaveryusa2• 4 years ago

A step-up prompt or notification wouldn't be a bad idea to approve ssh connections before they are established -- This is the same issue with ssh agents. I think there's an ACL setting in tailscale where you check ssh connections with a re-auth. it's time-based with a minimum of 1 minute though.

easton• 4 years ago

Could the Tailscale client be packaged as an extension so I can visit sites on my Tailnet without having to install a client? Sometimes I want to visit a "internal" site without having to install the client, if I'm using a temporary box for something. I'm not sure how much more work would have to be done, might have to dig into the open source pieces of this.

capdeck• 4 years ago

Putting my vote in for this feature as well... Also, why not compile VNC into WASM and get full remote desktop experience for graphical apps. It seems that hard work has already been done!

bradfitz3• 4 years ago

Also, why not compile VNC into WASM and get full remote desktop experience for graphical apps. It seems that hard work has already been done!

Yup. :)

In fact, that's mentioned in the original public bug: https://github.com/tailscale/tailscale/issues/3157

heliophobicdude• 4 years ago

Hi Mihai! Great work! I would love to see where this goes!

Forgive my ignorance but is there any sort of native client besides the browser running in the background to help with websocket to tcp? Or a tunnel to a cloud service to help there?

mihaip2• 4 years ago

No native client is running. The browser makes a WebSocket connection to our relay server, and we run the WireGuard tunnel over that.

ignoramous3• 4 years ago

You mean WireGuard (in userspace w/ netstack) tunneled in WebSockets to DERPServers, correct? What sorcery is this. (:

Scarbutt• 4 years ago

But can it run emacs?

xena• 4 years ago

Yes

skybrian• 4 years ago

In the old days, people said you shouldn't write crypto in JavaScript because it was somehow insecure. Have those concerns gone away with WebAssembly and https everywhere?

rany_• 4 years ago

I don't think the argument was ever that "JavaScript was insecure." It was that the websites hosting it may be compromised or may change the script at any time without any indication (or the FBI forcing a site to backdoor the JS for some investigation)

ignoramous3• 4 years ago

Yeah, but see also Code Verify (and sub-resource integrity): https://blog.cloudflare.com/cloudflare-verifies-code-whatsap...

kevin_thibedeau• 4 years ago

It will never go away because you can't guarantee constant time algorithms will be implemented as such when transformed by a JIT.

lxgr7• 4 years ago

If machine code can issue the necessary hints to the hardware to skip all time-variant optimizations, why couldn‘t the same work for a WASM runtime?

e12e• 4 years ago

I think the consensus is still that you can't write side-channel/timing proof crypto in (most) Javascript (runtimes) - but that with webcrypto(?) most runtimes will provide the secure crypto primitives you need in order to do (secure) crypto with Javascript?

Anunayj• 4 years ago

To add, certain concerns still stand since the Web crypto API is extremely limited in capabilities (Example it doesn't support secp256k1).

However on the bright side performance.now() has a much worse resolution on most browsers, making timing attacks harder.

easrng• 4 years ago

Now that HTTPS is everywhere and crypto.getRandomValues provides a secure RNG I think most of the concerns are mitigated.

lxgr7• 4 years ago

Maybe for Javascript, the language.

Definitely not for browsers as the execution environment, at least browsers running extensions.

jillesvangurp2• 4 years ago

Both javascript and browsers have been hardened a lot since then. Back in the day people were using javascript on plain http websites, without https. Cross site scripting prevention in browsers simply did not exist.

Nowadays, it's a lot harder to get past browser security and people run all sorts of applications in browsers such as banking, business critical SAAS, email, etc. So, perfectly fine to include some crypto in that and probably not optional to do so for a lot of applications.

What's still true is that you should not be rolling your own crypto libraries and instead use libraries from reputable sources that have been scrutinized by people that know what they are doing. That's true whether you use javascript, C or whatever. And of course with web assembly you can just compile those libraries and use them in a browser sandbox.

Spivak3• 4 years ago

To make this possible, we ported the following to WebAssembly: the Tailscale client, WireGuard®, a complete userspace network stack (from gVisor), and an SSH client.

I love that they were clearly inspired by fly.io. Warms my heart that a random blog post with a good idea can spread like this.

bradfitz3• 4 years ago

Which blog post are you referring to?

But yes, we love Fly and use them (and they use us) and we share a slack channel between our two companies for casual banter.

Spivak3• 4 years ago

This one!

https://fly.io/blog/ssh-and-user-mode-ip-wireguard/

chatmasta• 4 years ago

Cool feature! I was just looking at boringtun last night and wondering if it could compile to WASM, to get a virtualized network interface in the browser.

Did you experiment with the new WebTransport API [0] at all? It's only supported in Chromium browsers, but seems promising for this kind of use case.

[0] https://chromestatus.com/feature/4854144902889472

PaulWaldman• 4 years ago

To make this possible, we ported the following to WebAssembly: the Tailscale client, WireGuard®, a complete userspace network stack (from gVisor), and an SSH client.

Would it be possible to bundle the same into a portable application allowing you to use Tailscale without installing it? My understanding is that currently if you can't install Tailscale on a client you need to use Subnet Router. https://tailscale.com/kb/1109/devices-without-tailscale/

lxgr7• 4 years ago

You‘d have to redirect all network usage (i.e. the sockets API or your platform‘s equivalent) through the custom stack, which is possible if you can rebuild the source or by using something like LD_PRELOAD for binaries, but can get very tricky in the general case.

There‘s an utility called SSHuttle that does something similar for SSH instead of Tailscale/Wireguard: It redirects all sockets usage to go through an SSH connection, to allow usage of SSH port forwarding without explicit SOCKS support on the app‘s side.

Serow225• 4 years ago

SSHuttle was created by the CEO/co-founder of Tailscale :)

ithkuil3• 4 years ago

. You can use tools like proxychains to leverage LD_PRELOAD to redirect all network traffic via a socks5 proxy.

Tailscaled comes with a socks5 proxy

gunapologist994• 4 years ago

This significantly increases the threat model for your remote servers to include all sorts of remote attacks through the web, including:

  * garden-variety web attacks (i.e., XSS, CRSF, etc)

  * attacks that might become viable against the browser (for example, Mobile Safari has a history of vulnerabilities)

  * various attacks against the backend web server (API attacks)

  * attacks against the WASM layer

  * CDN injections

  * Tailscale's backend (various types of injections, timing attacks, or deeper attacks on Tailscale's infrastructure like the nightmares of HeartBleed, Shellshock, Meltdown, etc)

That's probably a very incomplete list.

Realistically, this essentially (actually, literally) opens a remote root shell into your entire infrastructure through a web page, with apparently nothing more than matching an IP address pair (https://news.ycombinator.com/item?id=33361837) to authenticate.

What could go wrong?

This design with its loose coupling between authenticated user and IP addresses for high-value targets makes me view Tailscale's security model in a whole new light.

sedatk3• 4 years ago

That's unnecessary sensationalism. Most of those vectors are behind an SSO login and are not exposed to Internet at all (from the article: "your browser becomes a Tailscale client, and joins your tailnet in the same way as any other device that you run Tailscale on").

Or, did you mean attacks on SSO? If that's the case, then SSH web wouldn't make any difference. Someone authenticating themselves could use regular SSH or whatever.

Similarly, Tailscale backend is already subject to the vectors you mentioned (API, side-channel attacks). This feature doesn't add any new attack vectors.

Again, attacks on browser means end of game already. Someone can use that vector to access to your local network in other ways. They don't need Tailscale's SSH web client for that.

gunapologist994• 4 years ago

Again, attacks on browser means end of game already.

A bad Chrome extension does not allow the bad guys to open a terminal on my machine, load my ssh keys, launch an authenticated SSH connection, and launch an authenticated SSH connection into an enumerated list of remote servers.

ignoramous3• 4 years ago

Well, this exists: https://engineering.fb.com/2022/03/10/security/code-verify/

dgentry2• 4 years ago

(Tailscale employee)

It is not a web page with a shell open.

It is the Tailscale client, compiled to WASM, maintaining the keys for connections to nodes on the tailnet. Connections opened from the browser engine don't get the ability to reach the tailnet.

otterley• 4 years ago

Given the apparent (yet possibly unfounded) concern, it might be useful to discuss this issue in the blog post. Would you consider updating it?

_hyn3• 4 years ago

This seems to really expand the scope of how keys are handled and authorized. It at least appears to be optional, however.

Or just use userify's public key distribution model (then you can stick with a simple and proven design that just automates the standard SSH design that's been around for decades)

Or just do it the old way, by hand: drop your teams' public keys on groups of remote servers, setup their sudo access, and you're done.

Or just use Ansible or Chef or Terraform, or any of those other server orchestration tools, too (just without the permission layer/bulk remote session kills/user deletes of Userify).

of course, if you really need an ssh terminal in a browser, then you have to go with one of these sorts of crazy things, and at least it's not using passwords!

aliqot• 4 years ago

I wouldn't be the target market for this, however more power to them. I understand that to an extent we should never roll our own, so to speak, however, I think that we should not put all of our eggs in one basket. In that regard, I think that once the beginning kinks are ironed out it will be a better thing and we should ultimately embrace these types of endeavors.

Let me say again, though I admire it, I'd never use this. I like to sleep soundly, as irrational as that may be.

ice3• 4 years ago

Interesting, I could see this as a nice replacement for Gravitational Teleport.

One of the things that Teleport lacks (IMO) is Wireguard

kybernetyk• 4 years ago

To make this possible, we ported the following to WebAssembly: the Tailscale client, WireGuard®, a complete userspace network stack (from gVisor), and an SSH client.

Wow, and they're proud of it.

zeroxfe• 4 years ago

It's a pretty good approach and pretty hard to pull off. I'd be proud of it too.

cynix• 4 years ago

Will Tailscale SSH support FreeBSD any time soon?

dekhn9• 4 years ago

I just learned there's a POSIX kernel in Go that can actually run apps. https://www.usenix.org/conference/osdi18/presentation/cutler

I think this should be compiled with WASM and deployed via web page. Then we can explore the idea of browser-hosted POSIX kernels.

Making an SSH client the hard way

Related Stories