Sshmuxd: SSH proxy to replace jump hosts

by Spiritus • 11 years ago

108 62 11 years ago

sshmux, and by extension, sshmuxd, can only forward normal sessions (ssh'ing directly to sshmuxd without a ProxyCommand) if agent forwarding is enabled.

This is very dangerous.

With the average setup, anyone with root access on the middle box can borrow the ssh key of any user connecting through it.

See http://unixwiz.net/techtips/ssh-agent-forwarding.html#sec for more info.

hlieberman• 11 years ago

Considering this seems intended for use in somewhat corporate situations, it may or may not be that serious of a problem.

This is yet another situation where Kerberos got it right years and years ago, but is almost impossible to actually /use/ even now after n years.

beagle33• 11 years ago

Could you give a quick summary (or a link) about how Kerberos works in this respect ?

joushou25• 11 years ago

Exactly. And yes, Kerberos got it somewhat right, but as someone who have used configured and used it, it is quite the exercise to set up. It took ages to get working, and even then, it was slightly cumbersome to use.

asdfaoeu• 11 years ago

Kerberos is just as open as ssh-agents. The middle server is still able to impersonate you.

joushou25• 11 years ago

Agent forwarding can be dangerous, yes, in the sense that you are giving the remote host you are connecting to (or jumphost in this case) permission to sign things with your private key.

In the normal case, this is done through a socket, which means that root on the machine can access it, and sign arbitrary things. A compromised machine is therefore dangerous. This is no different than getting root access on the local machine, as you can access your local ssh agent there as well.

However, with sshmux, the agent channel is not forwarded as a socket, but instead internally handled as the ssh channel it really is. This means that compromising the agent at sshmux requires arbitrary memory access, which is a considerably more difficult feat, and means that the machine is 100% compromised regardless.

noja• 11 years ago

Wait your last sentence lost me: why did you write "and means that the machine is 100% compromised regardless."

I am root, I have arbitrary memory access on the jumphost. I could write a tool, like the heartbleed poc, to find the data I want.

lunixbochs• 11 years ago

I wouldn't attack this with memory. I'd probably just replace the sshmux binary.

keeperofdakeys3• 11 years ago

Personally I have a confirmation dialogue for any agent forwarding request. When I require it, this seems like the best compromise. (I still have my key decrypted in memory, I just get a "Use key?" window that I need to hit enter for).

joushou25• 11 years ago

+1, although a bit more information would be nice ("Use key for session XYZ").

I'm curious, how did you get that set up?

zobzu2• 11 years ago

Its nice.. That said, one if the real issue is getting people to use -W ;)

dijit• 11 years ago

I'm looking at the man page, I had no idea about `-W`. However, what does this protect against that the standard:

    ProxyCommand ssh -q <jumpHost> nc %h 22

doesn't protect?

Edit: just tried `-W` as an addition to the line but I get

   littlefish :: ~ » ssh pgsql3
   Bad stdio forwarding specification '<jumpHost>'
   ssh_exchange_identification: Connection closed by remote host

joushou25• 11 years ago

sshmux gives fine-grained user controls. In the normal jumphost example, you can use ssh -W to connect to arbitrary ports and hosts on the network, poking around where you weren't intended (ssh -W is just netcat where the ssh server initiates the connection for you). With sshmux, ssh -W will only work to hosts permitted for that user, and any other request will just fail.

keeperofdakeys3• 11 years ago

Try formatting it like this:

    ProxyCommand ssh jumpHost -W %h:%p

-W is essentially the same as using nc, however it doesn't require executing a command on the jumphost. This project uses that fact to make the jumphost more secure (you can't execute arbitrary commands), and do destination white-listing for each user.

zobzu2• 11 years ago

-W is just a nicer wait of doing your nc command

joushou25• 11 years ago

That's true. Some people complained that this isn't documented enough, so there is a wiki page for sshmuxd explaining what it does.

Agent forwarding with sshmux should be safer than the general case, though, as the agent is handled in memory, rather than as a socket. The agent is also not forwarded to the destination.

ryan-c• 11 years ago

https://github.com/ryancdotorg/ssh-chain should just work with this.

Galanwe• 11 years ago

I don't really understand what this is supposed to be useful for. Relying on a random software to secure your ssh entry point, instead of a proper linux configuration, that seems like a risky tradeoff.

joushou25• 11 years ago

I'm the author of the project.

The project is meant to ensure that you can allow multiple users access through a jump-host style mechanism, while not permitting any other "abuse" of the jump host. You can lock down SSH a lot, but not as much as sshmux does. Security wise, the code is very, very simple and easy to follow, and even if it went rogue, that's no different than the trust you put in your usual SSH server. If this is a concern, do not use the agent forwarding mode, which would render a rogue server a pointless and unfruitful prank.

If you use the ProxyCommand mode, the only thing an "evil" sshmux would be able to do would be to break the connection. It won't be able to fake the endpoint if it is already in your known_hosts, as it does not have the real endpoints private host key. It is, therefore, secure in this mode.

erikb2• 11 years ago

Why not simply configure your ssh correctly?

guipsp• 11 years ago

Jump hosts exist for a purpose.

erikb2• 11 years ago

Yes, and there is a way to configure your client to use one.

tokenizerrr3• 11 years ago

Does this support SFTP? What about for windows users that use FileZilla and don't have an .ssh/config?

joushou25• 11 years ago

SFTP/SCP works if you use the ProxyCommand, or agent forwarding. Clients such as WinSCP support agent forwarding, at least. Not sure about FileZilla. I'm not a Windows user, so can't say. :/

tokenizerrr3• 11 years ago

I know FileZilla also supports agent forwarding, but I'm not entirely sure how this would work in practise. If I'm understanding things right you can have a single jumphost and give a user access to multiple targets. So when the user connects to the jumphost using SFTP then what files/directories do they see?

Maybe I should just install and try this for myself...

wila• 11 years ago

Looked at the code, it is nice and short.

However I'm not that familiar with go so might be overlooking something. Is there any logging included on who connected at what time from what IP?

joushou25• 11 years ago

Not at the current time. Will implement, just haven't had the time. I mainly code on this during my daily commute between Denmark and Sweden. :)

tinco• 11 years ago

You live in Copenhagen and work in Malmo or something? Is that common?

james_woods• 11 years ago

Why not simply use a VPN?

joushou25• 11 years ago

VPN's can be quite a pain, and is considerably more work than raw SSH. Depending on the VPN, the authentication strength is usually also quite a bit lower than that of SSH with RSA keys. There's also additional overhead. I have both VPN and jump hosts to get into our corporate network, and I most certainly prefer the jump host for convenience and performance.

There is also the user-level access restrictions of sshmux which will be much more difficult to replicate with VPN's. At least with pptpd, I believe it would require putting users on individual subnets with firewall rules restricting access to only their permitted hosts. Long story short, it wouldn't be a feasible solution.

VLM• 11 years ago

Probably 15 years ago I did something similar at an overall system level with a program called pdmenu which provided a jump host menu for semi-technical users. The technical details of course are entirely different, but it did the same general system idea of presenting individual users with custom tailored prompts to log into various systems (and a few other tasks, and logging, etc). pdmenu had (has?) a great menu CLI for end users.

jamiesonbecker2• 11 years ago

Jumpboxes aren't that bad to automate, actually.

We already help with automating jumpbox creation. (docs: https://userify.com/docs/tips/jumpbox/) and we're building even more jumpbox automation now. Don't allow root for any jumpbox accounts, but of course root escalation exploits abound. (I just found another in an AWS agent yesterday.)

Of course, as another commenter mentioned, if your jumpbox is compromised, than the jumpbox could serve as a gateway to your network. It's a tradeoff between exposing all of your servers to inbound SSH or only one.

There's another way, which is pure TCP forwarding on a different port for each server (ie 21321 -> inner server 22), but whether this actually reduces the attack surface is debatable, since the totality of open ports remains the same across the entire network.

My personal feeling is that using a jumpbox and locking it down (preferably to your company's IP ranges, etc) is the best way to go. You can also add MFA to the jumpbox entry point itself. We're going to help with automating all of that in the near future, too.

(disclaimer: CTO @Userify)

joushou25• 11 years ago

Hi CTO for Userify person!

One thing is setup of jump hosts (basic version just being a box with sshd enabled + users with authorized_keys, which is beyond simple), but it's the limitation of hosts. I have 1 big network where a set of users are only allowed to jump to 1 or more unique host each, with myself and a few others being able to jump to every single one. I believe that your solution requires things to be on different subnets to isolate them, with separate jump hosts for each subnet, correct? Here, I only need one sshmux for everyone.

Another "fun" feature is that sshmux can also throw unknown users to a separate host, such as ssh-chat, for support or hanging out.

Actually, in case of -oProxyCommand="ssh -W %h:%p jump_host", you're still fully secure in the case that the jump host (sorry for not calling it a jumpbox) is compromised. This is true both for normal SSH servers and for sshmux. They can mess with the original raw ssh connection that requests the forward, but this will just screw the connection up. They can connect it to a wrong host, but that will provide incorrect host keys. The agent isn't forwarded, so they don't get to sign anything with the private key. Messing with the new connection that gets established is subject to SSH's normal MITM resistance. This means that even root on the jump host would require real SSH protocol vulnerabilities to attack the connection.

Good luck with Userify!

jamiesonbecker2• 11 years ago

That's cool. Another way is to localtunnel to TCP forward to endhost in your config (ie two SSH connections, the first to the jump host, the second through the jump host, without agent forwarding), but that gets tedious fast without scripting support, especially if you want to use different portnums to avoid conflicts. ControlMaster would be a good choice to multiplex multiple connections to the same box.

I believe that your solution requires things to be on different subnets to isolate them, with separate jump hosts for each subnet, correct? Here, I only need one sshmux for everyone.

Right, we're automating the .ssh/authorized_keys (and sudoers) right now (https://github.com/userify/shim/blob/master/shim.py), so you can set it up however you like, as long as those boxes can get to either Userify for key updates, or your on-premises Userify server(s). Only one jumpbox is needed across any network, depending on how you prefer to lockdown your network. (The HOWTO refers to EC2, but it can be built anywhere, exactly the same way.)

sshmux looks like an interesting project, and love the simplicity. Wonder if Userify might automate the ProxyCommand setup! Seems really useful.

Good luck with Userify!

Thanks! We have 350 companies on the platform right now and growing fast, and Userify Enterprise (on-premise) is in beta at several large enterprises as well.

BTW, joushou, we're looking for brilliant SSH and golang hackers.. ping me if you get bored :)

zippie• 11 years ago

This is nice but the agent forwarding is a legitimate concern especially because a compromised host can take off with the private keys which are used while beginning the session. Most private keys are used for more than connecting to a single host.

In our setup we use jailkit allowing only ssh passthrough [1], we have added LDAP support to it (may release the patch later).

[1] http://olivier.sessink.nl/jailkit/howtos_ssh_only.html

joushou25• 11 years ago

What authentication do you have from the jump host to the target? To me, it looks like it has been reduced to either none or keyboard-interactive (password) login, which is considered bad practice. I could very easily implement support for this in sshmux, providing the setup you seem to use, as a way to avoid agent forwarding, but I just genuinely did not expect anyone to use password authentication, apart from in default config scenarios, before a public key has been installed.

Agent forwarding does not provide private keys, but only individual signing requests. This means that while the user is connected, and evil remote can request arbitrary signing, but only as long as the user is connected, and only as long as the users ssh agent is willing to do so. Using ssh-add -c further means that the user will have to accept each signing request. Also note that ssh -W, which is immune to any of these concerns, is fully supported by sshmux.

Correct me if I'm wrong, but "Jailkit" does not seem to stop an evil legitimate user from poking around other hosts with forwarding and such (which they can as long as ssh -W functions, within the limits of what a firewall allows the jump host to do in general). sshmux allows you to lock users to targets, rather than lock general capabilities of the entire jump host. This is because, while I provide clients access to various hosts, I do not trust them enough to have any unnecessary privileges.

thomashabets22• 11 years ago

https://blog.habets.se/2014/06/Another-way-to-protect-your-S...

Seems like the same thing. Even written in go, too.

joushou25• 11 years ago

I see why you might think that, but no. SSHProxy tries to move the private key from your computer to an external thing. SSHProxy doesn't have anything to do with sshmux.

SSHProxy is a two part thing:

1. A server that, when connected to, will ssh to the requested host, but authenticate with a local private key.

2. A client that will, when connected to the server, work like netcat for ProxyCommand purposes.

I'm skipping some info about additional SSH sessions being involved, as they don't provide any additional security.

The result is essentially ProxyCommand="ssh -W %h:%p jump_host", but instead of you having the private key, the jump_host has it. You then use a different private key to authenticate to the jump host.

I get the idea, and I by no means say he shouldn't have implemented it, but this provides no additional security what so ever. It just moves your private key from one machine to another. I hope he had fun implementing it, though! If you don't implement what might be a bad idea, you'll never get around to the good ones. Plus, bad ideas can be fun, even if they remain bad when done.

thomashabets22• 11 years ago

Ah yes, on closer inspection on a non-phone they are quite different.

You are just wrong on the no additional security though. In addition to enabling auditing, it also allows a username/password to be turned into an ssh key, so that you can know that if the rpi wasn't turned on, then nobody could log in to the target machine. It's similar to a TPM in this regard (as blog says), or can be a jumpgate protecting the systems behind it from compromised or malicious users.

I'm sorry you don't get it, but I'm not hurt by someone not understanding something when they criticize it. If you don't see the point of not having direct access to your key, then there's nothing I can do.

j_s• 11 years ago

I've used a similar setup to wrap RDP to allow employees to securely remotely access their office desktop. There is a market for someone willing to make this a turn-key replacement for Terminal Services.

Daviey• 11 years ago

Increasingly I am using sshuttle[0] to solve this problem. When you have lots of machines, ProxyCommand'ing always feels like such a burden.

[0] https://github.com/apenwarr/sshuttle

joushou25• 11 years ago

When I use SSH as VPN, it's usually because I want less flexible applications to use it, using socks5 configured as a system-level proxy. When doing more work on our corporate network, I usually SSH through the jump to my desktop and work from there, unless I specifically need to access another machine for something.

As for sshuttle, it's a very convenient device, although the design appears rather complicated. I'll look into it.

beagle33• 11 years ago

The design is possibly the simplest for what it does (and it does that exceedingly well).

sshuttle is:

- a tcp multiplexer (multiple connections onto one stream)

- a router (uses firewall rules to make normal connections go through the multiplexer)

- a few kludgy tunneling operations, to make the combination above seem like a real VPN (discovery of remote subnets, tunneling of DNS requests, ....)

- a default setup that runs the multiplexed stream through an ssh connection, thereby giving all the security guarantees that ssh provides (integrity, confidentiality, mitm resistance)

- packaged in such a way that the remote side needs to have a minimal python>2.6 install, and a user capable of making tcp connections. Nothing more.

In my experience, it works way better than IPSEC tunnels and some commercial VPNs that I've used. With two caveats that I'm ok with with: a) only TCP, and b) all connections seem to come from your remote ssh server. No VPN solution that I'm aware of makes so few demands; do you know of one that doesn't require root at the remote server?

elasticdog• 11 years ago

FYI, the main project is now located and maintained at [1]. Avery let someone else take over since he no longer had time for it [2].

[1] https://github.com/sshuttle/sshuttle

[2] https://groups.google.com/d/msg/sshuttle/jdTzJGjTDMg/IcQkCzb...

Sshmuxd: SSH proxy to replace jump hosts

Related Stories