Bastion hosts

https://www.pandastrike.com/posts/20141113-bastion-hosts

by slem • 12 years ago

62 57 12 years ago

slem• 12 years ago

Bastion host(s) are a useful and important component of a system management infrastructure. A bastion-host, in this context, is actually more properly, but more obscurely, called a jump server. In this post I will simply use the term bastion host. It is the most commonly used term for the system's function: a server, which has undergone security hardening steps, that is the operational and administrative control point for systems and hosts in a datacenter (or AWS Region).

neohaven2• 12 years ago

Direct quote from the page at the other end of this link :

"You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards.

Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)"

This is followed with links for Firefox and Chrome. I am running Firefox 24.4 from CentOS repos. This is unskippable, and there's no simple way to bypass it on the page. On a browser from March of this year.

Urgh.

GrinningFool• 12 years ago

Yeah. Welcome to the world of an HTML 'standard' that is 'living'.

tedunangst• 12 years ago

Since it's only a plaintext page, you can read it with curl. Or you could, except the server is broken and sends back gzipped data even if the client doesn't request it.

    curl https://www.pandastrike.com/posts/20141113-bastion-hosts | gunzip | more

neohaven2• 12 years ago

I shouldn't have to grab a page curl to see a website. A disclaimer of "Hey we notice $X is your browser, maybe switch to those?" with a "No thanks, I'll deal with it" option on some modal JS dialog.

cdubz• 12 years ago

TBH, > 6 months old is hopelessly outdated. Welcome to the future, grampa.

voltagex_• 12 years ago

Some of us are using browsers imposed upon us by $EMPLOYER, and may not be able to update of our own volition.

ryan-c• 12 years ago

I think Firefox 24 is still supported for a few more weeks, as it is a "long term" release.

vacri• 12 years ago

The point of those LTS browsers is so that an employer doesn't have to watch for UI changes in their browser every 6 weeks, and then retrain all their staff when the browser moves things around or makes things look different.

tptacek4• 12 years ago

Jump boxes are a good idea. A better idea, if you're in AWS, is to use a VPC and just VPN into your environment; the VPN terminator in the VPC serves the same purpose as a jump box.

If OpenVPN scares you, you can still do layer-4 filtering to minimize the impact of losing the OpenVPN server (ie, to get it to "no worse than if we hadn't used VPCs in the first place).

"Single-packet authentication" is, I think, silly, and I've never recommended it to anyone. It's hard to think of a case where hand-rolled authentication in front of SSH has saved anyone with a proper SSH configuration (keys-only-no-passwords). You can skip that step.

mahmoudimus• 12 years ago

Do you have a good reference to administering an OpenVPN server? Setting it up is easy, setting it up with HA is also no problem, but administrating OpenVPN for a startup that doesn't have a dedicated sysadmin would be a very useful guide.

phil21• 12 years ago

A better idea, if you're in AWS, is to use a VPC and just VPN into your environment;

Is it though? Is having your (and your co-workers) desktops/laptops/and possibly more directly able to access production servers at a packet level a good idea?

I would posit that it is not. A bastion host can be used to lock down more than firewalls, and reduces the attack surface living on your network considerably.

VPN terminator in the VPC serves the same purpose as a jump box.

It certainly serves the same network separation purposes a jump box does, but is not able to dictate the access environment nearly as much. If only your jump box has access to servers (and production servers properly filtered), every bit of remote access must go through that single ingress/egress point. And that point can have security policy built-in as well for multi-person teams. For example: I can enforce certain key types being utilized on remote hosts, logging of ssh sessions, etc.

I also agree the "single packet authentication" is silly though. Most bastion hosts these days will be behind VPN already. Just another riff on port knocking really.

jpgvm3• 12 years ago

Bastion hosts are generally a bad idea. There is not a single implementation of a bastion host that doesn't have significant security holes. This is doubly true if you want to do proper group based access control.

Most notable of these issues is the nature of either needing SSH agent forwarding enabled or storing a copy of the private keys on the bastion host (please don't do that). There are ofcourse ways of dealing with this but they aren't worth it.

It's fundamentally much better to just harden every system and implement proper federated authentication and authorization with LDAP. It also makes it much nicer to implement things like MFA and device level authentication.

Remember, it's very rarely that an attacker is going to be able to successfully walk in or break down the front door. Your security holes are very unlikely to be in your administrative channels like SSH/RDP but rather crappy applications or services you may be running on your systems.

Those crappy bits of software + insider threat are the 2 main things you generally need to deal with. Ensuring systems are sufficiently isolated from one another helps with both of these and bastion hosts just add another layer of shared stuff between workloads that should stay entirely separate.

akerl_2• 12 years ago

Calling fwknop "hand-rolled auth" seems like quite the stretch. If passing around requests via GPG signed and encrypted messages is "rolling your own crypto", then what isn't hand-rolled?

tptacek4• 12 years ago

I don't care what you call it; the point is, it doesn't save you from anything that the correct SSH configuration doesn't already protect you from.

If you really believe that OpenSSH is too scary to expose to the Internet for fear of memory corruption flaws, don't stop at a half-measure like "port authentication". After all: once you "authenticate" the port, the attacker can simply hijack your TCP connection to trigger the flaw.

Instead, write a plugboard proxy, and have it apply a static-keyed AES-CTR transform to the data.

I think this is silly, but if you're trying to protect yourself from OpenSSH bugs, at least do something that actually protects you from OpenSSH bugs.

sneak• 12 years ago

"Single-packet authentication" is, I think, silly, and I've never recommended it to anyone. It's hard to think of a case where hand-rolled authentication in front of SSH has saved anyone with a proper SSH configuration (keys-only-no-passwords). You can skip that step.

I agree. I usually use simple port knocking for that purpose. Surely you can agree that "no ports listening" is a better circumstance than "tcp/22 open to the world, with key-based-auth only"?

A VPN is redundant (and requires special client), whereas port knocking to open a port for ssh (that only supports key-based auth) is a simple matter of a few "nc -w 1" lines in a script before you ssh...

EDIT: Saw your posts below. You presuppose that such OpenSSH-zero-day holding attackers are specifically targeting you, which is not invalid; however a spray-and-pray quickly after public release is much more probable in a lot of cases.

JoachimSchipper• 12 years ago

This thread is dead, but: everyone really should enable OpenVPN's tls-auth option. tls-auth simply HMAC's each packet that would otherwise be fed to the SSL protocol, which is really good for reducing the impact of SSL-vulnerability-du-jour.

(tls-auth doesn't protect you from vulnerabilities in the OpenVPN data protocol, but that's a lot simpler.)

cthalupa2• 12 years ago

Jumphosts have their uses, but in this case, I really don't see it. As others have pointed out, better yet is VPN only access.

As for the port knocking silliness, well, I don't see any reason for it.

http://bsdly.blogspot.com/2012/04/why-not-use-port-knocking....

akerl_2• 12 years ago

It's worth noting that single packet auth is distinct from port knocking. fwknop isn't relying on "secret" port combinations: it's using GPG to sign/encrypt requests.

cthalupa2• 12 years ago

It's not meaningfully different - most of the same flaws still apply.

Say you have a bad actor behind the same NAT'ed IP. Good User A signs their SPA, and the firewall opens up for 30 seconds... during which Malicious User B has access.

What real advantage are you gaining vs. just having restricted firewall rules in the first place? Lowering the window of time an attacker has access? If you're not using password auth that could be brute forced, who cares about the window? Either they have the keys or an exploit needed to get in, or they don't.

rsync3• 12 years ago

I've been doing this sysadmin thing for a long, long time, and have built some large, high profile platforms,[1][2] and I can tell you that few things have made me happier than port knocking.

Yes, it is a password with a tiny number of "bits". But nobody suggested using only port knocking, so that's not significant.

Yes, if my attacker is sharing the exact same NAT'd IP as me then they have a window to attack ... my fully hardened server, which was no worse than what was presented to them before.

Finally (and this is what makes me happiest) my sshd is not exposed to whatever new exploit might be discovered tomorrow that doesn't require user credentials. It's not exposed to being DOS'd by a buffer overflow vuln. It's not filling my logs with brute forcers all day and night.

I love port knocking.

[1] JohnCompanies (first VPS provider)

[2] rsync.net

philjr• 12 years ago

just run ssh on a non-standard port and you get the same benefits

peterwwillis7• 12 years ago

Mr. Sysadmin, I have four questions:

1. When was the last time your sshd was hit with a 0day?

2. Why aren't you running grsec to prevent buffer overflows (amongst other things) ?

3. Wouldn't it be easier to DOS your host with excess traffic (like all modern DDOS attacks), rather than find a remote-code-execution buffer overflow 0day and use it solely to starve resources on a bastion host?

4. Why don't you use a VPN? If you used a udp-based VPN nobody would even know the port was open (connectionless) and you could even require a preshared key, which is the more secure equivalent of port knocking.

kev009• 12 years ago

This article goes from mediocre to bad when it starts advocating security by obscurity.

the_french• 12 years ago

Security by obscurity is perfectly fine when it's part of a layered approach. It will help keep attackers from learning the details of your architecture and will also stop skiddies in their tracks.

rsync3• 12 years ago

Exactly.

(Your most secure server) is not as secure as (your most secure server with some obscurity added).

arde6• 12 years ago

This. Obscurity has too much bad press. It's not to be depended on, but it can be a useful addition sometimes.

lmm• 12 years ago

Bastion hosts are the wrong approach, as are VPNs. They might have had some merit in the days of handcrafted servers, where "hardening" was something you could do to a single machine. But nowadays, especially on AWS, there's no point in a configuration that isn't automatically reproducible - in which case, why not harden all your machines?

Then you avoid having a single point of failure. If and when a single SSH key is compromised, there are a limited number of hosts that key had access to - and unless that's the log server, you have secured logs so you can figure out what happened. Different keys can have different capabilities, rather than a one-size-fits-all policy on the bastion. It makes it harder for an attacker to know where to get the boss's keys. And you can expand your system geographically without the performance compromises of a VPN; your servers already make encrypted, secured calls to each other (over SSH, SSL or a very small number of vetted protocols), expose only those ports that are actively used. Your attacker could steal your entire system documentation and it wouldn't make any difference, because the system is truly secured, not relying on obscurity.

jpgvm3• 12 years ago

Better yet. Store the authorized public keys in LDAP or similar system. (AuthorizedKeysCommand makes this super simple these days).

That way you can rotate them easily if the private keys become compromised.

I recently built a system using this technique and synced the keys to S3, very effective.

rudolf02• 12 years ago

You're describing an overly optimistic scenario. "Why not just harden all your machines?" Because it's more complicated than that. Yes, you can automate basic hardening (configuration of access controls, limited use of SSH keys), but how do you automate hardening of RPC between one server and another, or hardening of those 3 internal administration web apps which may or may not have some local file disclosure and crypto flaws?

Defense-in-depth is still the only sensible way to handle security. You do what you can on the host side, but bastion hosts, jump hosts, VPNs, etc. provide an additional layer to make lateral movement much more difficult for an attacker.

peterwwillis7• 12 years ago

In the Enterprise world, jump boxes are more typically just a hack implemented by IT workers who don't want to go about properly setting up a firewall and 802.1x/vlan/vpn/ipsec/etc to allow strict control to a protected network resource. (Or because they don't have the hardware capability to do it, or their network admin decided against it).

The only two proper excuses for ever using a bastion host are "I don't have the money to do it right" and "I don't have the time to do it right."

corford• 12 years ago

I'm a fan of the VPN terminator as jump point approach.

OpenVPN with tls-auth switched on is a pretty robust front door. If you combine that with enforced key only auth for ssh, ensure all internal machines run their own firewall and logically split different bits of your internal network off in to their own subnets you've got a fairly secure foundation to build upon. Another easy win is switching internal services over to TLS/SSL only (where it makes sense/is possible to do so).

nullc• 12 years ago

You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards.

Bastion hosts

Related Stories