Immutable Infrastructure: No SSH

https://boxfuse.com/blog/no-ssh.html

by axelfontaine • 11 years ago

89 80 11 years ago

boxfuse.com

huslage• 11 years ago

Docker?

bojo• 11 years ago

Still requires an underlying OS which may need to be patched or upgraded.

georgebarnett2• 11 years ago

Removing ssh ties the hands of your operations team when an outage hits because you have removed their debugging console.

With this change, the team loses access to all the standard OS tools which are useful in the event of a service outage for debugging. What usually ends up happening then is that the standard tools are replaced with substandard and patchy implementations.

If your infrastructure is drifting then a better solution is to restart it more regularly. If you find staff are still making untracked changes then that's not a technology problem - it's a people problem.

StavrosK• 11 years ago

A better alternative is to kill entire servers every once in a while and reprovision them. That way, console access effectively becomes read-only, since any changes will soon be lost, but you can still debug stuff.

nailer• 11 years ago

+1 netflix made Chaos Monkey specifically for that (not that anyone on HN couldn't code an equivalent, just that it's a technique used at scale).

georgebarnett2• 11 years ago

Absolutely correct. It's not clear from my reply however my intention is that servers should reset to a known config on restart. Edit; spelling

ninkendo• 11 years ago

For that matter, you could just take away the "server" abstraction altogether and run isolated processes in production instead, each having an ephemeral filesystem in the form of a container.

SSH access would be more like "heroku run bash" where you don't log into an existing instance, you log into a fresh ephemeral instance that just runs sshd and gets destroyed on logout.

Debugging running instances is more interesting though, and I think is possible if each container has an SSH daemon with ephemeral keys, and a central authority they register with that you can hop through to get shell on any instance.

Ops is a really fun job to be in when you stop thinking of servers as "things that people log into".

mpdehaan2• 11 years ago

This.

It's simply "don't perform install, maintaince, or upgrade mechanisms over manual means ever".

Do NOT cut off your SSH channel. You'll be kicking yourself when you want to sftp a file off of a box pretty quickly. And you might need it in an emergency.

Focus on good automation - whether immutable systems or otherwise, and be sure you can rebuild your infrastructure completely from source control. Do this by refusing to make manual changes, and knowing any manual changes will be often clobbered by the system.

aalbertson4• 11 years ago

While I agree (my boss and I are actually having this conversation presently), I would argue that if you're in any sort of cloud environment (which this article has set as a point of narrative), then just having an Ops s3 bucket or similar to send the files works. Although that of course still means SSH to get there "most likely". But could be accomplished without.

FTR, I'm in favor of restricting SSH/RDP, perhaps even disabling from the security groups (AWS), on a default basis, and enabling as needed for ops troubleshooting. If we have logging and monitoring in place cleanly enough, then we should be able to troubleshoot the majority of issues off instance. However that is almost NEVER the case (even if you have the tools, you still have too many edge cases). Case in point, I've been troubleshooting a Windows CPU usage issue recently. Our monitoring wasn't catching all of the disparate processes, so I couldn't see what was eating it up. Plus I would pull the node from the ELB prior to investigating, which then caused it to no longer peg CPU. Had I not been able to login while "live", I may not have ever seen what it was.

Obviously in order to perform such a function, one needs incredible reliability in your monitoring/logging tools. Sadly this is a more difficult function than it should be....

socketpuppet• 11 years ago

We do something similar with Nix (https://www.nixos.org). We have a CI server (Hydra) which creates a closure containing our software and all its dependencies, which we then upload to a network of AWS machines that are created with a few python scripts.

This works fairly well, but from experience we do need to keep an SSH setup on the machines. Just last week we had a load spike on a production server which was caused by a software bug, triggered by a usage pattern that was unexpected and not covered in our tests. If we did not have SSH (and access to the CLI tools on that machine) we would not have been able to debug this unexpected problem. I guess what I'm saying is that as long as software has bugs and hardware has glitches, we'll sometimes need access to low-level tools which can help us figure out the cause of these unexpected scenarios.

ris• 11 years ago

Exactly what I was thinking as I read this. Just use nix. Forget the dogma.

rotten• 11 years ago

Somewhere in the infrastructure there has to be mutable servers - databases and log servers at the least. I can't think of many architectures where everything can be a closed appliance.

Are they suggesting that every time you need to apply a security patch, you'll have to build a new server and cut your traffic over to it? That sounds like a lot of DNS and Load Balancer config updates (mutable!) even if the newly patched server builds are 99% automated.

The idea of black-box appliances has been around a long time, and it has its place in the modern infrastructure, but I'm not sure it really solves the problems they are trying to solve (which sound more like change management issues).

axelfontaine3• 11 years ago

This type of image is primarily designed for 12-factor apps where all persistent state is kept outside the instance in some geo-redundant highly available system like Amazon RDS or S3.

mugsie4• 11 years ago

RDS is not "geo-redundant".

It is an multi-AZ DB, which is very different to multi-region. (Availability Zones (AZs) are within a DC, regions are separate DCs)

falcolas2• 11 years ago

The problem with Amazon RDS, and they freely admit it, is that it's not built for the scale at which some companies operate. If you need advanced features of your DB, or need to operate at large scales (the magnitude of scale differs by RDS engine), you'll end up wanting to run your own DB on an EC2 instance.

eyko• 11 years ago

Trouble starts the minute you start relying on commands like this:

    sudo apt-get install mypkg

So what's wrong with `sudo apt-get install mypkg=<version>`?

yebyen3• 11 years ago

Do you do this every time? Must be cumbersome, there's no equivalent of Gemfile.lock and bundler for apt-get, is there?

(Or is there? Serious question)

oblio• 11 years ago

https://help.ubuntu.com/community/PinningHowto

My guess is that apt pinning/holding predates Gemfile.lock by quite a few years :)

icebraining5• 11 years ago

Usually configuration management tools like Ansible are used.

felipellrocha• 11 years ago

Well... In python, we do that the first time around. After that, we get a file saying exactly what version was installed for future installs. It'd be nice to have something like that for package management... Thinking about it, the issue of drifting seems to be more of a problem with the packaging system, than ssh...

sciurus• 11 years ago

Instead of using public repos and pinning the version you want from them, you can run your own repo and import the versions you want. Running your own mirror is a good idea anyway, and there are tools like http://www.aptly.info/ which simplify the process of locking packages in environments.

zimbatm• 11 years ago

mypkg-v1 depends on libsomething. mypkg-v2 doesn't depend on it anymore. Now you have machines with libsomething leftovers.

Another issue with debian packages is that dpkg essentially removes the old package before installing the new one. If you're packaging assets served by nginx/apache then there is an availability issue in the mean-time.

icebraining5• 11 years ago

mypkg-v1 depends on libsomething. mypkg-v2 doesn't depend on it anymore. Now you have machines with libsomething leftovers.

That's what "apt-get autoremove" is for.

acdha• 11 years ago

Both problems are only an issue if you don't have automated deployment or more than one server. Otherwise, you can simply drop a service from a load-balancer / cluster when doing an upgrade and add it back in once it's back up and passing health-checks.

Config drift can be avoided by using the same process to create new VMs and delete old ones, which is also part of confirming that your redundancy setup really works.

stonemetal• 11 years ago

Nothing in the heat of the moment. The problem comes six months down the road when something happens and you need to build that box again(horizontal scaling, box death, whatever). Your provisioning script is still point at correct version - 1.

Sanddancer2• 11 years ago

The new box will be brought up to netboot via PXE, which runs whichever automated installer your OS uses, including assigning IPs, partitioning the drives, etc. The script then points to the proper configuration management system you use, which chooses the proper script for whatever the box is -- web frontend, database, etc -- and then finishes out the install. You're never a version behind, because your installer is maintained with every other bit of configuration. A good deployment system means new boxes are never out of date.

therealmarv• 11 years ago

What about logs? What about looking into the server to find out what is going on on the server?

axelfontaine3• 11 years ago

Application logs should be shipped over the network to either your own ELK infrastructure or some hosted solution like Loggly, Logentries or Papertrail. Boot logs can be obtained from your infrastructure (like EC2 instance logs).

spacecowboy_lon2• 11 years ago

And when that stops working for some reason? of course the real solution to have out of band managemnet

dschiptsov• 11 years ago

May I suggest another "innovation" - statically link everything into a single blob and run it under systemd.

The building scripts should be written in JavaScript, of course. Package it as a startup. Become millionaire.

mugsie4• 11 years ago

"Vulnerabilities like ShellShock simply vanish" - eh .... not really.

Shellshock was not just from SSH, it was any program that used environment variables to pass data around.

Immutable infrastructure is a great ideal, but has various downsides. sometimes people need to access the raw system to do debugging. While shoot the node, and boot a new one has advantages, it does just move the problem down the line.

The BBC solution is a nice balance.

viraptor• 11 years ago

It's quite silly. I mean they mix up two completely separate issues. Deployment problem and people problem. Image deployments are not new - computing clusters have done it for ages. Immutable deployments aren't new either - actual immutable filesystems booted over network were there for many many years.

Yes, that's going to prevent drift at first. No, it's not going to prevent people from doing silly stuff. At least ssh allows them to do silly stuff in a "normal" and visible way and doesn't stop you from debugging issues when you actually need it. If you enforce no-login without teaching people, they'll just reinvent it... badly - for example by making executable configurations, or pseudo-web-shells.

And there's one big issue with images which will cause a lot of friction in real teams - they're slow to build. And even slower to test. If you're doing it properly it means first testing locally on some approximate environment, then handing the package over to a system which will test the whole deployment, which takes ages to boot up all components. (time scales with system complexity - it's not ages for a single webserver of course) Anyone developing something related to integrating services will get additional frustration unfortunately.

Enforcing immutability [...] You need to prevent log in.

No... If you need immutability, you make your system actually immutable and then audit logins when logins are needed. Preventing logins is just punishing people and slowing them down when they need it the most, because you don't trust them.

Vulnerabilities like ShellShock simply vanish

Said a person who does not understand why ShellShock was an issue and why it doesn't matter if a user who already has local access uses shellshock.

mugsie4• 11 years ago

As someone who worked in a company that didn't always have access to the customer machine that ran our software, I can confirm.

There might have been a magic page in the software that allowed raw SQL queries...

shangxiao• 11 years ago

Vulnerabilities like ShellShock simply vanish

I'm actually surprised at the naivety of whoever wrote this. I'm not an op at all and I understood enough about ShellShock to know that this isn't correct at all.

amalcon• 11 years ago

Vulnerabilities like ShellShock simply vanish.

What? No, they don't! ShellShock was about executing unintended code in a bash process where you control an environment variable but not stdin. If you control stdin, there's no need to sneak a command into an environment variable: you can just type it into the shell.

Apart from this, the article makes no argument for removing SSH: it only argues for restricting access to privileged userids. That is to say, the standard best practice for decades. Root should be reserved for special situations, not used by default.

exarch• 11 years ago

This is what happens when dev teams hate ops teams. Troubleshooting a production bug without a terminal is a recipe for lengthy outages.

jacques_chester2• 11 years ago

I worked on Cloud Foundry buildpacks for 7 months. I was working at the Pivotal Labs offices in NYC and I can tell you this for free: devs want SSH access too.

When a box dies in staging, it's nice to learn why.

lamontcg3• 11 years ago

Yes, exactly.

This is an old-school ops team that learned about "devops" and "immutable infrastructure" and figured now they have all the necessary arguments to kick the devs off the servers once and for all. Not progress.

asymmetric• 11 years ago

This is not even a problem if you're using Ansible (and possibly others): you just say you want the latest version of the package installed, and ansible will to an `apt-get update` for you as a preliminary step, and install/update the package if needed.

So basically, all the machines you're running ansible on will have the latest version (if that's what you want, of course)

stephen• 11 years ago

Interesting; so, boxfuse is one of these (..."micro" kernels? What's the name, of when you run in a hypervisor without a real OS, so no user isolation, no process isolation, etc.) OS/image-creation tools...

I didn't realize they were production ready yet, but sounds pretty spiffy.

What's interesting is that I think Boxfuse is spinning what most people would call a weakness ("if stuff breaks, I can't SSH into the shell and fix it (because, not only is that a 'bad idea', there just isn't a Linux OS there to SSH into anyway)" and calling it a strength.

Nice spin. :-)

If I tried boxfuse, I'd probably look at embedding the Apache Java-based SSHD server, not for app setup, but debugging ... assuming its terminal session would provide useful information in the Boxfuse "there is no OS" environment.

(The boxfuse site uses the term "Secure Micro OS, Few MBs", so if I'm misinterpreting what their platform does, someone please correct me.)

kazinator• 11 years ago

"Sign up for your Boxfuse account."

What? And disturb the tranquil immutability of boxfuse.com?

waspleg• 11 years ago

I admit I lol'd.

kaonashi• 11 years ago

It actually gets swapped with a new Boxfuse.com on write.

grhmc3• 11 years ago

So I've considered this before, and I know it is coming from a good place. It seems great for the good times. At Clarify.io we use immutable infrastructure and we don't log in to boxes. Except when there are problems. And there are problems. Always.

You don't want to have to fight your way in in the middle of a disaster.

At Clarify.io we've considered having an on-login event fire when an admin uses SSH to log in to the box. This would schedule the server for termination within 24 hours. This gets you best of all worlds:

- Gives tools to debug and restore service when there is a failure

- Forces admins to not depend on SSH to bring a system up

- Replaces servers when they are potentially drifting

- Encourages a "cattle" mentality about the servers

aalbertson4• 11 years ago

I really like this idea. Would be keen to see what your orchestration code looks like to handle that.

grhmc3• 11 years ago

We already use JanitorMonkey. It would likely involve tagging the instance in AWS with a tainted tag, and adding code to JanitorMonkey to do the normal mark-and-sweep.

HugoDaniel• 11 years ago

On *BSD kernels you can use securelevel to achieve proper immutability to some extent (like unchangeable pf rules, read-only raw disk devices, etc...). Disabling ssh only gives you the guarantee that ssh is going to be disabled, and that is quite different from immutability.

tgeek2• 11 years ago

I find the better pattern here is to limit and discourage SSH and then monitor and log the hell out of it. There are numerous tools out there that can centralize any actions being taken on a host and sending it to a log that can be centralized. Outright removing all SSH puts you in a rough spot if things go south with some piece of software that your system monitoring/centralized logging don't cover 100%, and makes it way hard to do things like strace on a process.

grhmc3• 11 years ago

Or if one of the problems of the server caused logging to not correctly start :)

aalbertson4• 11 years ago

Which is certainly a valid case, but I would argue that if that IS the case, and it cannot autorecover, then that is a good candidate for the instance to self terminate.

Sanddancer2• 11 years ago

If logging never started, then your service heartbeat script should have caught it and raised an alert. Logging being down is Kinda Important.

sneak• 11 years ago

Also, make it part of the process that each time ssh is used, logging or monitoring is set up to catch what it was used for, much the same way that a test is added when a bug is discovered.

mwcampbell2• 11 years ago

A common response on this thread has been that if you take away SSH, you can't log into individual machines to debug. But you know the other situation where you can't do that? When your software is running on your users' machines rather than yours, i.e. mobile and desktop apps. I agree with the position of the OP, that it's better to make the software robust by design and through testing.

icebraining5• 11 years ago

it's better to make the software robust by design and through testing.

But that's a false dichotomy; having SSH in no way prevents you from designing and testing your software just as well.

juliangregorian2• 11 years ago

I don't know, I think there's a legitimate argument to be made that some tools can become a crutch and a detriment. At one of my consults all the developers lean heavily on IDE debugging at the expense of developing well-defined contracts and interfaces. Nobody blinks twice at having a ton of threads mutating global variables because they have all these great debugging and tracing tools.

breischl2• 11 years ago

If you're unable to debug due to the nature of your software then that sucks but it's life. And notice that lot of devs in that situation try to fix it with various remote debugging tools.

The article seems to advocate putting yourself into that situation when you don't have to, which is silly.

mwcampbell2• 11 years ago

The hard reality of no remote debugging in software deployed to end-user machines leads a wise developer to make potentially inconvenient trade-offs to compensate, e.g. reducing the chance of failure by minimizing dynamism. I believe, and I think the OP would agree, that applying the same discipline to server-side software leads to more robust applications, whereas if you know you have the crutch of remote debugging (via SSH or otherwise) when something goes wrong, you may well be more lax about preventing failures in the first place.

ebiester• 11 years ago

And if you are well funded, that may not be an issue. However, many people don't live in a world where they can afford to build the tools needed before they determine the viability of their product.

peterwwillis• 11 years ago

What is drift? Local edits, updates not applied, literally ntpd/ntpdate not updating the time. Your host is misconfigured; it's broken.

How do you deal with systems breaking? By monitoring for them. It also helps to make them 'disposable', but this is not a replacement for monitoring. Good monitoring will tell you when time is off, when disks are about to fill up, when permissions are bad and when packages aren't up to date.

Removing ssh does not give you monitoring. It does not prevent system state from changing. It really doesn't do anything but remove a very secure and simple method of communication and file transfer.

Minimal/disposable images? That's fine. It won't make your servers or services immutable. One day your hardware's going to catch on fire and your first clue won't be the smoke billowing into the neighboring rack, it will be the errors in your logs.

Sleaker• 11 years ago

Uhh... you now how we handle drift? We manage the update repository ourselves. New packages don't get released until we are ready to release them....

You don't need to limit ssh to help handle state drift. Article writer doesn't seem to understand the difference between user access, and software deployment.

njharman• 11 years ago

I ssh into machines for many reasons other than to change them. In fact most changes are done remotely with Ansible.

lamontcg3• 11 years ago

This is just dumb as nails.

Now I can't login to the production server in order to strace the process to figure out why production is misbehaving.

And don't say you should always replicate the issue in preprod, sometimes you simply can't. Some issues only emerge at prod scale+load+latency that you don't have the financial resources or technical ability to be able to exactly replicate in a testbed.

You are drunk on too much koolaid, go home.

juliangregorian2• 11 years ago

That's great if strace et. al. can actually tell you why production is misbehaving. Of course, strace doesn't exist in vacuum, it changes the way your code executes. Everything is going to be slower, and race conditions and locks may not present themselves so readily. Then you also have the added stress of screwing around with a live server, what if you do something that messes it up worse?

Sometimes it makes more sense just to kill that server and add a fresh one to the pool. Better yet if that process is automated. Good logging can actually be more valuable than mucking about in the live environment (or as I call it, panic mode).

siliconc0w• 11 years ago

I'm a fan of immutable infrastructure but there is still a need to get access to a container now and again to troubleshoot things. We don't run SSH inside the containers but it's easy enough to write some tooling to find a host running an instance and use docker exec to get a shell on it.

OriPekelman• 11 years ago

This is so the wrong question. Immutable infrastructure does not require nor warrant restricting the form of access you have to your services.

Having everything build-oriented (basically building the builder on every deployment, than using it to build the service) is what prevents drift.

Running on a read-only file system is what guarantees immutability and prevents drift.

Not allowing SSH is plugging a single hole which guarantees nothing. Can the app-server write on its own disk? Can it modify configuration files? Maybe even code?

As a developer I want to be able to write applications that are not specific to any one deployment method. I want to be able to change my mind to host this on different kind of infrastructure. These kind of limits force you to create a strong adherence between your application and its tooling and the deployment method.

Of course, the Read/Write domain has to be explicit and limited, but this is good practice anyway. Running on a distributed storage grid based on something like CEPH can allow you to address this domain not only explicitly but also in a fault-tolerant and scalable way.

When you cut off SSH you are cutting off half of the tooling we have and you make everything overly complicated. I want to be able to tail any log on any host. I want to be able to scp for inspection any file to my dev machine.

At platform.sh (a PaaS running on a distributed grid of micro-containers) we run a build oriented immutable infrastructure so there is no, there can be no infrastructure drift.

We proxy SSH so security stays the responsibility of the orchestration layer not any single host or service. This means we can filter by role any connection to any service, but developers can still just use the tools that work.

Immutable Infrastructure: No SSH

Related Stories