Who's your SSH buddy?

http://blog.jgc.org/2011/07/whos-your-ssh-buddy.html

by jgrahamc • 15 years ago

149 101 15 years ago

blog.jgc.org

rb2k_• 15 years ago

> someone I could call and give credentials to so they could log in and shutdown the machine.

I'm not going to be able to read out my private key over the phone. I guess this is only for password based authentication.

bhousel• 15 years ago

Well, ideally you'd work all this out with your friend before the emergency...

omh• 15 years ago

And this means you can add their public key as well, meaning you can control (i.e. revoke) their access easily and don't have to give them any root passwords.

markbao3• 15 years ago

Maybe a play off of In case of emergency, break glass with hammer and copy private key: a one-time-only retrieval of a private key at some URL, so when your buddy retrieves it, it triggers an email that goes out to you.

joelhaasnoot• 15 years ago

I think I need that for myself: still scared I'll forget a password or passkey somewhere, or forget what that one really long password is for.

LeafStorm• 15 years ago

Privnote (https://privnote.com/) can do exactly that. For added bonus, the message is stored encrypted on their servers and decrypted with a key in the URL's fragment.

JoachimSchipper• 15 years ago

An alternative: give out a public key, but restrict it to run e.g. reboot only. Put something like the following in your ~/.ssh/authorized_keys:

    command="/sbin/reboot",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB3Nza...LiPk== user@example.net

See sshd(8) "AUTHORIZED_KEYS FILE FORMAT" for details.

bostonvaulter2• 15 years ago

I think you mean give out a private key.

buro9• 15 years ago

Why not just pastebin your private key, shortURL it and when the time comes you only have to give the shortURL.

A key alone, with no idea what it will unlock, is going to be a useless thing for anyone else.

Obviously I'm aware how crazy a public store of private keys sounds, but keys are really only useful if you know what lock they fit.

Auguste• 15 years ago

Just for the record, Pastebin is not completely secure. Even private pastes can still be accessed if you know the URL - they aren't password protected like your account may be.

But like you said, if you posted the key alone anonymously, it will be hard to guess what lock it fits.

emilsedgh• 15 years ago

Sharing your private key for 'more security'? I will never share my private key with anything or anyone (pastebin or my ssh buddy).

koushikn• 15 years ago

No ssh client still on iphones?

jgrahamc2• 15 years ago

I should have had one installed. Now I do. But I still need an SSH buddy because I might not have data access (for example, if I am abroad).

corin_2• 15 years ago

Hook it up so that a few preset operations can be called via SMS, e.g. shutting down the server?

kondro5• 15 years ago

It's expensive, but have data-roaming enabled on your phone. You'll only be using it for emergency server-maintenance after all - and surely you can bill it to your employer in those emergencies anyway.

espo• 15 years ago

My SSH buddy is iSSH from the App Store. Always available when I need him.

ceejayoz2• 15 years ago

Assuming you've got a data connection, that is.

bahman2000• 15 years ago

can you not ssh from your phone?

thehodge• 15 years ago

Isn't there a service where you can SMS something like shutdown, restart or other commands for the box to respond to, that would help in that instance where net access is poor

ErrantX4• 15 years ago

My custom monitoring program (probably not as advanced, it only does some light monitoring) accepts a text message back to shutdown the machine.

Not perfect but can work well as a line of defence.

markbao3• 15 years ago

That's cool. How does it work? Is the text message just via email, and the server can accept an email back with a specific command?

ErrantX4• 15 years ago

Servers are with Gandi - which has a hosting API. So... SMS gateway via some shared hosting I have had for ages. This calls a script which figures out which server I mean and uses the API to shut it down.

I did used to use Twilio until they dropped international text messaging etc.

ma2rten• 15 years ago

I'm not an expert in this things, but can't sms messages be spoofed quite easily?

ErrantX4• 15 years ago

I presumed they can be :) so my systems needs a text like this:

It is on my todo list to improve this at some point - but not had chance yet.

kondro5• 15 years ago

Umm... my iPhone, iPad and MacBook are my SSH buddies?

keyle• 15 years ago

Any idea of what he uses to send SMS from his box?

troels2• 15 years ago

If you don't want/can afford an sms-gateway, you might want to look at notifo [1]

[1] http://notifo.com/

kevin_morrill• 15 years ago

Twilio would work.

http://www.twilio.com/docs/api/rest/sending-sms

ramidarigaz2• 15 years ago

I dunno about jgc, but I just send an email to myphonenumber@txt.att.net

Works like a charm!

urbanjunkie• 15 years ago

Before you get into half-arsed solutions like having an ssh buddy, disable root login and password authentication.

troels2• 15 years ago

I assume that he already did this. That the alert comes when someone assumes root. But maybe I'm being naïve.

jgrahamc2• 15 years ago

I don't understand what's 'half-arsed' about deciding up front who I can trust to have login credentials for my personal machines.

Zakuzaa• 15 years ago

Prevention is better than cure.

kahawe7• 15 years ago

Ideally you should not have to circumvent the default PermitRootLogin settings; it is there for a reason. And no need to give out root passwords, if you are so security conscious.

Our polo-necked messiah has given us iPhones that can even run an SSH client so how sort-of pointless to talk about "ssh buddies" and showing iPhone screenshots in the very same article? Since you were getting IDS alarms on your iPhone anyway, why didn't you SSH in and shutdown the machine yourself?

And on the root login bit: you would only need a sort-of dummy account for you logging in or for your "ssh buddy", if you love that idea. And all they need to be able to run is "sudo shutdown".

siphr• 15 years ago

Well on the surface nothing, however login credentials can be stolen easily. Focus is on "easily". Sure keys maybe hijacked but the problem becomes non trivial and I am all for making burglars jump through a monkey bar before they even get to the door.

code_duck• 15 years ago

Nothing wrong with deciding that. The proper way to give them access, however is to give them an SSH key, not a name/password set.

iamichi• 15 years ago

Doesn't seem like front page news. Use key based auth and disable root login. You can also get an ssh client on your iphone if you don't mind keeping your private keys on it, just make sure to use a long password and be prepared to change all your server keys if you lose your phone. I also ban every IP that even tries to login as root.

yaix• 15 years ago

The remote root login was my first thought too. Especially when I saw the sentence "... I'm pretty careful with machine security ...".

What's bad about password auth if your password is strong enough?

eru• 15 years ago

With passwords you are handing out what is needed to log in. With the ssh public key scheme, you only prove that you have the private key, but don't hand it out to the server.

munin• 15 years ago

passwords are replayable tokens. one-time passwords are not. people who actually are careful with machine security use ssh keys + sudo/opie.

ars4• 15 years ago

In /etc/ssh/sshd_config do:

PermitRootLogin without-password

That way root must use a public key to login, but other people can still use passwords.

(But watch out for sudo.)

Udo2• 15 years ago

Yes, but the issue here is not root login, it's the prospect of something going wrong with your server(s) and having a trusted buddy around whom you can call on for assistance. It's actually pretty reasonable. My servers always crash when I'm literally somewhere in the jungle in Asia and can't possibly get to an online connection for at least a day. It's great if you can just call someone who takes care of the most pressing problems if/when they happen.

jrockway3• 15 years ago

My servers always crash when I'm literally somewhere in the jungle in Asia and can't possibly get to an online connection for at least a day.

Get a watchdog timer.

moe• 15 years ago

I think that guy needs to work on his paranoia.

kahawe7• 15 years ago

...or at least work on the security in a more well-thought way!

dmoney• 15 years ago

Why not have the IDS shut down the machine itself?

modokode2• 15 years ago

Wouldn't a OTP scheme like OPIE be able to do this decently? My own computers do not permit root login via ssh, but still it'd be more or less a trivial matter to set up a local account for your "SSH buddy", and then over the phone give him the next OTP in sequence for the root account or so for a successful su - -c halt.

skrebbel• 15 years ago

i don't get this. is it about a personal site? if so, how paranoid can you be? or is it about a professional site? if so, who not try calling a colleague?

hardy263• 15 years ago

Though the main problem if someone breaks into your system, the first person you'd suspect is your SSH buddy, because there is a possibility of them using a public machine to log in. Who would want to carry the burden of being your first suspect?

forgotusername• 15 years ago

This is a pretty pointless idea. If someone has root on your machine for even 500ms that's already too late to trust any of its state without a complete reinstall (and even this is not enough for an advanced attacker).

Inevitably, if you have such a buddy, all they can really say is "yup, I'm not the only user here", and you're back at square one.

A better investment of the time would be systematizing your configurations, so in this situation you can just rebuild a new machine and kill the old one (ideally, after figuring out how the password was compromised in the first place).

davidw• 15 years ago

Presumably the point is not "have someone that can fix your machine if it actually gets rooted", but "have someone you can call in case of an emergency who is competent".

Seems like for a business, this would be a great service to have available from the likes of Linode, which pays people to be there in any event, rather than your 'buddy', who might go on vacation when you do.

ceejayoz2• 15 years ago

> This is a pretty pointless idea. If someone has root on your machine for even 500ms that's already too late to trust any of its state without a complete reinstall (and even this is not enough for an advanced attacker).

As the article says, he just wanted it shut down. Fixing it can come later, but stopping it from sitting on your network sending out millions of spam e-mails is still useful.

nodata2• 15 years ago

This is a perfect example of what not to do: the guy has ignored the lessons of denyhosts (etc.) and came up with his own regular expression that doesn't do the right thing.

Code re-use please!

ars4• 15 years ago

Or:

apt-get install fail2ban

Works out of the box to prevent brute force attacks against ssh, and can also be configured for other services, like web authentication, POP, IMAP, etc.

xolox• 15 years ago

Just a friendly tip: Try to verify whether Fail2ban actually works and keep monitoring your log files (Logwatch). I once found some dumb ass trying to brute force his way into my Postfix server, unfortunately Fail2ban didn't catch it and I only noticed when Logwatch showed me the authentication failures (several hours and thousands of attempts later).

tete• 15 years ago

Also why would you allow either password based and root login when you are "pretty careful with machine security"?

I had to smirk when I clicked on complex password scheme.

kahawe7• 15 years ago

I found that password scheme pretty OCD and not very secure considering once you have that paper, you have a pretty good chance of very quickly brute-forcing your way in.

scott_s• 15 years ago

If someone breaks into your house or office to get that piece of paper, your problems are bigger than a compromised server.

Kwpolska• 15 years ago

I have a `poweroff' account on my machine with sudo poweroff as a shell with possibility to use the command without a password.

yock• 15 years ago

> But it made me realize that I need to tighten up my SSH buddy plans for the next time.

I'm going to ask a really audacious question: Why?

His systems all worked as expected. Injection attacks were thwarted and login attempts failed. He received (as it turns out, erroneous) notification about the login attempts and knows he needs to do something about it for the future.

Why is that something the sharing of credentials? Why is it that people still allow for remote root login? Why do people still allow user SSH access via password?

There's a better way, and it leaves you a damn sight better prepared for intrusion attempts than receiving SMS messages that, as he so perfectly demonstrated, were not actionable. Limit logons to PKI-only. Live happier. Sleep easier.

gcb• 15 years ago

now that your friend logged as root, you did have a breach.

jarofgreen• 15 years ago

Isn't the real problem that he only has 1 sys-admin for his servers (as far as I can tell)? I mean he can have as many SSH Buddies as he wants but if the alert comes in when he is asleep or drunk or whatever it's pointless.

Maybe the alert goes to multiple buddies who have the tech skills to handle it ... maybe he could be their SSH Buddy in return? In other words, rather than a SSH-Buddy, have a pool of server sysadmin friends for emergencies.

ps. If the original poster is reading this, I would have posted this on your blog but I didn't have other account passwords to hand ... if your moderating all comments anyway, why not allow anonymous comments?

AccordionGuy• 15 years ago

SSH buddy? I don't even have a porn buddy! http://www.youtube.com/watch?v=mgnDHbeVGG4

jerf• 15 years ago

At the risk of talking about the actual topic at hand instead of diving off into minutia about putting SSH on your phone, I'm not sure what this gets you, from a paranoia point of view. Assuming the idea is that you want 100% coverage of your server, this might marginally increase your odds, but not really all that much. Times when you aren't available to administer your server are likely to correlate highly with the times when your buddy isn't available. And what if you'd never gotten the SMS in the first place because you were entirely out of service?

One person simply can't cover a server 100%, adding a "buddy" doesn't help much. Either you need a full-on netops operation, or you need to be able to deal with not having 100%.

davidandgoliath• 15 years ago

Something to sincerely rethink is the idea of shutting the system down in the first place. Call the network provider & have them unplug the ethernet.

Turning the system off could potentially reduce your ability to audit where they originally got in. Anywhom.

SSH keys :)

naner• 15 years ago

Ultimately this turned out to be a false alarm. Although the machine was under attack (on many levels: there was activity hitting the packet filter, trying all sorts of injection at the Apache level and having a go at SSH) the actual alert (based on looking in auth.log) was a false alarm based on a bad regexp.

Any particular reason you didn't want to use Snort or Bro?

tingletech• 15 years ago

The unix on call operator in my data center is my ssh buddy

peterwwillis2• 15 years ago

Alternative Solution, with iptables and perl below.

You send the magic string to any tcp port and it'll instantly kill all SSH logins and disable the root shell. Send the string again and the process is reversed. Caveat: if your syslog contains trigger entries more than a year old this will blow up.

I also recommend you disable root logins and password authentication, but if you insist on enabling them, this may work for you. Modify as necessary.

  iptables -t raw -A PREROUTING -p tcp -m string --algo bm --string "_-()ThisIsAReallyLongAndComplicatedRandomStringToMatchOn()-_" --from 0 --to lengthofthelongstring -j LOG --log-prefix "29CharacterMaxTriggerString "

  #!/usr/bin/perl
  # sentinel - disable root based on a syslog trigger
  # Copyright (C) 2011 Peter Willis <peterwwillis@yahoo.com>
  use strict;
  use POSIX qw(mktime);
  my $LOG = "/var/log/syslog";
  my $LOCKED = 0;
  my $TRIGGER = "29CharacterMaxTriggerString";
  my %M = ( "jan"=>0, "feb"=>1, "mar"=>2, "apr"=>3, "may"=>4, "jun"=>5, "jul"=>6, "aug"=>7, "sep"=>8, "oct"=>9, "nov"=>10, "dec"=>11 );
  
  for ( ;; ) {
      sleep(1);
      open(F, "<$LOG") || die "Error: $!";
      for ( ;; ) {
          sleep(1);
          while ( <F> ) {
              select(undef, undef, undef, 0.001);
              #print STDERR "Reading \"$_\"\n";
              if ( /^(\w+) (\d+) (\d+):(\d+):(\d+) \w+ kernel: $TRIGGER / ) {
                  my $time = time();
                  my $stamp = mktime($5, $4, $3, $2, $M{lc $1}, (localtime($time))[5]);
                  if ( $stamp > $^T ) {
                      #print STDERR "Found Trigger\n";
                      trigger();
                  } else {
                      #print STDERR "Error: found trigger but timestamp $stamp is before script begin time $^T\n";
                  }
              }
          }
      }
      close(F);
  }
  
  sub trigger {
      if ( $LOCKED ) {
          system("/usr/bin/chsh -s /bin/bash root");
          $LOCKED=0;
      } else {
          my @procs = map { @_=split(/\s+/,$_); $_[1] } grep(/^root\s+.*sshd:/, `ps -aux 2>/dev/null`);
          #print STDERR "Killing processes: @procs\n";
          kill(15, @procs);
          kill(9, @procs);
          system("/usr/bin/chsh -s /bin/false root");
          $LOCKED=1;
      }
  }

antics• 15 years ago

I can't believe no one has suggested this yet: this is the PERFECT argument for two-factor authentication. He's even already sending texts to himself.

For every new connector, send some sort of code to your phone, and require it to be inputted back into the computer before any access is given whatsoever. Summarily disregard every single connection and command given without this code re-input.

You're even most of the way there: if you're sending messages to yourself, you might as well implement this behavior with it.

sixtofour• 15 years ago

Many messages here like "why not do X?" or "why in the world are you doing Y?" All good suggestions, but sometimes X or Y may not be good enough, or you may have implemented them incorrectly (like the author's errant regex), or the completely unexpected Z might happen; no one expects Z.

I don't see anything wrong with having a human backup. Heck, what if you die and things need to be wound down?

peterwwillis2• 15 years ago

Dying is outside the scope of application. Please contact nearest heavenly body for ticket submission process.

nhooey• 15 years ago

I've had an "ssh buddy" for years, but for a variety of things such as asking for a "download on how to hotwire a motorcycle".

I have an agreement with many friends to be an operator so they can look something up on the web, or anything else requiring some efficient technical prowess.

We just call up each other and say "operator", and if you're near a computer, you help out with whatever it is.

Who's your SSH buddy?

Related Stories