If you’re not using SSH certificates you’re doing SSH wrong (2019)

https://smallstep.com/blog/use-ssh-certificates/

by noyesno • 4 years ago

459 336 4 years ago

tomrod9• 4 years ago

Rant engaged. As a person who feels responsible for ensuring what I build is secure, the security space feels inscrutably defeating. Is there a dummies guide, MOOC, cert, or other instructional material to better get a handle on all these things?

SSH keys make sense. But certificates? Is this OIDC, SAML, what? Is it unreasonable to request better and deeper "how to do {new security thing}" when PKI is a new acronym to someone? Where can I point my data science managers so they can understand the need and how to implement measures to have security on PII-laden dashboards? As so on.

jpgvm• 4 years ago

I would say PKI (and especially the associated X509 standards) is by far the least understood (or most misunderstood) part of actually building secure stuff.

It would be nice if there was a dummies guide but I'm not really aware of one. Doesn't help that most of "how to PKI" on the web amounts to a bunch of unexplained cryptic openssl CLI incantations.

dfox3• 4 years ago

It is combination of two things. X.509 is arguably overly complex ASN.1/X.500 thing, but that is not the main issue.

Main issue is that most people do not even grasp the concept of a certificate (ie. binding of public key to some additional information that is signed by some other entity).

raxxorrax2• 4 years ago

Also it is a moving space. Browsers don't accept a single certificate for a site anymore, you also have to have that signed by a CA. You can create such a certificate yourself too, but as of today you will need at least two certificates for browsers to fully accept a TLS secured connection. It hasn't been that long since that rule is in place.

So it isn't only the technicalities of asynchronous encryption, there is also specific behavior of applications that use certificates to prove identities.

brandonmenc• 4 years ago

I recommend Security without Obscurity: A Guide to PKI Operations by W. Clay Epstein and Bulletproof TLS and PKI by Ivan Ristić.

I started working in this space a year ago (I'm on a project deploying zero trust networking at a large company) and these books have been invaluable.

https://www.amazon.com/gp/product/036765864X

https://www.feistyduck.com/books/bulletproof-tls-and-pki/

blueflow17• 4 years ago

I can surely recommend reading into SSH certificates using the ssh-keygen manpage. No any extra tools required.

I sign SSH certificates for all my keypairs on my client devices, principal is set to my unix username, expiry is some weeks or months.

The servers have my CA set via TrustedUserCAKeys in sshd_config (see manpages). SSH into root is forbidden per default, i SSH into an account with my principal name and then sudo or doas.

My gain in all of this: I have n clients and m servers. Instead of having to maintain all keys for all clients on all servers, i now only need to maintain the certificate on each client individually. If i loose or forget a client, its certificate runs out and becomes invalidated.

jas-2• 4 years ago

Now you have a centralized single point of failure. While the ease of use is inherently obvious with the implementation, if/when it does fail you will have to fall back to public key/password auth anyways.

XorNot• 4 years ago

Expiries are not protection against compromise.

Compromises happen in seconds - milliseconds, and once they do they will establish persistence. Expiry systems do not and have never been protection against compromise. They're an auxiliary to revocation systems to let you keep revocation lists manageable.

If you don't have revocation lists, or your number of changes is small, you should go ahead and just set your credential expiries to whatever you want - infinity, 100 years, whatever - it won't make the slightest bit of difference.

Particularly in the case when they're protecting sudo user credentials, they're no defense at all.

westurner• 4 years ago

`ssh-keygen` #Certificates: https://man7.org/linux/man-pages/man1/ssh-keygen.1.html#CERT...

"DevSec SSH Baseline" ssh_spec.rb, sshd_spec.rb https://github.com/dev-sec/ssh-baseline/blob/master/controls...

"SLIP-0039: Shamir's Secret-Sharing for Mnemonic Codes" https://github.com/satoshilabs/slips/blob/master/slip-0039.m...

Shamir's secret-sharing provides a better mechanism for backing up secrets by distributing custodianship among a number of trusted parties in a manner that can prevent loss even if one or a few of those parties become compromised.
However, the lack of SSS standardization to date presents a risk of being unable to perform secret recovery in the future should the tooling change. Therefore, we propose standardizing SSS so that SLIP-0039 compatible implementations will be interoperable.

michael_j_ward2• 4 years ago

I have the same feeling, and it motivated me to recently purchase this "Bulletproof TLS and PKI" [0]

I haven't read it yet, so I'm posting i hope of someone else giving a quick review.

https://www.feistyduck.com/books/bulletproof-tls-and-pki/

enriquto3• 4 years ago

SSH keys make sense. But certificates?

I am equally mystified... Never understood how involving a possibly malicious third party can make communication more trustworthy.

But then again, I was also sure when I first heard about it that public key cryptography was obviously impossible. You just could not have secret communication when everything is on the open! Is there any simple explanation that we ignorant people can read about certificates to get an "aha! insight" moment? For the case of public key cryptography, the moment where everything snapped together was when I read the mathematical description of the Diffie-Helman key exchange [0].

I'm not interested in how to do certificates with ssh, but on what problem do certificates solve, exactly.

[0] https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exc...

dsr_3• 4 years ago

A certificate is just a formatted list of attributes that has been signed by a particular private key. Username, UID, GID, membership in this, privilege for that, good-after date and good-until date, for instance.

Everybody knows the public key associated with that private key, so you can verify that the private key did sign this list of attributes.

An ssh keypair is an actual public/private keypair, but a certificate is just signed and encoded (but not encrypted) formatted data.

If an ssh daemon has knowledge of a public key used to sign a cert, and has been instructed to trust that cert, and all the dates are good, then the ssh daemon can accept that cert as proof of identity and allow a login.

slivanes• 4 years ago

I really like: https://www.abc.net.au/news/2017-09-15/how-encryption-works-...

adrian_b• 4 years ago

Using certificates does not provide anything useful for a small private network, with a single administrator, or where all the users are trusted.

On the other hand, they are useful for large organizations, with needs for differentiated access rights and management rights.

The centralized control over the certification authority allows the delegation of restricted rights to other levels of network administration.

yosamino• 4 years ago

One example also mentioned in the article:

If you connect to a ssh-server for the first time, ssh will give you a warning and let you know that you have to verify the fingerprint of the host key.

This becomes annoying when you connect to many different servers and I would not trust everyone (including me) to do this check correctly every single time.

SSH certificates solve this by having the ssh-host-key be signed in a way that your ssh-client can verify and you only have to add a key-signing-key to you known_hosts once.

Now you have to sign the ssh-host-key but you only have to do it once per server as opposed to having each user having to do it locally on every first connect.

tptacek13• 4 years ago

People talk too much about "certificates" as a good thing, but they're just a means to an end. The two major goals you're trying to solve with SSH:

(1) You want all of your authentication to route through a single point of control where you can enforce group-based access control, MFA, onboarding/offboarding, and audit logging.

(2) You want the actual secrets that allow you access to an SSH server not to live for a long time on anyone's laptop, because it is effectively impossible to ensure that, on a sufficiently large engineering team, nobody's laptop will ever get compromised; there's just too many of them, and developers do weird shit so the machines can't be ruthlessly locked down. You want people to have SSH login secrets for exactly as long as they need them for a specific server, and no longer.

Certificates solve the problem of having dynamic access control to SSH servers without having some weird system that is constantly replacing authorized_keys on all your servers; instead, there's a single root of trust on all the SSH servers (the CA public key) and a single place that mints valid certificates that enforces all the stuff I mentioned in (1) above.

It's worth knowing here that SSH certificates are nothing like X.509 certs; they're far simpler, and you could bang out an implementation of them yourself in a couple hours if you wanted.

capn_duck• 4 years ago

Practical Cryptography by Bruce Schneier and Niels Ferguson is decent in that it gives a good lay of the land without diving too deep in to the mathematical rigor. The first half explains at a high level the concepts of encryption, key exchange, asymmetric encryption, digital signatures, and lays out the problem statement that PKI solves.

It's nice in that it will list out a bunch of available encryption algorithms or hash algorithms, but at the end of the chapter say "Just use this one, it's considered safe right now." i.e. AES256 and SHA256.

Unfortunately, it mostly avoids the practical steps of web security, like its not going to print out the command to type in to your shell to generate an SSL signing certificate. So I wouldn't recommend it if you're looking for an immediately practical book to help you secure your web server. But it orients you to the landscape so you have a general idea of what you're trying to achieve, and can google yourself the rest of the way there.

tomrod9• 4 years ago

I feel this is the exact right thing for me right now -- people trusted in industry. I can follow tutorials and documentation. The part where a concept is explained is often missing and can be guessed at (albeit often wrongly).

I'll look into this and perhaps supplement with some good tutorials for my developers and data scientists. I appreciate your input!

vngzs• 4 years ago

If they're willing to read a book on security design, I would recommend Security Engineering, 3rd Edition [0]. It includes a broad survey of what matters in the security space (rather than just cryptography), and generally in sufficient depth to understand how we may build secure platforms in the face of adversity.

Also, many of the chapters are available to read for free - read author's text under the cover photo.

[0] https://www.cl.cam.ac.uk/~rja14/book.html

tptacek13• 4 years ago

I don't think Practical Cryptography is going to give you much of an intuition about why this article is advocating for certificates.

tener5• 4 years ago

I feel for you. Security is a complex, evolving topic, with a dizzying array of concepts.

At work, we develop Teleport (https://goteleport.com/) to provide a secure access solution that is also easy to use and hard to get wrong. (Note: you cannot truly have "hard to use" and "secure" access, because people will always develop "backdoors" that are easier to use but not secure.)

If you are interested in some accessible writing about security check out: https://goteleport.com/blog/

On SAML: https://goteleport.com/blog/how-saml-authentication-works/

On OIDC: https://goteleport.com/blog/how-oidc-authentication-works/

I can recommend the YouTube channel too: https://www.youtube.com/channel/UCmtTJaeEKYxCjfNGiijOyJw

LambdaComplex2• 4 years ago

Teleport seems like a genuinely cool product.

With that said, the company really needs to improve its interview process--my experience was downright terrible, and Glassdoor shows that other people had a similar experience

robomc2• 4 years ago

I'd love to use this product in my organisation, but I don't want to self host, and it's really unclear what it would cost me.

Seeing an "enterprise, call for a quote" type tier makes me assume it's going to be too expensive for agency securing 10-20 servers.

diarized• 4 years ago

Scalable and secure access with SSH @FB https://news.ycombinator.com/item?id=12482212

hitpointdrew• 4 years ago

Check out teleport. It abstracts away the certificate bit and manages it for you. You run 'tsh login' once and you get a cert good for 12 hours (then you can get access to all the teleport resources you are allowed to, weather that is ssh server access, db access, kubernets access, etc.) I am evaluating the product now and am quite impressed.

https://goteleport.com/

throwaway9843933• 4 years ago

It's not unreasonable, but we need some kind of universal knowledge base for tech stuff. Security is just one of many inscrutable topics in tech where you need weeks of research to understand the best practices.

chefandy• 4 years ago

My needs may be entirely different than yours and I don't want to downplay the importance of security, but...

Security writing in "engagement" obsessed media yields lots of people screaming "FIRE!" whenever seeing something even theoretically flammable, and bandwagoneers— already imagining their hair is on fire— lambasting everyone not immediately evacuating for being careless about fire safety. It reminds me of politicians being 'tough on crime'— they reflexively jump at opportunities to tighten the screws regardless of its necessity or efficacy. It's an emotional response involving self-image, peer pressure, and fashion rather than rational cost benefit analysis.

Perfect is the enemy of good. Attacking every theoretical threat like an international bank's network admin yields no practical benefit for most. Not nobody but most. If this TLA is new to me, there will be another new one that people will lambast me for not knowing in a couple of years, max.

For me, this problem was a better fit for the Wizard of Oz than a security education resource— what I really needed was the right frame of mind rather than learning the implementation details of every incremental certificate authority update.

I evaluate my attack surfaces and reduce them if I can, evaluate the real importance of keeping what I'm protecting secret, implement standard precautions and architecture to mitigate those risks, pay attention to the systems, pay attention to new vulnerabilities, and re-evaluate upon changes. The process is technology-agnostic and only requires you deep dive into the stuff you need to know without feeling like you need a new certification ever 6 months to run your company's CalDAV server.

scottLobster• 4 years ago

Relevant article: How I learned to stop worrying (mostly) and love my threat model

https://arstechnica.com/information-technology/2017/07/how-i...

TedDoesntTalk• 4 years ago

evaluate the real importance of keeping what I'm protecting secret

Excellent. This is often ignored by the security obsessed, those people yelling FIRE! as you say.

Securing access to my cloud-hosted cat photos does not demand the same energy as securing ICBM launch codes.

kbenson• 4 years ago

When you start dealing with hundreds of servers or more (perhaps it starts earlier at the high tens), you start looking at all things as trade-offs, and doing so yields interesting insights that aren't necessarily obvious when you're working at smaller levels.

What is the cost (in time and effort and manpower and complexity) to implement? What is the cost to maintain? What is the cost to manage, when you are adding and removing people often? What are the failure scenarios, when any one server that needs to manage things starts to become a liability for disaster recovery and redundancy purposes?

Sometimes the destination is clearly better than your current place, but the road to get there has a cost all its own that makes traveling it the non-optimal choice.

It's very easy for 1-3 admins to decide to implement something over 10-30 servers and keep themselves up to date and with the right access and knowledge to manage and maintain it. It's quite another thing when you're talking about hundreds of servers and you've implemented clear delineations about access and you have 10-20 admins ranging from junior to expert with associated levels of access to servers and tools, and the fleet of servers has evolved over years (or multiple decades, in some cases). Applying changes over that type of system can be complex and error prone and when it affects your ability to actually access and maintain the systems in question, it can be very hard to reason about the problems until you start encountering them. Change comes with risk, and risk assessment of technology becomes a large part of the planning requirements.

pphysch3• 4 years ago

I spent an afternoon implementing a SSH cert service in Go using the standard "x/crypto/ssh" package, with little to no prior knowledge of SSH internals.

SaaS like Smallstep and Teleport are trying to middleman and monetize what is actually a simple process that more developers should be comfortable implementing themselves.

This isn't "rolling your own crypto", this is standard SSH key stuff plus a bit more nuance and LOC to make things more secure.

When you pay for those services, you are essentially paying for a wrapped SSH command + dead-simple web app + someone to be your CA (read: store the resulting files of `ssh-keygen`, hopefully securely). And all the potential headaches of relying on yet another SaaS.

Plenty of developers are comfortable writing a script, a simple web app, and securely storing a file on the webserver. This is all that is required to build SSH cert support into your internal apps/tools, plus an afternoon understanding how CAs work (in short, CA private key can sign any SSH public key, then that SSH public key can be validated by anyone holding the CA public key, no TOFU required).

Aeolun• 4 years ago

This blog post seemed eminently understandable to me, at least as someone aware of public key, but not certificate based authentication.

ozim• 4 years ago

What is the scale you are operating on?

If you are having 10-50 servers and 5-10 people working on those - SSH keys are definitely good enough, it might be a bit of hassle to manage keys but quite OK.

If you go into large corporation area with more than 100 servers and more than 50 tech people that need to login to those servers you probably would already found out that there are other options and you probably have to run your internal CA (certificate authority).

If your org grows you would probably have CTO and other technical people who will have experience knowledge to implement things differently.

el_duderino• 4 years ago

Previous discussion from 2019: https://news.ycombinator.com/item?id=20955465

andix• 4 years ago

As long as it is not as easy as let’s encrypt, it won’t take off.

inetknght4• 4 years ago

I toyed with SSH certificates a few years ago. They seemed cool and secure but, ultimately, the x509 stuff was quite (!) arcane. And trying to get the IT team on board would have been a nightmare.

I quite agree: as long as it's not as easy as Let's Encrypt then it won't take off.

blueflow17• 4 years ago

man ssh-keygen lists me this:

Note that OpenSSH certificates are a different, and much simpler, format to the X.509 certificates used in ssl(8).

Where do you work with x.509 certificates when doing an SSH ca? I use SSH certificates productively and only ever used ssh-keygen commands.

sgarg1• 4 years ago

SSH with x.509 requires patching the SSH client and daemon. It is an unofficial patch to add the functionality.

SSH certificates themselves are similar to x.509 but a lighter version. They are supported out of the box.

tfigment3• 4 years ago

Server setup isnt hard. No need for authorized_keys files anymore as you just trust signing server instead. What is hard is user experience. Sshagent doesnt support the certs. Putty (most popular client on windows?) doesnt support certs. I want to use them but need better client support. Vault makes almost easy to create and manage certs but have to use openssh client only with extra argument every time.

seanw444• 4 years ago

I didn't know people thought Let's Encrypt was easy. My first foray into setting up certificates for a web server let me realize how convoluted the system itself is, not just specifically Let's Encrypt. I guess it's easier compared to the alternatives. But holy cow I hate the process.

crispyambulance• 4 years ago

yeah, it's really confusing.

Many folks are comfortable with jumping through the hoops (following instructions) WITHOUT understanding what's really happening.

Those of us who aren't comfortable with "just works" need to plow through a enormous amount of persnickety jargon and easily forgettable material to reach an understanding that gives us confidence.

pxeger1• 4 years ago

The USP of Step is to make setting up a custom CA as easy as it is with Let's Encrypt

cj• 4 years ago

I'm tempted to write a competing article: "If you're using SSH in 2022, you're doing it wrong"

There's no need to open port 22 if tools like AWS Session Manager (and GCP's equivalent) are available to you.

core-utility• 4 years ago

Don't forget about the no-ops "If you're opening a shell on any server, you're doing it wrong (2022)" method.

dtgriscom• 4 years ago

What's next? "If you're using a computer, you're doing it wrong"?

GekkePrutser7• 4 years ago

But even if you do it through that, SSH is a much nicer protocol than typing on a remote console. You get file transfer, X, agent and port forwarding, terminal window scaling and much lower bandwidth.

Also, not all servers are on cloud platforms.

twic• 4 years ago

Is it possible to run Zmodem over this AWS pseudo-console the way you can over SSH?

Not all servers are on cloud platforms, but there are somewhat comparable ways to shunt a serial or out-of-band management console over the network.

onpensionsterm• 4 years ago

There are actually people working on things other than CRUD web apps!

throwaway9843933• 4 years ago

I think you're being downvoted by the same nerds obsessed with self-hosting. "But how can I self-host without ssh?!"

martopix• 4 years ago

And can I access my university's server with that? My office computer from home?

client4• 4 years ago

Keybase had an elegant solution I use https://keybase.io/blog/keybase-ssh-ca

bloopernova• 4 years ago

I used to like Keybase so much, it felt like a Next Big Thing. I just wish they had stuck to identity management and validation, providing SSO etc. Instead they tried to be a chat client, git host, and crypto wallet too. They spread themselves too thinly trying to compete with dozens of rivals for each function they added.

I wish they'd have been a standard that Microsoft, Apple, Google etc provided implementations of.

sjaak• 4 years ago

I'm not doing anything wrong.

My sshd is behind spiped + logging in requires me to physically tap my yubikey.

GekkePrutser7• 4 years ago

SSH certificates make sense. But can you use hardware backed ones like the OpenPGP applet on yubikey with this?

I currently use this method to store my SSH keys safely. But I don't know how this would work with certificates. If I have to store them in the computer instead of a hardware token it's a huge step back in security.

By the way what do home users use to set up a PKI? Scripting everything with OpenSSL is but very nice. It would be cool if there were an open source PKI platform ideally even with IDP built in. With a nice web interface and easy to install with docker. Never found one though.

blueflow17• 4 years ago

Yes, ssh-keygen allows you to use private keys from any PKCS#11 backend using the -D option. This includes smartcards and tokens.

Including for the CA key.

GekkePrutser7• 4 years ago

Ah Yes I used this method before with PIV cards and OpenSC/OpenCT.

The toolchain is a real PITA though. Unstable, difficult to provision. Proprietary tools for card management. Not all platforms support PKCS modules. As far as I remember openssh on macOS was compiled with support for it. So I had to replace it with one from brew which is much harder these days. Mind you we're talking 5-6 years ago.

OpenPGP is really user-friendly. Nice config menu with gpg --card-edit . SSH agent functionality built into the gpg agent.

I kinda want to retain this level of comfort to be honest.

Cpoll• 4 years ago

I wish revocation was covered as well. The article mentions this issue in SSH:

Keys are trusted permanently, so mistakes are fail-open.

But you can make the same mistake using certificates, by issuing a certificate with a large expiration date. AFAIR you _can_ revoke valid certs, but that involves making changes to every box running ssh, much like when revoking a public key.

blueflow17• 4 years ago

You can distribute a key revocation list along with your regular configuration management. The key revocation list can be generated and maintained via ssh-keygen.

I used to have a central location with an revocation list. All servers had a cronjob fetching it.

camtarn2• 4 years ago

And that's pretty much the exact same infrastructure you'd need for distributing authorized_keys.

Instant revocation seems to be a giant hole in this article's argument which isn't addressed at all. If you fire somebody partway through the work day, end-of-day cert expiry is not good enough to prevent compromises.

emrvb• 4 years ago

I would love this to work, as it indeed fixes several issues addressed in the article.

However, I don't think openssh-server supports OCSP natively, so while you might be doing SSH right, you're doing certificates wrong.

egberts17• 4 years ago

OCSP natively? No.

Does the current OpenSSL have its own KRL?

temp89642• 4 years ago

There are cases where if you leave SSH enabled then you are doing it wrong. If you only need SSH to do a few things once a while, you should just turn it off, and only turn it on when needed. In these situations, I feel it’s OK to just use password to login.

Enginerrrd• 4 years ago

Honestly, I agree. My ssh passwords aren't going to be brute-forced anytime this century. It's also pretty easy to put a fake sshd on port 22 and set the real one to some other port (preferably one low enough to still require root privileges though).

I don't have encrypted drives on all my devices. I don't want to have to worry about what could happen if one of those gets lost/stolen. I'd rather not leave keys or certificates lying around.

Also, things sometimes go wrong and I need to get access to a server from a device I've never been on. It's nice to be able to do that. Passwords do that.

To be fair, I usually have a single VPS which I keep as locked down as possible that has VPN access to the server I really need. The VPS doesn't even need to be running most of the time. So I can spin it up to get access to the VPN, then ssh into the server with a password. If the VPS gets compromised, the VPN alone won't give an attacker immediate access to the server like it would if I left keys / certificates on there. I have to trust the VPS, and if it gets compromised without me noticing, and I then log in to my server, yeah, I'm SOL, but certificates don't solve that problem.

remram4• 4 years ago

How do I turn it on if I can't SSH into the machine? In most cases, SSH is what you use to do management tasks.

temp89642• 4 years ago

Many hosting services have a control panel to do this on VPS. Some hardwares like Synology also have web interface to do this.

withinboredom3• 4 years ago

brew install step

The audacity of mac users who think everyone uses a mac.

dljsjr2• 4 years ago

Brew has been available on Linux for a long time and works very very well.

withinboredom3• 4 years ago

There are other OSes than Linux and MacOS, in fact, one of those other OSes is far more popular than both of those put together: Windows. It has built-in openSSH support since 2018.

loudmax• 4 years ago

Sure, but if you're not on a Mac, brew shouldn't be the first tool you reach for. You can use brew if you've transitioning from Mac and you need to time to get acclimated, or if the software hasn't been integrated into an the local OS's package management system yet.

Having said that, `brew` appears once in passing in the entire article. This article is almost completely about ssh without regard to platform. There's nothing "audacious" about suggesting users use brew, so the anti Apple sniping here is entirely unwarranted.

tomrod9• 4 years ago

Right? Most folks just need dnf.

wintermutestwin• 4 years ago

The same audacity of how tos with "apt get" while lacking any distro specific language.

Linux desperately needs a standard for writing how tos. Even better would be definitive how tos for every common thing a user might need to do being posted and maintained on a distro owned site.

jdoss• 4 years ago

I work at smallstep and we support a wide range of Linux distros already. https://smallstep.com/docs/step-cli/installation

One of the things on my punch list is to get step CLI into upstream Fedora, Debian, and Ubuntu to make installation a bit more easy.

ARandomerDude• 4 years ago

Bro you almost made me spit out my latte.

xrd• 4 years ago

This article was very illuminating. And, I wish it was written like this:

  To really secure your SSH server, do these three things:
  1. Setup a trusted authority
  2. Use ssh certificates
  3. With ssh certs, you can easily add MFA for logins expire until you reauth.
  Now that we've established this, here are the gory details.
  Lorem ipsum, lorem ipsum, lorem ipsum.

It's a great narrative, but I wish I knew the big payout in advance.

michael_j_ward2• 4 years ago

Is that what I get out of the box using tailscale for ssh?

https://tailscale.com/kb/1009/protect-ssh-servers/

tener5• 4 years ago

Tailscale (and other similar solutions) works on the network level. This is not a bad idea in itself, but SSH certs operate on the application level.

The fact you can ping the server shouldn't mean you are allowed to actually access it.

tptacek13• 4 years ago

Two examples of things Tailscale doesn't give you for this usage model that SSH CAs can:

* Transcript-level audit trails for what people are actually doing on SSH sessions.

* Differential access to different groups of users to the same machines.

Tailscale and SSH CAs work together nicely: require membership in the right Tailscale group to talk to SSH at all, thus tying access to SSH to your (e.g.) Google login and MFA requirement, and use something like Teleport for the actual SSH login, to get the audit log, group access, and an additional authentication factor.

AtlasBarfed4• 4 years ago

MFA for ssh means you can't automate cluster-level ops. Unless, well, you automate the MFA, which defeats the MFA.

This drives me a bit nuts about security people in the age of cloudscale. They assume you don't mind MFA'ing every hour and are manually doing logins and accesses for everything. Yeah, uh, I need to script orchestration on several hundred machines at once, and orchestrate/access on the scale of hours or even days for some things like "Big Data" backups or restores.

If certificates are anything like SSL certs and the horrorshow cli tools / options / management involved in those, no thanks. I'd rather have an automated sshkey switchover, or for stateless just routinely cycle the infrastructure with new keys.

It's been a long time tenet of security that you want an open algorithm that gets broadly and publicly challenged so you know its secure. Well, in the age of the state actors, this might not be the whole truth.

I think layering some klugy not-invented-here obfuscation atop the more battletested methods is a useful and important deterrent/delay. Sure someone will figure it out, but they have to TRY HARD. A lot of the institutional attacks seem to be based on human attacks on standardized systems, which HAVE to allow human access and vectors.

So in AWS land, secrets manager is secure unless you get the permissions. Then you have everything. But if each of those secrets has some whacko obfuscation for the various apps, then that is a big slowdown to the human attack vectors.

And the state actors? Well, even they have budgets. They'll probably move on to easier targets. If a state actor is motivated at targeting your company in particular, well, given that they'll have malware in the firmware of your hard drives and motherboards and the like, you're probably helpless.

Finally, what really bothers me about most security is that it leaves one of the most important "canaries in the coal mines" aspects of security: honeypots. Sure do the diligence on securing the access, but how about some turnkey approaches for setting up honeypots to detect when people are poking around? Honeypots are perfect for that, because the devs only care about the stuff they are working on. The intruders are doing the scanning.

fishpen02• 4 years ago

Cluster-level ops really shouldn't be done by direct SSH. Tools like ansible, salt, chef, puppet, etc... all can be run in some form of daemon mode with a central management server for a reason. You should be authenticating against the management service and running your configuration or automation scripts from there, not as a massive pool of CSSH or whatever directly from your laptop.

Over the last 6 years I've worked for three different companies that all universally disabled ssh and we never had troubles running management scripts or tooling.

artificialLimbs• 4 years ago

You mean like Thinkst Canary?

Or OpenCanary if you can't afford $arm&leg?

zepearl• 4 years ago

I agree, something like that. From what I've experienced it's (for me) a common problem in that realm:

a lot of blah-blah that makes my eyes glaze/lose focus before getting to the core/overview.

~15 years ago, when the company I'm working for first implemented PKI, I needed something like 100 hours of "help" from local security engineers and the external software-vendor's programmers to understand how that worked and what the SW was supposed to do. After that, explaining to colleagues at least on a high level how that works became a matter of minutes.

To be fair towards myself, even the local scrty eng gurus (relaxed, as they themselves didn't have to "deliver" anything) and the external vendor's programming gurus (hardcore, as nothing would be paid if the SW could not be implmemented) were often absolutely not understanding each other, so I ended up becoming their unofficial mediator/translator:

if I felt like an obvious question was not dared to be asked by one of the parties then I sacrificed my self-esteem to dare asking it, if "even I" did manage to understand some concept then both parties were supposed to get it as well and if not then at least I could act as gateway to explain/expand offline :P

It was an interesting time - not nice nor very bad (a little bit bad, as we had of course an implementation deadline), but at least I learned a lot, as well in the area of social skills :)

FriedrichN• 4 years ago

Stuff like this is really cool. But the problem is that certain clients (as in clients that pay your bills) can't even get public keys to work and we end up allowing password logins. So I don't know how I could make this work in the real world.

That's where security ends, when 'it just has to work' because 'they're the ones paying/in charge'.

brightball• 4 years ago

Isn’t this the process that Teleport streamlines?

pzduniak• 4 years ago

Teleport completely streamlined all sorts of access management in my projects. Every new feature they release (Kubernetes, Database, App access) work as advertised and the Helm chart + Terraform plugin (which for some reason is not published in the marketplace??) are great for automating the whole deployment + configuration. Great piece of software.

1970-01-01• 4 years ago

Meh, you're not wrong for skipping SSH certs. They're mature security, but not mature enough for everyone. And like everything else, they break, predictably and unpredictably. If you're not ready for someone to emergency access their way into prod to fix a broken SSH issue on Christmas morning, you're not ready for SSH certs. Maintenance will get you, one way or another..

sergiosgc• 4 years ago

The SSO flow is way too complex for something that must work on an emergency.

teknopaul5• 4 years ago

Is there something wrong with installing client public keys in the server and preventing password based logins?

Seems more secure than handing over the security of your servers to a CA?

Managing the authenticated_keys file is trivial and I like simple file based solutions.

I find simpler is often more secure because its easier to understand.

I have had plenty of ssh servers on the Internet even, (on non-22 ports to prevent logs filling) and not had a problem yet AFAIK.

I have also been on call when certs expired gawd knows how many times.

aidenn03• 4 years ago

I'm not sure I buy it, but, TFA tries to explain:

Situation: A person's machine dies and takes the SSH private key with it

No Certs: They now need to get IT to distribute the public keys to a set of hosts, where the full list is possibly not known by any one person, so for the next week it will be "oh shit, I forgot to ask IT to put the pub key on server-12, so I'll open a ticket and not get any work done for the next few hours"

With Certs: They get IT to sign the new key.

LinAGKar• 4 years ago

Too bad you can't do that with dropbear

ayushnix9• 4 years ago

Which is why I installed openssh-server on my OpenWrt hosts. However, due to a bug [0] in openssh-server package in the latest 21.02.2 release of OpenWrt, OpenSSH doesn't allow you to login in failsafe mode on an OpenWrt host. Because my OpenWrt host lacked a recovery mode, it was essentially soft bricked.

I was able to recover it using the serial port but even after all this, the comfort of using SSH certificates on all of my nodes was enough to keep making me use it instead of Dropbear.

[0] https://github.com/openwrt/packages/issues/17833

cestith• 4 years ago

The article does a pretty good job explaining advantages of certificates. It overlooks the existence of solutions to some of the individual problems it mentions with keys, though. Tools like sssd exist, with which a way for the end user to keep their public key updated in a directory takes care of key distribution and at the same time limits who can use sudo, all without changing any files per-user on the server.

IgorPartola2• 4 years ago

If this is true then why does ssh not get set up this way by default?

Honestly what I would prefer is a better (and faster) version of monkeysphere [1]. That was honestly the most natural-feeling solution to SSH security.

(1) https://www.systutorials.com/docs/linux/man/1-monkeysphere/

dsr_3• 4 years ago

Because it's not suitable for everyone in all circumstances.

IgorPartola2• 4 years ago

So if I am not using certificates them I am not doing it wrong? :)

rantanplan• 4 years ago

Yes, SSH certificates are the way to go and pretty easy to set up. But what these articles fail to address is the user management aspect.

For the SSH certificate to be accepted, the unix user must first be present on the system. As far as I can understand, FreeIPA(or similar LDAP systems) cannot be used in conjunction with SSH certs. Whereas SSH keys are supported by these systems.

Can anyone provide any insight/experience with this?

ehnto• 4 years ago

I imagine it's a matter of automating the certificate insertion on the target servers when it's updated on the user's account in the LDAP server. In other words, it depends entirely on your systems and how far your administration is willing to go to automate it.

dsr_3• 4 years ago

In most circumstances, you want these two things (user is authorized on the system, user can be identified and authenticated) to be different. Having a process that creates the user on system in order to authorize them to login is pretty similar to all your other configuration management tasks.

blueflow17• 4 years ago

Its not the username that needs to match, its the principal. You can allow any principal for the root user, for example.

You can define principals when allowing a CA via authorized_keys, or you can configure allowed principals globally using sshd_config directives like AuthorizedPrincipals* .

mbreese• 4 years ago

Many years ago, I did all of this with an LDAP system. Public keys were generated by the user and entered into LDAP (or you could auto-generate keys, etc). Users were authenticated with their ssh key (stored in ldap, password based access was restricted). Authorization for access to each host was also in LDAP, as was sudoer status (as a group setting).

It was actually quite an elegant setup. You would still need to setup a CA for generating local certificates for TLS connections to LDAPS, but the auth was handled all in the LDAP server.

I think the main downside would be trying to have the authentication overhead on a single server (the ldap server) when you are dealing with many hosts. Over a handful of systems, it’s great. But it doesn’t scale when you’re taking thousands of hosts (or cloud vms that spin up/down).

jimmaswell• 4 years ago

I still don't care about any of this nonsense for my personal stuff when I can avoid it. Passwords all day for me. 0 security incidents in my lifetime.

Sucks that Github and some other things force SSH keys which are just passwords except always saved to your disk so that anyone who steals your laptop gets access.

It adds insult to injury when you try to capitulate to this malarkey, generate a key in PuTTy's key generator, then Github whines that the default setting isn't overkill enough and you have to make a whole NEW key with some other setting. I miss the good old days.

krnlpnc5• 4 years ago

Sucks that Github and some other things force SSH keys which are just passwords except always saved to your disk so that anyone who steals your laptop gets access.

This is the reason to encrypt ssh private keys with a passphrase. If the key is leaked it's still protected by the password.

It's a built-in feature of ssh. For an existing key downloaded from a cloud provider, use ssh-keygen -p to add/change the passphrase.

gizzlon• 4 years ago

How hard is it to crack these passwords? Since it's local (unlike the github password) I guess you can run brute force attacks etc at full speed ?

tener5• 4 years ago

I miss the good old days before passwords were a thing and everyone just trusted others to behave ;-)

You can secure access to Github (and other places) with hardware keys, e.g. from https://www.yubico.com/.

toolz• 4 years ago

not if you encrypt your ssh keys, which is what everyone I know does - then your potentially weak password requires physical access to exploit while things accessible over the internet can't really be brute forced.

further, this is even more convenient when paired with an ssh-agent that will securely hold your private key in memory and not allow anyone to export that key...you could dump the memory but that would require root access, which again should be password protected

teknopaul5• 4 years ago

My servers expect an environment variable to be sent too, (new sshd and ssh can do this). This gives basic 2fa without typing anything.

teknopaul5• 4 years ago

Ever don't the math on the amount of time spent entering passwords? I have. Stopped using passwords for personal stuff.

I multiplied by cost per min for downtime in professional support and the cost of typing passwords was more than my yearly wage.

raxxorrax2• 4 years ago

Passwords are safe if you can memorize them. It is not too hard in my opinion. I also think they should always be an option for any kind of auth. Maybe I want to authenticate against a system but I don't want others to know my ID. For that use case a password is the better solution.

kenmacd2• 4 years ago

But you add a password to the key, so it's the same.

And not everyone saves it to disk. My ssh key is my gpg key. It's stored on a yubikey and can't ever leave it. If I do a `git pull` then my yubikey flashes and I have to tap it to allow that connection to happen. Steal my yubikey, well you can't unlock it. Hack my laptop and you can't tap the key.

jalgos_eminator• 4 years ago

The github change also messed up my workflow, which involves pulling/cloning my company's git repo from lots of machines, many of them being short lived or disposable. Now I have to save the password forced on me in a file because I'm unable to memorize it easily and that made our setup less secure. Thanks github...

outworlder2• 4 years ago

I still don't care about any of this nonsense for my personal stuff when I can avoid it. Passwords all day for me. 0 security incidents in my lifetime.

Even for personal stuff, why would you want to use passwords? Keys are more secure AND more convenient. Sure you don't need certificates but I don't understand how keys are more 'nonsensical' than passwords.

Keys have more flexibility, you can use SSH Agent, you can do SSH agent forwarding, etc.

except always saved to your disk so that anyone who steals your laptop gets access.

This is wrong. First of all, your laptop should have disk encryption. Always. I don't care what your threat model is, encrypt the disk. Second, SSH keys can (and SHOULD) have a passphrase.

sneak• 4 years ago

This contains several misconceptions.

Keys are not just "passwords saved to disk". My private keys exist in hardware, on Yubikeys. They aren't on disk. The hardware requires authentication to access.

You're typing your passwords zillions of times. I log in to my system once, authenticate to my HSM once, and I can access many hosts via scripts and automated tools. This is impossible to do securely with password auth.

You're also training yourself to manually input the entirety of your authentication credential multiple times per day (or hour). This is bad practice, as anyone stealing it then has the keys (ha) to your kingdom (and they have way more opportunities to steal it!). Even if you just replace password auth with a password-protected key on disk, and don't use a password-caching agent that holds the decrypted key in ram (as would be typical), so that you're still typing your password each and every authentication, you've raised the bar substantially because someone would need to steal your encrypted key from disk in addition to obtaining your password.

Then there's the issue of cycling credentials, and the mental loads involved. I can cycle my keys without changing my workflow or having to type anything differently.

Passwords are not good authentication tools. Use actual cryptography.

krnlpnc5• 4 years ago

It depends on the scale. If a company has a handfull of hosts I'd argue that deploying the full AAA and PKI systems to back cert auth is doing it wrong.

Traditional ssh-key auth is simple and reliable, it's not until you have a large, complex and diverse user base that you need something more. That's why the huge fang sites use it. Every org doesn't need to mimic fang.

fvold• 4 years ago

Exactly!

I don't understand this obsession with enterprise-level security on home networks and hobby projects. If you think it's fun and educational to set up, then you're doing it for fun and education, not security. If you're doing it for security, you're basically setting up anti aircraft guns to do what a drone jammer could do with way less resources spent.

hedora2• 4 years ago

Anti-aircraft guns can't actually take out most drones.

Good analogy.

ayushnix9• 4 years ago

I'm using SSH certificates to manage a few nodes in my homelab and it's a pleasure to not have to deal with managing the known_hosts file on my clients and authorized_keys file on my servers. There's only 1 line in my known_hosts for my nodes and authorized_keys doesn't even exist on any of my servers. If I add a new node to my homelab, I don't have to make any changes in known_hosts or authorized_keys in the existing nodes and it's easy to bootstrap the same known_hosts and sshd_config that I use everywhere in the new node.

SSH keys would make managing these few nodes a lot more complex that it is.

nostoc• 4 years ago

True, and while the title is somewhate clickbaity, I think your point was pretty clear in the article.

aseipp3• 4 years ago

You really don't need to be anywhere close to mega-scale to benefit from SSH certificates and integrated authentication flows, though. Even at the scale of "only" 10 people with SSH access, the whole system can be massively simplified and made more secure by integrating centralized logins, and SSH certificates are rather perfect for this.

I implemented my own SSH certificate authority myself more or less, and while it's overkill for my own homelab-level stuff, I absolutely would never use anything else once I have more than like, 5 people logging into some set of machines. The benefits of centralized SSH access control that you can freely integrate (and pretty easily too, thanks to OpenSSH!) with your existing identity provider is really nice.

tacostakohashi2• 4 years ago

This is definitely true... and a lot of people are doing it wrong.

This is one of those things that drives me nuts as an experienced/old developer, seeing people type passwords for ssh/git/whatever several times per day. Sometimes there are tasks that require copying / checking some file on N servers, and these people seem to think that cannot be done in a shell script because the password needs to be entered interactively.

Then there's ssh port forwarding, X11 forwarding, etc... but its amazing how many people use ssh for years without so much as glancing at the man page.

Symbiote• 4 years ago

It is amazing how many people use any CLI tools for years without reading the man page.

The man page for the shell (man bash; man zsh) is a good place to start.

tacostakohashi2• 4 years ago

Yes that too... "the kids of today" seem to regard basic shell tools and scripting as a dark art rather than an everyday part of using a computer and doing development productively. It's kind of sad.

toast02• 4 years ago

these people seem to think that cannot be done in a shell script because the password needs to be entered interactively.

That's what autopw[1] is for. ;p

[1] https://github.com/jschauma/sshscan/blob/master/src/autopw

mnd999• 4 years ago

Aside from being a pain to set up, what’s wrong with using GSSAPI / Kerberos? (within an org)

en4bz• 4 years ago

Kids theses days hate using technology that's older than themselves? Kerberos is amazing and it make me sad to see all these people reinventing stuff that was solved 30 years ago. The number of companies selling SSO and doing it poorly (see this weeks okta hack) is unfortunate.

risson2• 4 years ago

21yo here, I think Kerberos is bloody awesome, but then I was introduced to it by a somewhat old timer, who showed me its benefits when properly integrated in the company.

zengargoyle• 4 years ago

Yeah, my org had a Kerberos setup for some DB/middleware/CLI tools. I inherited a bunch of random script tools that took the whole typing in passwords for everything. Screw that, first thing I did was write a Kerberos module and set up a Kerberos keystore. Presto, no more typing passwords for everything. Typically it ends up boiling down to whether or not the backend thing you're talking to supports Kerberos, many don't and only have the whole SSL thing. A well setup Kerberos was much nicer to work with than chained SSL certs.

YATA0• 4 years ago

This typically still involves typing in your domain username and password, no?

cduzz• 4 years ago

I've long ago made up a corollary to Greenspun's tenth rule; any sufficiently complex or mature access regime will re-implement half of kerberos, poorly.

tomxor• 4 years ago

What you’re supposed to do is verify the key fingerprint out-of-band by asking an administrator or consulting a database or something. But no one does that.

Actually they do where I work, pretty much every time, and I didn't ask them to. Eyebrows will raise even higher if I forget to notify everyone I replaced a server at a domain causing a new signature (resulting in the scary "possible MITM attack" message). This is a good thing, but I should probably make it more efficient by publishing the fingerprints.

Although the article does point out this specific disadvantage of domain reuse with known_hosts. I can see why this solution could make things easier at larger scales.

toast02• 4 years ago

I should probably make it more efficient by publishing the fingerprints.

When I last ran a fleet of servers, I published the known hosts files via git, and strongly suggested using that in new user documentation (you needed to setup ssh_config for our jumphost/bastion anyway, may as well link to the known host keys). There's a tool to generate the files that comes with openssh, iirc.

michaelmior2• 4 years ago

What seems to be missing from this is how I assign permissions to individual hosts. When I'm using public keys, I do so by only adding the user's public key to the hosts I want them to access. It seems the way that certificates were presented that adding a user implicitly gives them access to all hosts using the same CA. I'm sure there's a solution to this problem, but does anyone have any pointers?

For example, I want hostA and hostB to use the same CA. But some users should only have access to hostA but others should only have access to hostB. Others may have access to both.

gouggoug• 4 years ago

Simply don’t create a user account for these users on the hosts they shouldn’t access.

blueflow17• 4 years ago

Set the principal name in the Certificates to the name of the users team.

On the server side, set the AuthorizedPrincipals* for root to the list of allowed teams.

tener5• 4 years ago

For scalable, centrally managed solution see: https://goteleport.com/docs/access-controls/reference/#rbac-...

Disclaimer: day work.

tuananh• 4 years ago

it's covered in this post https://engineering.fb.com/2016/09/12/security/scalable-and-...

michaelmior2• 4 years ago

Thanks! This seems like a pretty simple approach to implement but I'd also imagine it would scale reasonably well after automating various pieces.

gotaquestion• 4 years ago

Cool, how do I set up a CA?

0xdeadb00f3• 4 years ago

Precisely this. From what I've read it isn't that easy to setup a CA.

Look into step-ca though, I've heard it's.. Okay? I don't know. It seems too complicated still - I'd rather stick with pubkey auth

tfigment3• 4 years ago

Setup Hashicorp Vault. Almost easy but actyally hard to do right. Policies are easy to make too open and possibly insecure.

aidenn03• 4 years ago

All you need is an ssh key, which you can generate like this:

  ssh-keygen -f ca.key

Then you can generate certificates like this:

  # user key
  ssh-keygen -s ca.key -I key_id /path/to/user_key.pub
  # host key
  ssh-keygen -s ca.key -I key_id -h /path/to/host_key.pub

Secure ca.key according to whatever level of paranoia you desire. e.g. Passphrase, hardware security module (PKCS#12 is supported for generating certs), airgap the machine. anyone who gets access to ca.key has access to everything that trusts ca.key

0xdeadb00f3• 4 years ago

Users are exposed to key material and encouraged to reuse keys across devices. Keys are trusted permanently, so mistakes are fail-open.

What? This just sounds like you're doing it all wrong then blaming the tools.

Maybe I'm just thinking about it from my POV as a mainly hobbyist user of SSH, but I rotate my keys (not as frequently as I should, but I do), and I remove old ones from .authorized_keys files and I use different keypairs for different servers sometimes too. I never copy keys to other devices - I ssh-keygen on every device, and add my pub key to the authorized_keys file manually.

It isn't even "hard" to do it this way. Sure, it doesn't "scale" for big corps - but I don't need it to scale, so I'm not "doing SSH wrong" by not using certificates.

metalliqaz• 4 years ago

It's more effort than I would like. I should not be bothered with updating hidden config files containing data that is mostly not human readable. I want to prove my identity once and let the system take care of the rest.

0xdeadb00f3• 4 years ago

Well that's fair enough, but I take issue more with the title of the post telling me I do "ssh wrong", because I can't be bothered to setup a CA and fuck around with certificates.

To me, setting up a CA is more effort than I would like.

chasil• 4 years ago

I decided that I wanted to automate my rotation, so I built a shell script to do it, then wrote about it.

If you Google "ssh key rotation" you will find my article as a "featured snippet."

I don't think that I need a CA, as I only have personal and admin keys (two sets) and I'm flipping these once a quarter. Plus, I don't have expired entries in my authorized_keys (do CA users ever clean out authorized_keys?), and these are very clean as I see them regularly.

https://www.linuxjournal.com/content/ssh-key-rotation-posix-...

charles_f• 4 years ago

you’re doing SSH wrong

I understand the theoretical superiority to keys, but do we have some data per practically how many times key security actually failed someone?

haarts• 4 years ago

I would very much like to read about that too.

Setting some sane security parameters for your SSH setup looks like a less jarring/drastic approach into securing SSH further[1]:

- Use keys.

- Allowing only strong cyphers.

- Remove weak primes.

[1] https://disknotifier.com/blog/simple-ssh-security/

JonChesterfield• 4 years ago

The 'ideal ssh flow' involves a program that I think the author has written and a website login. Do these certificates require an existing system running single sign on or similar to hand out access to other machines, in single point of failure fashion?

soraminazuki6• 4 years ago

I feel that trying to make SSH keys short-lived is becoming more painful each year because there's an increase of tools that use SSH keys for purposes other than SSH logins. For example, age [1] encrypts files with SSH keys, agenix [2] does secrets management with it, Git can now sign commits with it [3], and even ssh-keygen can now sign arbitrary data [4]. All of these become useless the moment you start using short-lived keys.

[1] https://github.com/FiloSottile/age

[2] https://github.com/ryantm/agenix

[3] https://calebhearth.com/sign-git-with-ssh

[4] https://www.man7.org/linux/man-pages/man1/ssh-keygen.1.html

imwillofficial3• 4 years ago

Different strokes for different folks.

Different keys for uhh, different complex application use cases.

ayushnix9• 4 years ago

Umm, please correct me if I'm wrong but I think you're confusing SSH keys with SSH certificates. A SSH client key can be reused to create short lived SSH client certificates. You can keep using that SSH client key to encrypt data, sign data, login to GitHub etc. There's no such thing as "short-lived keys", there's short lived SSH certificates.

pphysch3• 4 years ago

Yes, a cert is just a public key that's been "stamped" by a certificate-authority (CA), allowing it to be validated by servers holding the CA public key (as well as enforcing other policies like lifespan, principles). It is a totally separate file and does not modify the original public or private key, which indeed have no notion of lifespan.

If you are constantly regenerating uncompromised SSH keys, you are probably doing something wrong.

The GP is misleading in this way.

soraminazuki6• 4 years ago

Yes, but the entire point of the described setup is to get rid of traditional long lasting keys in favor of ephemeral certificates (which I believe is another way of saying signed keys) obtained through SSO. Signing certificates with your existing keys kind of make the whole point moot.

hedora2• 4 years ago

Ughh. Go read up on ACME (Let's Encrypt). Unless you run your own Certificate Authority root, or configure things very carefully, using TLS certificates grants host level access to your DNS provider, and every organization that reliably routes external traffic to your host.

To what end? The threat model rekeying tries to protect against involves compromised authenticated client machines.

Once you have those, the attacker has shell on the server, and it's game over. There are these things called "advanced persistent threats" that have been in the news a lot already.

remram4• 4 years ago

This article has nothing to do with Let's Encrypt, ACME, TLS, or DNS.

pilif• 4 years ago

I love ssh certificates for access and indeed we're using this for accessing our production network using a few small pieces of home-grown infrastructure.

However, there's one big issue nobody is talking about: There is zero support for certificate authentication in any SSH clients for iOS and most people who have network access here also have iOS devices.

And even if there were support, just supporting the certificates alone is not enough - there would need to be some automatable way of getting a new certificate into an app as the whole idea of the certificates is that they are very short-lived (days or even hours if possible)

throw7• 4 years ago

The comparison to public-key authentication is wrong. Introducing cert authentication brings a third party: public key infrastructure (which isn't addressed at all and is described as some type of security panacea).

The more apt comparison should be made to other authentication technologies like, in particular, kerberos.

politician3• 4 years ago

The SSH servers that I'm familiar with are spun up with a host cert, so all of the FUD in this article about connecting to an unknown host is a non-issue. Check that the host cert matches the one you expect once, and the tooling makes sure to notify you if it changes.

As far as provisioning, maintaining a secure CA signing practice is a nightmare. It's K8S level of self-inflicted pain for a startup. If you're running at a larger scale and can dedicate a team to it, fine. If you're a dozen people trying to launch, getting the devops guy to run `ssh-copy-id` is not the challenge that this article makes it out to be. Nor is the slightly more automated Terraform script that installs and uninstalls authorized keys from servers.

gernb• 4 years ago

dumb question but .... isn't it a problem that private ssh keys are stored in ~/.ssh and that any random app, npm dependency, build script, etc could copy them across the network?

marwis• 4 years ago

You can password protect them and optionally load them to agent at logon.

dmuth• 4 years ago

Agreed--SSH certificate authorities (and principals) are powerful things that can be used to manage SSH access at scale. My workplace is a large enterprise that uses our own CA for getting access to systems--the keys it issues are good for 8 hours, then we have to grab a new key (using an internal utility).

For anyone who is interested, I put together a little playground which can be spun up in Docker that allows you to play around with and learn how SSH CAs and Principals work:

https://github.com/dmuth/ssh-principal-and-ca-playground

tonymet• 4 years ago

Great article. All of your internal authentication should be using certificates. Web auth, Wifi, VPN, SSH

In the late 90s we came close to having this for the public internet as well but it never caught on. We paid the price with endless breaches and unmanageable credentials.

_wldu• 4 years ago

SSH certificates are useful in large environments when scaling, automatic onboarding and offboarding are important, but IMO, small teams can (and should) continue using authorized key files as they have for years. They don't really need these features.

defanor2• 4 years ago

This makes it operationally challenging to reuse host names. If prod01.example.com has a hardware failure, and it’s replaced with a new host using the same name, host key verification failures will ensue.

A new machine should just have a new name. If one really wants to pretend that it's the old one, they'd better really copy it, including the keys. But even skipping that, sorting this out doesn't seem like a big deal (at least at a small scale; I suspect the article makes more sense in some scenarios than in others).

Curiously, OpenSSH chooses to soft-fail with an easily bypassed prompt when the key isn’t known (TOFU), but hard-fails with a much scarier and harder to bypass error when there’s a mismatch.

Seems to me like a sensible behaviour for TOFU, not sure what's curious about it. Sounds like it implies that an unknown key is at least as bad as a different-than-known key, but that sounds wrong in context of TOFU.

Once the user completes SSO, a bearer token (e.g., an OIDC identity token) is returned to the login utility. The utility generates a new key pair and requests a signed certificate from the CA, using the bearer token to authenticate and authorize the certificate request.

So the weakest point will likely be the SSO and the related infrastructure, instead of SSH and actual keys, and you'll probably depend on third-party services and/or custom/uncommon self-hosted infrastructure. Likely with a SPOF too. Doesn't sound good in general.

It probably does make sense in some organizations, but this particular setup doesn't seem to apply to all SSH uses, and to justify the title.

tptacek13• 4 years ago

Again, the alternative to SSO's single SPOF is many SPOFs for each individual engineer with access.

otabdeveloper42• 4 years ago

Or you could automate key distribution and revocation instead of giving away the keys to your little kingdom to Google.

(Let's not pretend that "SSO" doesn't mean "let Google or Microsoft handle password storage for me".)

defanor2• 4 years ago

How so? Are you picturing that alternative as regular records in ~/.ssh/authorized_keys, or something else?

risson2• 4 years ago

Anyone have experience with using SSHFP records to avoid the so-called anti pattern of trust on first use?

egberts17• 4 years ago

Biggest problem with SSHFS RR is the trustworthiness of DNS to deliver the answer record.

Most everything do not enforce their DNS resolver to only return the DNSSEC-verified Answer RR.

Not that problem at all if you set the resolver to return only the DNSSEC-verified answer RRs; then again, most common websites would then stop working simply because they don’t use or have a proper setup of their DNSSEC overhead.

Most implementation of distribution of the SSH public keys are delivered under cover of TLS, IPSec, or variants of secured tunneling just because … because it IS A metadata.

nix23• 4 years ago

If you trust a third party you doing ssh wrong :)

sigmonsays• 4 years ago

Who really wants to run a CA though?

quillo• 4 years ago

I have built a small signing service that works by a user SSHing in, performing LDAP (password) authentication and 2FA with duo, then injecting a time-limited certificate signed by Hashicorp Vault back to their user agent (although it could be modified to remove the Vault requirement). The UX is very simple (SSH to this address once per day), but the backend is complicated, so if there is any demand for me to put this on Github I am happy to do so.

egberts17• 4 years ago

Did it make a good use of `AuthorizedKeysCommand` option of `sshd_config`?

eternityforest• 4 years ago

AFAIK unless something SSH passwords don't use any kind of PAKE or zero-knowledge. It just straight up sends it to the server after authenticating, as the password box on a website login page would.

Really missed a great opportunity to add perhaps even more security than the server cert can(Because of how many users override it).

Come to think of it, website logins shouldn't exist either, that's way too common and I don't see why there's no PAKE based http basic auth feature.

gunapologist99• 4 years ago

So now simple and reliable SSH keys must now be replaced by a far more complex security architecture with a lot of interworking parts and a full-blown PKI and certificate authority, and the opportunity for any of the nodes to DoS the CA and prevent me from logging in.

I guess it's time to toss out my local (self-hosted) Userify setup that has been reliably working for years, where I can just instantly update my keys across all servers, and still log in even if some bad guys start DDoS'ing my Userify host, and just switch over to certs.

Oh, wait, now I see. This is a sales pitch for their web-based SSO app. If you don't use it, you're "doing SSH wrong". Good to know.

skies457• 4 years ago

Anyone using GitHub as a public key repository? I found it very convenient to set up a cron job that pulls https://github.com/USERNAME.keys into authorized_keys and SSH from any client that has GitHub access.

9dev3• 4 years ago

SSH certificates are great in theory, but the whole certificate management, ad-hoc issuance, and revocation require boatloads of infrastructure. If you do it right, certificates will be signed as needed and have a short validity period, say half an hour or something. That means you need an automated signing application, or a very cheap full-time certificate manager.

I’ve actually started working on such an app recently, including a web portal, CA rotation, automated configuration distribution, etc. Still far from usable, but if you’re interested in contributing: https://github.com/Radiergummi/fides

junon• 4 years ago

Another headline that completely ignores the concept of threat modeling. If you're preaching about what's "right" and "wrong" regarding security without considering the threat model, you're only doing harm to your readers.

egberts17• 4 years ago

I am only disappointed that SSH certificate didn’t leverage OpenSSL for their powerful capability.

I guess that’s the small price to pay for speed of development without going through the international committee of ASN.1

egberts17• 4 years ago

You can always host your own SSH CA pubkey server.

It is called an “AuthorizedKeysCommand” in `/etc/ssh/sshd_config`.

https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand...

parasense• 4 years ago

I disagree.

SSH keys seem great at first, 4KiB ~ 8 KiB public/private key-pairs are tremendously more secure than something like an 10-character password. The math checks out at an academic level, but the implementation has a glaring flaw. One cannot easily ensure private keys are themselves protected by a 10-character password unlock. Put another way, people using private keys NOT protected by a secret pass/phrase are super vulnerable to compromise. For example, physically take the laptop that contains the private key, and BOOM!

That's the jist. Private keys can be setup with passwords, but the person in control of the private key can change their key's pass/phrase at anytime after, so straight-forward key escrow strategies don't work. Inspecting the public key does not indicate the associated private key has any protection, and that's good insofar as one key not leaking information about its' counterpart.

If you’re not using SSH certificates you’re doing SSH wrong (2019)

Related Stories