jamiesonbecker

Rotating keys is easy with the right software. (I work @ Userify) Agree with the auditing point

Token-based keys, to tptacek's point, is that they can be a giant pain once you start scripting across fleets.

One key per device is exactly what we recommend too. Private keys should always be protected as much as possible within that device and should never leave that device.

Just paste all of your devices' public keys into your authorized_keys file and leave a comment at the end for what device it's for. in Userify, it literally goes right into your nodes' authorized_keys file almost verbatim. (disclaimer: I work at https://Userify.com)

And then, if you leave your token or laptop at the airport or whatever, just remove that key right from your phone and it'll take effect in seconds across all the nodes/instances (if you're using Userify) or you can just write a quick for-inline-sed loop to remove it from your authorized keys everywhere.

The next one linked at the bottom, https://jonno.nz/posts/stealing-nanoclaw-patterns-for-webapp... has this bold and frankly unbelievable claim:

"70% of startups fail due to premature scaling"

.. which is a link to another blog post somewhere else that says nothing even slightly related.

Then install your own:

    curl i.userify.com | sudo - sE

The experience might be better right up until you're running it in prod and someone happens to ask about:

   Cert revocation (or even expiration)

   Sudo roles

   User removal and process termination

   Is the cert server HA and locked down

   How you log in when the cert server is down or under attack (rich target!)

   How to easily add Alice to server group A, Bob to B, and Carlos to both A and B, and then to remove them..

Honestly, we used to replace a lot of pam_ldap and similar sorts of awful solutions. With those, if your LDAP went down even for a heartbeat, you couldn't log in at all.

So I totally agree: if I had to do certificates and didn't have something like Userify, a 1 hour (or even shorter if possible) expiration seems quite worth chasing, especially with suitable highly available configuration. (Of course, TFA doesn't even bother mentioning revocation and expiration, which should give you a clue as to how much fun those are lol)

And for more normal, lower-security requirements or non-HA, 6 or 8 hours or so would probably work and give you plenty of time for even serious system outages before the certs expired.

Not to hard shill or anything (apologies in advance, just skip if you're not interested), but there are two significant security and reliability differences between standard SSH (with or without certificates) and Userify:

1. Userify Cloud updates by default every three minutes, and on-premise Userify Express/Enterprise updates every ten seconds, but it doesn't have to update at all; even if your Userify server goes offline forever, you can still log in because the accounts are standard UNIX accounts (literally created with `useradd`)

2. When accounts are removed, Userify also completely nukes the user account, removes its sudo perms, and totally kill -9 's any tmux/screen/etc sessions (all processes owned by the user are terminated across the entire enterprise within seconds), which is also not something that a certificate expiration would ever do.

Great question. Not yet ;)

Well, TOFU is really just the model for how the chain of trust is established.

In practice there isn’t really trust on first use: there’s verify the key matches what’s expected, or distribute keys out-of-band (including certs).

If that verification step isn’t happening, then it’s not TOFU, it’s just blind trust.

From an automation/autoscaling angle, the same thing shows up again:

1. either keys are pre-baked / distributed

2. or, something signs them at boot

Signing an instance key is just another way of distributing trust. It doesn’t remove the need for a root of trust, it moves it.

Certificates just add extra steps around the same underlying task.

That works for authn in the happy path: short-lived cert, grab it, connect, done.

Except for everything around that:

* user lifecycle (create/remove/rename accounts)

* authz (who gets sudo, what groups, per-host differences)

* cleanup (what happens when someone leaves)

* visibility (what state is this box actually in right now?)

SSH certs don’t really touch any of that. They answer can this key log in right now, not what should exist on this machine.

So in practice, something else ends up managing users, groups, sudoers, home dirs, etc. Now there are two systems that both have to be correct.

On the availability point: "reasonably available" is doing a lot of work ;)

Even with 1-hour certs:

* new sessions depend on the signer

* fleet-wide issues hit everything at once

* incident response gets awkward if the signer is part of the blast radius

The failure mode shifts from a few boxes don't work to nobody can get in anywhere

The pull model just leans the other way:

* nodes converge to desired state

* access continues even if control plane hiccups

* authn and authz live together on the box

Both models can work - it’s more about which failure mode is tolerable to you.

SSH certs quietly hurt in prod. Short-lived creds + centralized CA just moves complexity upward without solving the core problem: user management.

The system shifts from many small local states to one highly coupled control point. That control point has to be correct and reachable all the time. When it isn’t, failures go wide instead of narrow.

Example: a few boxes get popped and start hammering the CA. Now what? Access is broken everywhere at once.

Common friction points:

     1. your signer that has to be up and correct all the time
     2. trust roots everywhere (and drifting)
     3. TTL tuning nonsense (too short = random lockouts, too long = what was the point)
     4. limited on-box state makes debugging harder than it should be
     5. failures tend to fan out instead of staying contained

What actually happens is people reintroduce state anyway: sidecars, caches, agents… because you need it.

We went the opposite direction:

     1. nodes pull over outbound HTTPS
     2. local authorized_keys is the source of truth locally
     3. users/roles are visible on the box
     4. drift fixes itself quickly
     5. no inbound ports, no CA signatures (WELL, not strictly true*!)

That’s basically what we do at Userify (https://userify.com). Less elegant than certs, more survivable at 2am. Also actually handles authz, not just part of authn.

And the part that usually gets hand-waved with SSH CAs:

     1. creating the user account
     2. managing sudo roles
     3. deciding what happens to home directories on removal
     4. cleanup vs retention for compliance/forensics

* (TLS still exists here, just at the transport layer using the system trust store. That channel delivers users, keys, and roles. The rest is handled explicitly instead of implied.)

But then you can't log in if your box goes offline for any reason.

We're in the process of updating the experience to this century! ;)

We've always taken the stance that crusty is better than vulnerable, but it turns out that not having a modern experience after 15 years is starting to feel like maybe we need to step up the features and shininess :)

Exactly. We'd had discussions about building https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity.

It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.

Classic OpenSSH safety check: if /home/$user (or ~/.ssh) is too open, or ownership/modes are off, sshd will refuse pubkey auth. Annoying, but correct.

If you still have some access (console, password login, another sudo user), this usually fixes it:

    username=bob
    sudo chown "$username:$username" /home/$username
    sudo chmod 700 /home/$username

    sudo install -d -m 700 -o "$username" -g "$username" /home/$username/.ssh
    echo "ssh-ed25519 AAAA....insertyourpubkeyhere" | sudo tee /home/$username/.ssh/authorized_keys >/dev/null
    sudo chown "$username:$username" /home/$username/.ssh/authorized_keys
    sudo chmod 600 /home/$username/.ssh/authorized_keys

    echo "$username ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/$username >/dev/null
    sudo chmod 440 /etc/sudoers.d/$username

Shim (old but readable): https://github.com/userify/shim/blob/master/shim.py#L308 (obligatory): https://userify.com

at least it had a minimum of Clause. Clause. Punchline.

I like it too, even though it has that distinctive odor of being totally written by chatgpt though. (a bit distracting tbh)

One of the big issues was latency.

Thank you for the comprehensive answers!

(By the way: NVIDIA AIstore is NOT a proxying/caching engine, although it can, which is somewhat unique among these types of stores. AIstore is actually a full S3 engine in its own right, and it's actually extremely capable, although live cluster resizing and ETL requires k8s :( )

These questions are meant to be constructively critical, but not hyper-critical: I'm genuinely interested and a big fan of open-source projects in this space:

* In terms of a high-performance AI-focused S3 competitor, how does this compare to NVIDIA's AIstore? https://aistore.nvidia.com/

* What's the clustering story? Is it complex like ceph, requires K8s like AIstore for full functionality, or is it more flexible like Garage, Minio, etc?

* You spend a lot of time talking about performance; do you have any benchmarks?

* Obviously most of the page was written by ChatGPT: what percentage of the code was written by AI, and has it been reviewed by a human?

* How does the object storage itself work? How is it architected? Do you DHT, for example? What tradeoffs are there (CAP, for example) vs the 1.4 gazillion alternatives?

* Are there any front-end or admin tools (and screenshots)?

* Can a cluster scale horizontally or only vertically (ie Minio)

* Why not instead just fork a previous version of Minio and then put a high-speed metadata layer on top?

* Is there any telemetry?

* Although it doesn't matter as much for my use case as for others, what is the specific jurisdiction of origin?

* Is there a CLA and does that CLA involve assigning rights like copyright (helps prevent the 'rug-pull' closing-source scenario)?

* Is there a non-profit Foundation, goal for CNCF sponsorship or other trusted third-party to ensure that the software remains open source (although forks of prior versions mostly mitigates that concern)?

Thanks!

8. Google kills Gemini Cloud Services (killedbygoogle.com)

So, Google renamed itself to Gemini?

Instead of Google having a product named Gemini, Gemini has a product named Google.

pattern recognition much?

Except that the definition of a script kiddie is a point-and-shoot shotgun approach - so the AI is basically another script kiddie.

To your point, though, the AI is a highly skilled script kiddie!

Expectations for pay seem to be very high even for people only just out of college.

Darn kids get off my lawn!

But yes, you're right. Salary expectations are surprisingly high considering how little fresh grads bring to the table!

We're talking huge amounts of training just to get to a basic level of competency, especially since most CS degrees are focused on things that were cool five years ago (or much, much longer).

This is the age-old tale, though, but now time is compressed and we need people to hit the ground running much, much faster.

Yes, you're exactly right. ( https://github.com/userify/shim/blob/master/shim.py#L209 )

We've thought about porting Userify to work with CA's too but haven't had many requests for that for some reason, even though I'm sure many companies do have CA's set up alongside their other PKI for SSH.

CA's have a lot of management and logistical issues and potential for misuse. The simplicity and TOFU design of the SSH key system (which obv can bring along some issues of its own) can bring a lot of benefits, especially for people who don't want to introduce a CA or PKI.

(obligatory disclaimer, I work at Userify and we have a server-side product that automates SSH key management and distribution. For example, the CA design doesn't kick someone out once their access is removed, but Userify's shim actually terminates all of sessions instantly, like screen or tmux, across all of the servers they're logged into and removes (but retains for historical record) their home directory.)

I agree!

People should generate at least a single ssh key per client device. (On Userify, rotating your key is just a matter of pasting the new public key into your keybox in your dashboard.) One per client device will let you revoke/rotate only that key when it's compromised. This also helps keep you from copying the private key somewhere else (which you should never do).

It does look like this wants to be a replacement for ssh-agent/ssh-add; also check out GNU keychain by Daniel Robbins, which is in most distro repos.

(blatant plug - we actually developed Userify for these three use cases, especially on cloud instances with constantly changing IP's)

Right. We usually recommend a single key per client device (laptop, desktop, etc), because that way you can rotate that key if it gets lost/stolen without changing your other devices as well. This way, those private keys stay totally local to the device and never actually need to move, which is much safer. (I work at Userify.)

Very nice for a first project! (the clipboard integration is a nice touch)

Seeing this passion is great! TBH, some of the negative comments here might be warranted: SSH is an especially important and tricky area to start in as your first Github project or for those inexperienced in security.

However, we're always looking for people at Userify to help us build the next wave of SSH UX and who aren't afraid to put something out there. It's a lot of work to get things built and it's very exciting to see some new ideas. Hit me up if you want to talk!

That's not what the article is talking about.

I replied below about our homegrown solution, but this is very cool! I wish it'd existed when we wrote Userify, but we'll definitely plan on it for our next app! Kudos for solving this so elegantly!

Agreed, but Okta's still a $14B company.