Keycloak with PostgreSQL on Kubernetes

https://blog.brakmic.com/keycloak-with-postgresql-on-kubernetes/

by brakmic • 3 years ago

158 77 3 years ago

blog.brakmic.com

vxxzy• 3 years ago

Ah nice! I use Keycloak in conjunction with NetMaker. It seems to work well! I’d like to figure out a way to somehow get ssh authentication with keycloak. I’ve read of oauth + ssh certs, but all of it seems so cumbersome. It would be cool to have an open source alternative to StrongDM.

bebop• 3 years ago

Super roughly, but you might be able to implement an Authentication SPI[0] and wire that into an Authorization Code flow.

[0] https://www.keycloak.org/docs/latest/server_development/#_au...

hotpotamus2• 3 years ago

I've been down this road a bit, though actually in Docker Swarm. One aspect I spend a lot of time digging into was running multiple keycloak containers with shared cache. On metal or a VM with multicast, they'll find each other no problem, and it works beautifully, but I'm not aware of any container orchestration that brings multicast out of the box (and I don't think AWS does either). Keycloak has a built in Kubernetes DNS discovery mechanism to find its peer containers and share cache which also worked quite well on Swarm, though I lost a day or two tweaking it.

rad_gruchalski12• 3 years ago

AWS supports multicast in VPCs.

hotpotamus2• 3 years ago

Curious - I've seen several references that it doesn't support it, and that keycloak has a dedicated ec2 cache discovery option. But I don't use AWS anyway, so I'm far from knowledgable about it.

vbezhenar7• 3 years ago

Yes, Keycloak cluster works fine on Kubernetes. It takes some time to read all the docs and understand things, but nothing outrageous, that was my experience at least.

xupybd4• 3 years ago

I've just started using Keycloak to provide OpenID for F# Safe stack applications.

Wow the learning curve was steep on that one. Not having ever touched OpenID or anything other than forms based authentication and not knowing ASP.Net very well.

But it's neat to get it all up and running. Still a few issues with getting Keycloak to redirect to HTTPS but we will get there.

rad_gruchalski12• 3 years ago

Maybe this will be helpful? https://gruchalski.com/posts/2022-02-20-keycloak-1700-with-t.... I’m the author.

xupybd4• 3 years ago

Thank you!

That looks like the exact problem I'm facing. I'll try it out today!

Thanks again!

andix3• 3 years ago

That’s exactly what needs to be done. It is also in the keycloak documentation, but not as easy to find as in your post.

andix3• 3 years ago

The most disappointing problem with asp.net for me was, that there is no backchannel logout. So you can’t easily force-logout users from oidc/keycloak.

Everything else was going pretty smooth, although the authentication documentation for asp.net really sucks.

xupybd4• 3 years ago

Yeah I've hit that. So if you log them out in your application that will remove the cookies and they won't show as logged in but the next redirect to Keycloak shows an error.

The documentation sucks for ASP.net and it's far worse for the Safe stack.

You have to understand the stack so you have to read up on the following.

  ASP.net
  Giraffe   
  Saturn
  Fable remoting
  Keycloak
  OpenID

Once you have a good understanding of all of those you can start to understand the half a dozen blog posts that attempt something similar.

pharmakom• 3 years ago

Been doing similar stuff and found that ASP.Net stuff wasn’t worth the hassle compared to a custom set of functions on top of Giraffe.

xupybd4• 3 years ago

Yeah you're right.

I built my own auth, forms based with blowfish encryption is a few hours. Then I felt like I was doing it wrong. So I looked at OpenID. It's been two weeks and it just working. It'll take me. A few days to document it well enough to be happy I can keep it running.

I made a poor choice and introduced too much complexity.

photonios3• 3 years ago

If there's anyone reading this that is planning on deploying Keycloak in a high availability environment, I would highly recommend that you persist all sessions in the database as offline sessions.

At work, I ran 9 Keycloak clusters in production, handling tens of millions of sessions where the cost of losing sessions was high. The amount of time we wasted on getting it to work reliably with its default configuration of storing the sessions in its distributed, in-memory cache (Infinispan) is insane. It just isn't designed to handle such a work load reliably. Unless you're willing to spent months tuning it for every possible scenario, you WILL lose sessions.

If you are in this situation, shoot me an email. I have been through this pain and it took a lot of painstaking work to get to a highly reliable set up at scale.

ArchOversight• 3 years ago

Do you have a blog post or something detailing what you did and how you did it?

rad_gruchalski12• 3 years ago

I found this: https://www.janua.fr/offline-sessions-and-offline-tokens-wit.... janua.fr is a very solid Keycloak resource. The write up is for a pretty aged Keycloak version but there are probably some decent pointers in there.

zakokor• 3 years ago

I would highly recommend that you persist all sessions in the database as offline sessions.

Please! Post it, thanks

paulmd• 3 years ago

Was it something about Infinispan, or Keycloak?

We were wondering about Redis in a similar IAM use-case (PingFederate) but it wasn't officially supported, so we decided to just go with persistent Postgres. I wonder if we saved ourselves a bunch of heartache.

photonios3• 3 years ago

We often experienced cascading failure, especially during rolling restarts. A node would start shutting down and Infinispan would start to try to rebalance. Due to the large volume of sessions, other nodes would start to become unresponsive and stop replying. Eventually, you'd end up in a situation where it would give give up trying to shut the node down cleanly and just kill itself. That wouldn't be a big deal if you weren't doing a rolling restart. When the first node doesn't shut down cleanly, the data should be "safe" since it is replicated to at least N owners. In practice, the other nodes also get restarted, also shut down uncleanly and sessions are lost. Secondly, as the cluster became unresponsive, requests to refresh sessions would start to time out, which would also cause those sessions to be "lost" since they would eventually hit the maximum idle time.

As long as we wouldn't do any restarts, it would sort of work. Problems would pop up when due to high load, one or more nodes would become unresponsive and liveness probes would restart nodes. That would often cause the kind of cascading failure described above.

Most of these problems are also the result of running it in Kubernetes. We very quickly learned to remove the liveness probes and to massively increase the grace period. This helped, but only so much. We still had rather frequent failures similar to the one I just described.

Maybe if we wouldn't have run it in Kubernetes and we would be more knowledgeable about Infinispan, we could've gotten a stable set up. For us, as a small team without that specialized knowledge, we struggled to get a stable set up.

thegreatunknown• 3 years ago

Hi, You might want to take a look at the new storage in keycloak[1].

Newer keycloak versions (19 and up) have a configurable storage for the auth sessions (see storage-area-auth-session and storage-area-user-session). I haven't checked them but the documentation is promising.

For older session (last time I checked keycloak 15) you might want to use offline sessions but they don't allow SSO after the auth session was evicted from infinispan.

1 - https://www.keycloak.org/2022/07/storage-map.html

vsviridov5• 3 years ago

As a possible alternative, I've recently started using Zitadel (https://zitadel.com/) which is a very full-fledged open source IDP, in active development.

upcoming-sesame• 3 years ago

Ory Hydra / Kratos is another good one

https://www.ory.sh

vsviridov5• 3 years ago

I guess with Zitadel they don't paywall any features, and with self-hosted option you get essentially the same thing as with hosted. I think you can probably even do multi-instance, however maybe without a management interface for that part of it... When I read Ory language, it says to me "you can try it out locally and itegrate, but we want you in our hosted solution right after" (I could be wrong, I was just glancing casually...)

vbezhenar7• 3 years ago

Is there solid documentation or tutorial on deploying hydra + kratos? That's what stopped me last year. I've found some subtle pointers in some buried github issues, but nothing definitive. Which surprised me because one would thought that's the main use-case for many people and should be documented well.

dariusj18• 3 years ago

This looks very interesting, I have recently tried using auth0 and was so horribly disappointed with how you go from 0 to enterprise as soon as you need any modern security feature. Plus I had assumed that they had a mature product, but is chaotic and difficult to know what you are doing for even the simplest use cases.

jonas-w• 3 years ago

How does zitadel compare to authentik (https://goauthentik.io/) or authelia (https://www.authelia.com/)?

vsviridov5• 3 years ago

I don't think authelia has a UI, and it also has a mode where it integrates a bit more deeply with the routing mesh, to protect apps that do not do auth themselves. Authentik I've not looked into. It also seems that they differentiate self-hosted options into free and paid with different features...

zinclozenge3• 3 years ago

Zitadel could be amazing, but as far as I can tell they don’t allow using your own UI screens, and it’s not obvious to me how you’d build a multi tenant SSO feature. They have the concept of organizations, but it’s not obvious to me how you’d route a user to the right login.

vsviridov5• 3 years ago

Organizations are tied to domains, but yeah, that part is more confusing than the rest...

Instance also has a domain on top of that, but there are plans for a "simple" mode, assuming single org.

mffap2• 3 years ago

You can enable Domain Discovery to route users to the correct organization. Or you send a reserved scope with the auth request to select the organization. Building an own Login UI will be available in a couple of weeks (https://github.com/zitadel/zitadel/issues/5015)

hardwaresofton2• 3 years ago

Wow Zitadel looks absolutely amazing. All the features you want out of login, permissive license, good pl choice, easy to deploy, postgres-first.

Since I somehow can’t resist complaining about absolute mana from heaven:

The only issue I see is this reliance on event sourcing — I get the reasoning but I much prefer regular state saving + audit log approaches.

Event sourcing seems like a complexity and performance liability — does anyone have any insight on the implementation/why I am wrong about my misgivings?

vsviridov5• 3 years ago

The amount of events in such system isn't going to be too crazy, unless it's some massive enterprise with thousands of principals, I would imagine...

It also seems cockroachdb first, but I'm glad i can use postgres. One fewer database to deploy and manage, and for my use case (basically myself and occasional friends and family) that's perfectly fine.

ahachete3• 3 years ago

This is good and interesting recipe to get Keycloak and Postgres on Kubernetes.

There is an important improvement, though: the Postgres deployed here is not production ready (high availability, backups, monitoring, etc).

We run Keycloak on StackGres [1] which gives us production-ready Postgres setup (disclaimer: it's dogfooding). Happy to share the YAML manifests used to deploy Keycloak with StackGres. Maybe we will write a blog post as a follow-up to this one, for completeness.

[1] https://stackgres.io

rad_gruchalski12• 3 years ago

There is an important improvement, though: the Postgres deployed here is not production ready (high availability, backups, monitoring, etc).

Another omission is that one could use a Keycloak operator instead of rolling custom YAML.

ahachete3• 3 years ago

Yes, absolutely.

bbu• 3 years ago

I gotta say the (new) keycloak operator is super basic and doesn't really support changes in image tag. it always assumes a keycloak upgrade and will automatically scale down your keycloak to 1 instance to do an "upgrade". of course this will overload that one node, it will crash, and all your sessions are gone. I'm not sure if anybody is actually using keycloak & kc-operator in production on kubernetes. the state of the documentation and guides make it look like it's an abandoned product.

slig2• 3 years ago

How do you compare StackGres to CrunchyData's `postgres-operator`?

ahachete3• 3 years ago

I don't want to go too offtopic on this one --feel free to join StackGres Slack Community [1] to discuss further.

As a one-liner, though, for completeness: StackGres is fully open source (unlike Crunchy that needs a license for production); comes with a Web Console; 150+ Postgres extensions (including Timescale, Citus and many others); and many Day 2 operations fully automated.

[1] https://slack.stackgres.io

hsn9154• 3 years ago

Am I the only one for whom this kind of thing is painful to read?

I am kinda curious though about the kind of personality type that enjoys this kind of stuff.

Of course, I have never heard of "Keycloak" before, so I checked their homepage:

"No need to deal with storing users or authenticating users."

Wait a second, is dealing with storing users and authenticaing them _so much pain_ that you rather inflict yourself with the pain of setting up and managing a k8s cluster?

I seriously don't get it.

rad_gruchalski12• 3 years ago

No. The article talks about setting up Keycloak in Kubernetes (and it does that poorly because it’s not a production setup) but you don’t need Kubernetes: https://gruchalski.com/posts/2022-02-20-keycloak-1700-with-t....

hsn9154• 3 years ago

I find that equally triggering

rubentanlz2• 3 years ago

Hopefully this doesn't come across as off-topic but the "smooth scrolling" or whatever that is that hijacks the normal scroll behavior is throwing off my scrollwheel, making the website nigh impossible to navigate. Only way I can scroll properly is by clicking on the scroll bar and dragging it up and down manually.

toomanydoubts• 3 years ago

Yes. As a rule of thumb, unless you're making some kind of experiment/poc/artistic work please don't hijack the native browser scroll ever. Hate that this still has to be said in 2023. We like and (maybe) configured the scroll on the OS to our own taste, it's pointless to try and impose an slower, inferior and bug-ridden non-native implementation of the scroll just to add some kind of "smoothness".

brakmic3• 3 years ago

Hi, OP here.

Thanks for the hint.

I just disabled smooth scrolling. Sorry, not a sophisticated designer guy, using wordpress plus some UI themes.

Regards,

rubentanlz2• 3 years ago

Thanks, the article content is very useful for me as we're using k8 and keycloak together right now.

boris-ning-usds• 3 years ago

I've been reading up on Keycloak recently, and had questions for hosting keycloak in prod.

How do people in the field handle configuration updates with code? For example, if I want to set it up as an identity broker to an idp, I would want that configuration backed by code, reviewed by my team. Is anybody using the keycloak terraform provider https://registry.terraform.io/providers/mrparkers/keycloak/l... in production?

Do people diff the realm json configuration as code and use that instead?

vbezhenar7• 3 years ago

Same pain here. I've tried terraform and it works, but I just hate it because of state management. We're doing things manually, like someone changes stuff on test server, writes everything down and then at deploy hour repeats those changes on production server. This is not nice.

skhro872• 3 years ago

you can try keycloak-config-cli https://github.com/adorsys/keycloak-config-cli we've been using it in production for 2 years and it works well! we are running it as part of our CICD, to sync settings to all Keycloak realms. As the tool supports variable substitution, it makes it quite flexible. The config file it uses is basically the same realm.json you can export from Keycloak, so it doesn't re-invent the wheel.

vbezhenar7• 3 years ago

Why postgres is running as privileged?

brakmic3• 3 years ago

Hi,

Many thanks for the hint. It must have been a leftover setting from one of the other variants I am using locally.

Yes, postgres doesn't need to run as privileged. I changed it to "false" and updated the github repo.

Regards,

vbezhenar7• 3 years ago

I miss the ability to use some kind of GitOps with Keycloak. There's Terraform plugin, but I hate it (because of state). I wish there was some kind of config file which Keycloak would read at startup and create/update/delete its resources according to it. I know that I can initialize realm with JSON (with unreadable structure), but I can't maintain realm with config file.

pat2man• 3 years ago

Have you tried the operator? https://www.keycloak.org/operator/advanced-configuration

vbezhenar7• 3 years ago

I looked at it but I don't see the features I'm talking about. I can configure pod myself, so I don't understand why would I need this operator.

skhro872• 3 years ago

vbezhenar7• 3 years ago

It looks promising, thanks!

Too• 3 years ago

I may be wrong but is it not preferable to use StatefulSet for databases, rather than PV, PVC and Deployments?

You will not be able to scale anything up anyway since it’s a single instance mounting the same data.

brakmic3• 3 years ago

Hi,

I don't think you're wrong. StatefulSets are simply on a "higher level" than raw Deployments and manually provided PersistentVolumes/Claims. I am thinking about writing another article that shows how to use StatefulSets in similar scenarios.

Regards,

Keycloak with PostgreSQL on Kubernetes

Related Stories