Fermilab/CERN recommendation for Linux distribution

https://news.fnal.gov/2022/12/fermilab-cern-recommendation-for-linux-distribution/

by jaboutboul • 4 years ago

225 139 4 years ago

news.fnal.gov

HexDecOctBin• 4 years ago

reflecting recent experience

What happened? Is this about CentOS's demise?

cozzyd2• 4 years ago

Yes. It's too bad because they used to support Scientific Linux, but stopped because CentOS was so popular...

Anyway, I'm glad I went with Alma over Rocky, albeit for the wrong reasons (my daughter's name is Alma...)

sgc• 4 years ago

Sounds like a wholesome reason to me.

thiagocsf• 4 years ago

You probably know this, but Alma means “soul” in Portuguese.

Is this why you picked the name? I like it. I went with Ada myself (after Lovelace - wife only found-out why years later).

djha-skin2• 4 years ago

All this time I had thought that Rocky Linux was winning the fork war over Alma (in the fight to be the successor to CentOS), but this post might change that with a good chunk of the science community throwing their weight behind Alma.

Do we have anyone else in the audience that has any insights over whether Alma is more prevalent over Rocky or is it the other way around?

I know I can run Rocky Linux from DigitalOcean which I appreciate.

darthrupert• 4 years ago

Alma

Having just played through the F.E.A.R. games that name gives me weird feelings.

themodelplumber• 4 years ago

Having been assigned the name in a creepy religious temple ceremony we are in the same boat

pelagicAustral• 4 years ago

I love that game, but the first thing that came to mind was the satellite dish array somewhere in the desert.

actualwitch• 4 years ago

Having just watched through undone series that name also gives me weird feelings.

spookie• 4 years ago

Means soul in Spanish/Portuguese

friendzis• 4 years ago

Do we have anyone else in the audience that has any insights over whether Alma is more prevalent over Rocky or is it the other way around?

This is wrong reasoning. Distributions live on the shoulders of maintainers/developers. While popularity may be a factor attracting maintainers/developers it is not a deciding factor. If the less popular distribution attracts more maintenance/development effort (again, not necessarily head count) it will win back market share in medium term and once popular distribution will stagnate and fade.

sliken• 4 years ago

We picked Rocky Linux because it was the first to get full support from Mellanox/Nvidia.

rurban2• 4 years ago

I have insights.

Alma is done by professionals for professionals (the hosting industry). Those guys were behind CentOS, and are running 70% of the internet.

Rocky is an unpaid community effort.

dapozza• 4 years ago

No.

Rocky is done by the guy that started CentOS.

bclemens5• 4 years ago

No.

Alma Linux is done by CloudLinux, and still uses their infrastructure, secure boot certificate, etc (according to their page at https://web.archive.org/web/20221208102246/https://wiki.alma...). They have never had anything to do with CentOS.

Rocky Linux _is_ community oriented, and _is not_ beholden to a specific company, but it is not necessarily unpaid. The majority of the most active contributors to the project are being paid by their respective organizations (CIQ, OpenLogic / Perforce, etc) to do so.

bclemens5• 4 years ago

Rocky is "winning" if by "winning" you mean more widely used. Here are some graphs charting usage through EPEL statistics: https://rocky-stats.tiuxo.com.

The source used to generate the graphs is available at https://github.com/brianclemens/rocky-stats, however please be kind to the Fedora servers and don't download the stats database too often if you use it.

Also Rocky Linux is far larger in terms of community size, etc.

comfypotato• 4 years ago

Wow: those Alma Linux curves look nice. The Rocky Linux trends are definitely more aggressive, but they look a little bit tulip-fever-ish.

Also noteworthy is the linear nature of the RH progress.

I welcome any further interpretation in a similar (or contradictory) vein to my thoughts.

KronisLV3• 4 years ago

Here are some graphs charting usage through EPEL statistics: https://rocky-stats.tiuxo.com.

As someone who has had to use Oracle Linux for quite a few projects due to requirements, those graphs are actually a sobering look on things.

Of course, all of those distros are reasonably similar at the end of the day, but it's pretty clear that the popularity of Rocky Linux, Alma Linux, and even CentOS Stream are all formidable.

fomine32• 4 years ago

Also note that genuine RHEL users generally don't want to use EPEL that is out of support.

The EPEL repository is used by a significant portion of Enterprise Linux users. The data they share are unbiased, and it is reasonable to assume approximately equal proportions of users of each Enterprise Linux distribution use the EPEL repository.

From the article but I disagree.

rwmj• 4 years ago

Wouldn't it make sense for Alma & Rocky to now combine their efforts?

axytol• 4 years ago

It depends. As a counter example think of Scientific Linux [0] which was also an RH clone supported by Fermilab and CERN. It eventually got replaced by this "obscure" distro called CentOS.

[0] https://en.m.wikipedia.org/wiki/Scientific_Linux

infinet• 4 years ago

Alma is not exactly prevalent. It however releases faster than Rocky. Alma's mailing list is also more informative. Its Announce list post security advisory frequently. Rocky's Announce list in comparison is quiet.

mhitza• 4 years ago

While I didn't have many projects recently that focused on EL distros. Last year, I wanted to publish a public AMI built on top of Rocky. However, someone at the time decided to configure the published Rocky Linux AMIs to disallow republishing, so that's the only time I actually picked Alma over Rocky Linux.

After the dust settled behind the CentOS model changes, I think there might be a reasonable amount of users that also shifted to CentOS Stream and Fedora linux (Amazon Linux 2022 is also Fedora based instead of RedHat). For the scientific community something that changes way less is obviously a different story.

notorandit• 4 years ago

Not really interesting. Without any technical reasoning other than "we want red hat with extra perks".

djha-skin2• 4 years ago

It's interesting because there's a question in the air right now over what fork of CentOS will be its spiritual successor since it got bought out. Will it be Alma, or Rocky? This post holds in favor of Alma.

dotancohen3• 4 years ago

As much as it pains me to say it, FOSS devs are a finite resource and it seems that being divided is just as if having been conquered. Neither Rocky nor Alma are progressing at the rate that a CentOS fork with a hostile RH upstream would progress if they were working together and not duplicating efforts.

What is the philosophical schism between the two? Why can't they split the workload on a single distro and reunite the communities?

ilyt3• 4 years ago

Probably "we picked RedHat ecosystem and are now stuck in it with significant cost to change", especially if they build their own pacakges

1letterunixname2• 4 years ago

I went through the Rocks Cluster cycles in 3, 4, and 5 with SLC and Scientific until those fell by the wayside.

Rocky is the underdog you want to win.

Alma is the leader except in terms of security update latency.

The problem is that Cent 8/9 Stream has quicker critical CVE patches because it's essentially the source and is closer to mirroring RHEL.

It's hard to convince corporate folks to use Alma when Cent is still the "safer" choice technologically and provides continuity even if its governance and lifecycle maybe worse.

jacooper3• 4 years ago

Apperantly stream is the slowest. https://news.ycombinator.com/item?id=33905616

carlwgeorge4• 4 years ago

Not exactly. For most CVEs, CentOS Stream gets them months before RHEL and RHEL clones. But CVEs rated important/critical (or otherwise embargoed) are required to go out to RHEL customers first. Once the fix is live for RHEL customers, two things happen more or less simultaneously:

- The RHEL package source with the fix is published, allowing clone distros to start their rebuild work. - RHEL maintainer starts working on the fix for CentOS Stream in public. This may or may not be the same patch as what was released in RHEL, depending on whatever other changes have already happened in CentOS Stream.

These tasks are not the same and take different amounts of time. On top of that, the release pipelines to get the fix out to users are different between the various distros. Sometimes the fix is live for CentOS Stream before the rebuilds, sometimes after. There were some notably slower exceptions with CentOS Stream 8 in the early days, because CentOS Stream 8 is built "inside out".

https://twitter.com/carlwgeorge/status/1439724296742576130

In CentOS Stream 9 going forward, RHEL maintainers own their CentOS builds and things are working much better. There is work in progress to migrate 8 workflows to match 9.

bayesian_horse2• 4 years ago

What I didn't like about AlmaLinux so far is that it takes more effort to google solutions to problems than for Ubuntu.

That may sound stupid or lazy, but there it is.

keewee72• 4 years ago

AlmaLinux is highly compatible with Fedora and CentOS. You should be able to use the same solutions.

bayesian_horse2• 4 years ago

I am. The impression remains.

flumpcakes5• 4 years ago

Ubuntu was/is by far the more popular distro to use outside of the corporate world. There's a lot more blogs, container base images, tutorials out there because it was popular for tinkerers. This is also the reason it has now become popular inside the corporate world.

You may find it harder to find resources for AlmaLinux/CentOS/etc. if you are using the same methods for searching for resources as you would for a Ubuntu (or even Debian) based OS.

However, there are much better and more fully comprehensive training material out there for 'Enterprise Linux' distributions (RHEL, CentOS which means also AlmaLinux and Rocky) because of the widely industry accepted RH certifications.

So to wildly make broad statements: Ubuntu based distros are good for finding resources like blogs and tutorials that are aimed at people who are tinkering and just starting out, maybe from a dev background. EL distros are good for comprehensive training for people who probably have had "Sys admin" as a job title at some point.

KronisLV3• 4 years ago

You may find it harder to find resources for AlmaLinux/CentOS/etc. if you are using the same methods for searching for resources as you would for a Ubuntu (or even Debian) based OS.

Not only that, but as soon as you venture outside of the "happy path", you'll run into difficulties if others haven't also solved those problems ahead of time, which can be the case with many DEB distros but isn't always so with RPM ones.

For example, in many RPM distros you'd typically reach for using Podman for running container workloads, maybe even use something like OpenShift (their MicroShift project still being in the works).

In contrast, on DEB you'd at least sometimes go with Docker, because it's widely known, widely used and still has lots of market share - offering such lighweight orchestrators like Nomad or even Docker Swarm out of the box, which still is an okay choice for getting a dev environment running on a few boxes with 8 GB of RAM, when you don't have a separate node for managing the cluster.

Now suppose that you decide to go with Docker on RPM. It's definitely feasible and has gotten better, but I recall just a few years back Docker having issues with SELinux and also the container networking utterly breaking because of firewall configuration. You had to use the masquerade option if I recall correctly, which definitely fixed the issue but also meant that the software package ships broken by default, in a sense.

On an unrelated note, I recently ran into an issue with one of the RPM distros a week back, where if the node has swap enabled and it runs low on swap memory, it's going to have the kswapd process start eating a lot of CPU resources and basically bring everything else on the node on a standstill.

Not to say that RPM is worse, the LTS EOL cycle alone is amazing, so I wish Rocky and Alma (As well as other distros) the best of luck! But I can also definitely recognize why many would stick with Debian or Ubuntu (or other DEB distros).

vbezhenar3• 4 years ago

Half of my searches are thoroughly explained in access.redhat by very competent RH support answers. Ubuntu answers usually are just bad.

jeffbee2• 4 years ago

It’s totally impossible to google the solutions for Ubuntu because they’ve completely changed the way everything works, multiple times, so all the answers are wrong.

Mikhail_K6• 4 years ago

Is AlmaLinux systemd-free?

Dracophoenix• 4 years ago

It's a RHEL clone, so no. However, you can install runit or OpenRC if you want.

Mikhail_K6• 4 years ago

That doesn't solve the problem, because systemd developers have insinuated dependencies on it into multiple unrelated system components.

rahen5• 4 years ago

You're asking if Redhat, the creator and main proponent of systemd, uses it in its distribution.

Use FreeBSD, Devuan or Alpine if you want production-grade Unixes without systemd.

Mikhail_K6• 4 years ago

You're asking if Redhat, the creator and main proponent of systemd, uses it in its distribution.

I thought the main promoter of systemd these days is Microsoft?

keewee72• 4 years ago

How are these long-term support Linux distros able to provide LTS for the thousands of packages they include?

If a security bug is discovered in an old version of RabbitMQ do the distro maintainers learn Erlang/OTP to patch the bug?

IceWreck• 4 years ago

Alma/Rocky are just rebuilds of RHEL. And afaik, Red Hat does employ people to do that.

jabl• 4 years ago

And, the package selection in RHEL is much smaller than in, say, Debian or Ubuntu [1]. There are the popular EPEL repositories with lots of extra packages, but these are community-maintained on a best-effort basis.

[1] And I think the Ubuntu 'guarantees' only apply to the 'main' repository, not the MUCH larger 'universe'.

qwerty4561272• 4 years ago

Obviously stability is the most important factor for the people running nuclear experiments so they choose RHEL. Nevertheless, if I were them, I would consider trying Nix or Guix to potrntially turn the configs/builds into comparably strict math and making these reproducible.

acatton3• 4 years ago

Config can also be made reproducible with VM images, containers, home-built packages, configuration-management tools (ansible, puppet, …) or a mix of all or some of them.

Nix/Guix is no silverbullet.

amelius2• 4 years ago

How do you grow your system, or do you suggest that you build your system as a bunch of microservices running inside containers?

tremon• 4 years ago

Personally, I think the chase for silver bullets does more harm than good. I'd rather work with multiple configuration management tools that do one thing well, than to deal with new contenders that seem to want to do it all and spend little effort in being interoperable with other needs outside their use case.

As a single example: Guix does not support setting capabilities on binaries in the store; if you want to set CAP_NET_ADMIN on ping, or you want some service to run with CAP_NET_BIND_SERVICE, you're stuck. There is no way to make it happen inside Guix, so you're left with very ugly manual hacks (mount --bind /gnu/store /mnt ; setcap...; umount /mnt). Similarly, neither Nix nor Guix can be used as a deployment tool only, since they do not preserve post-deployment configuration changes (to the point that GuixSD even deletes user accounts if they're not in the system configuration).

xorcist• 4 years ago

Alma produces a rather snazzy script to migrate from CentOS which works really well.

I'm told it also works with Rocky, but I haven't tried it.

bclemens5• 4 years ago

Rocky Linux has migration scripts available at https://github.com/rocky-linux/rocky-tools/tree/main/migrate...

An easy shortcut though, in case you have to hand type it out somewhere, is https://rockylinux.org/migrate2rocky.sh

jonathanspw2• 4 years ago

Pretty sure he was talking about ELevate to go from EL7 to Alma 8.

fancyfredbot• 4 years ago

What happened to scientific Linux? It's a bit odd that it seems to be being discontinued rather than rebased on Alma? Do they no longer believe scientific Linux adds value?

brnt• 4 years ago

It had been discontinued a while ago, in favour of a flavour of CentOS (at the time CentOS 7 was released, instead of Scientific Linux 7).

jl64• 4 years ago

I’m out of the loop on RHEL forks. Can someone explain why it is important to be binary compatible with RHEL? Is there a lot of software whose binaries will only run on RHEL? Or is this more a case of wanting something free that can still make use of RHEL-specific skills and knowledge?

sllabres• 4 years ago

As soon as you use something like SAP or Oracle the number of offically _supported_ distributions quickly decreases. As example SAP HANA: RHEL and SLES or for example Oracle Database: RHEL, OEL (based on RHEL AFAIK) and again SLES.

So it is not only a question if it would theoretically/technically fail on another platform.

jl64• 4 years ago

I don’t imagine SAP certification would extend to a community-supported fork of RHEL even if it was a 1:1 clone.

Symmetry2• 4 years ago

When I was working for the US government we had to obey rules against shipping "freeware," which RHEL wasn't because we payed for it. But it was cheaper and easier to do most of our development on CentOS systems and because the software was the same we weren't worried about compatibility issues.

mistrial92• 4 years ago

we had to obey rules against shipping "freeware"

I am in the USA but this is new to me.. can you say more or better, links? regarding science applications, specifically...

e40• 4 years ago

I think the replies to you missed the main point: standing on the shoulders of giants. RH has a LOT of resources and being a fork of it means you get all that person power for free.

hsbauauvhabzb4• 4 years ago

I would prefer to use RedHat based distros but their lack of official non free repos annoy me - I don’t want to add a community repo, I want packages maintained by trusted core engineers. Sure the community repos probably have that, but I last I checked I couldn’t find any assurance on the security of the community repos. SBOMs don’t mean shit if you’re installing stuff from random no-assurance locations.

ilyt3• 4 years ago

Then you want Debian. We've had FAR less problems with Debians than anything from Red Hat stable

MisterTea• 4 years ago

I've used a bunch of Linux distros over the years and always find myself installing Debian when I just want a desktop.

GrinningFool• 4 years ago

Aren't those assurances what you get by paying red hat?

hsbauauvhabzb4• 4 years ago

No, paying RedHat gets you support, not an official proprietary blob repo

jacooper3• 4 years ago

And after all of that, dnf auto update still doesn't support auto restarts for kernel updates, amazing.

Also no in-place upgrades since for ever.

vbezhenar3• 4 years ago

I'm afraid to even update ordinary packages on a running system. People who want to update kernel on the fly are just reckless!

jacooper3• 4 years ago

Since everything is already running as a container, it really doesn't matter

warinukraine2• 4 years ago

Not a linux expert.

Skimming wikipedia it seems the main selling point of both Rocky and Alma is that they're binary-compatible with red hat, right? Could some experts clarify: if you want to be binary-compatible with red hat, why not just use red had?

zamalek• 4 years ago

RHEL costs money.

warinukraine2• 4 years ago

Yikes. Why do people use it exactly? (I use debian, that's all I know)

jeffbee2• 4 years ago

I bet these guys have a lot of unusual legacy requirements, like wanting Motif and remote X11/Xdm to keep working, which takes Ubuntu out of the competition.

egberts1• 4 years ago

Absolutely love the integrated signing of containers (Sigstore technology).

If more distros support this, then our life would be far less stressful.

https://www.sigstore.dev/

Fermilab/CERN recommendation for Linux distribution

Related Stories