CERN migrates to open-source technologies

https://home.cern/news/news/computing/migrating-open-source-technologies

by tdhttt • 7 years ago

498 141 7 years ago

I think this was posted at the time. It's worth saying that CERN have been massive open source users for decades

pletnes• 7 years ago

It’s also worth noting that CERN has been running (not just using!) many open source projects since way back. Geant, madx, root and cling come to mind.

raxxorrax• 7 years ago

I suspect this is specifically targeting Windows. I think this would be a good idea for anyone really. I don't know whose or which decision brought MS into this mess, but this isn't a sustainable OS for the future.

Our company still relies on it, but any new piece of software has to be platform independent. Licensing income probably skyrocketed on MS end, but I don't think they made any friends with W10.

Maybe MS is correct that W10 will be the last Windows. But maybe not because of rolling releases.

Anyway, good news and the correct strategy in my opinion.

zucker42• 7 years ago

As long as schools use Windows, it won't die.

madez7• 7 years ago

Yes, this is why societies must take care about which hardware and software they use in schools. In Germany, some schools opted for Apple hardware and software, which is equally bad.

jakear• 7 years ago

Many schools are moving to chrome books.

But enterprise is the bigger deal anyways.

dragonwriter• 7 years ago

As long as schools use Windows, it won't die.

Kind of funny in this context, since the specific impetus here is Microsoft narrowing the applicability of academic licensing.

cwyers5• 7 years ago

Businesses aren't using Windows because schools use it (I think Chromebooks and iPads are more popular among schools now than Windows machines because children can trash a PC easily). They use it because of a long tail of software applications that only run on Windows, they use it because of stuff like Group Policy and Active Directory, they use it because of a lot of things that Linux doesn't do as well.

dukwon7• 7 years ago

I suspect this is specifically targeting Windows.

It's not: it's things like Active Directory, Skype4B, Office, Exchange etc. Desktop OS's aren't covered by MAlt.

josephagoss• 7 years ago

Does this mean that no scientist at CERN can use applications like Mathematica? Or is this just focusing on essential foundation software like operating systems (moving away from Windows)

lokimedes2• 7 years ago

While some rely on Mathematica most at Cern uses ROOT for data science. The impact on the science program is more likely to come from ditching Oracle databases and LabView... That said, this is an IT department initiative, users also have access to software from their home institutes.

deven88• 7 years ago

Is there any reliable and good opensource alternative of LabView ?

dguest• 7 years ago

There's a pretty sharp migration away from ROOT to numpy / pandas these days (at least in the younger generation).

akubera• 7 years ago

The latter. It's about replacing things like Active Directory, Exchange, Skype, etc. None of the projects relate to the actual science, except maybe Visual Studio licenses? But I know of nobody that uses ROOT on Windows with VS.

nxpnsv• 7 years ago

The number one commercial tool used is powerpoint...

linker3000• 7 years ago

Is there a public-facing list of project activities?

Apofis• 7 years ago

E-mail and ip telephone migration to start... doesn't seem like a big deal but they have been with Microsoft for 20 years.

lokimedes2• 7 years ago

When I worked there, they had 10k+ active user accounts and a wealth of systems integrated into Microsoft active directory. The nature of CERN also means that multiple institutes around the world will have to adapt their systems when CERN changes.

lowlevel• 7 years ago

My gut says this is related to forced updates and unwanted interface changes in current Microsoft products. It's been super annoying where I work, and staff are pretty upset.

toomuchtodo• 7 years ago

It was because CERN lost their academic discount for MS licensing.

https://www.geekwire.com/2019/microsoft-mum-cerns-planned-mi...

addicted• 7 years ago

This should be an HBS use case.

In a quest for money they didn't need and likely were not gonna get much of anyways, by removing perks MS currently offered academic institutions, they've risked teh creation of a much greater threat that can undermine their entire business. CERN is an organization which could truly help create a set of alternative tools to MS products.

Talk about killing the golden goose.

cwyers5• 7 years ago

I mean, they list a pretty compelling reason from their point of view: Microsoft changed their licensing terms from academic to business licenses, which are more expensive.

I suspect they will discover over time that the fully loaded costs of switching are higher than the costs of the licenses, unless this is just a way of trying to leverage MS into giving them academic licenses again.

perlgeek• 7 years ago

I suspect they will discover over time that the fully loaded costs of switching are higher than the costs of the licenses

I suspect the switching costs will be higher in the short term for sure, but this also seems to be a strategic move.

nullspace• 7 years ago

They say that the licensing costs increased by "by more than a factor of ten".

Given that figure, do you really think that one-time project implementation and switching costs amortized over the next X years + ongoing support would really come to more than 10x? How do you make such a strong claim that easily, any prior examples?

Not saying you're obviously wrong or that CERN is obviously right, but this seems to be a much harder call (and intuitively feels right in fact), given the number.

eitland3• 7 years ago

I suspect they will discover over time that the fully loaded costs of switching are higher than the costs of the licenses,

Can't speak for large organizations but some of the more well functioning places I have worked on has been running without much Microsoft services.

Also for many of us sysadmins in many cases we would say we would possibly save more from not having to deal with the hassle that is Windows than from just the licenses.

I mean: having to run and patch Apache httpd or other standard technologies on Windows is such a major hassle compared to Ubuntu or or any other mainstream commercial distro.

You can of course just use IIS, but then you have to deal with the hassle that is IIS.

This is a bit tongue in cheek, but the idea that Windows is somehow easier for everything is a bit unnuanced I think.

C1sc0cat4• 7 years ago

But CERN is academic what do you think all those people doing PHD's

Beltiras3• 7 years ago

I'm waiting for some of these documents being made public. I am working for a government institution that is making much of the same moves. Would be nice to see what an immense IT undertaking like this comes up with.

Boulth• 7 years ago

Are your org's documents public? I'd be excited to read about the practical effects of such deployment (from CERN too).

Beltiras3• 7 years ago

Ours is a more ad-hoc process than that. I work for Statistics Iceland. We are not changing OS or productivity suite (yet at least). Those that want Linux on their laptops get Ubuntu. We deploy Libreoffice alongside Office. Users using other statistical suites are encouraged to use R. On the infrastructure side we are making more use of Samba (one new DC, next up is migrating fileservers from old WServers over to Samba on Linux), just replaced Exchange with Kopano. We are an org of about 120-150 people/workstations.

williamxd3• 7 years ago

That moment when left-pad creates a black hole on earth

MayeulC• 7 years ago

Matrix would be a good fit for most research institutions, allowing cooperation trough federation. Heck, even Mastodon could be interesting, and ActivityPub as a transport layer for publications.

However, one issue is that small institutions cannot afford to self-host everything, or make the necessary adjustments themselves. I wonder if CERN and other big institutions could perform some heavy lifting (and maybe provide some shared services, hosting servers, etc.) without necessarily centralizing everything like it is done nowadays.

secfirstmd• 7 years ago

I'm a big Matrix fan but am curious about what options they have for integrating and managing it in a typical enterprise environment. For example:

-what happens for stuff like user management with Active Directory etc? -what happens for things like FOIA requirements considering it is e2e encrypted?

somepig• 7 years ago

e2ee can be turned off client side (and is off by default until ux improves), and while I don't think it's possible with the stock home server, it seems entirely possible to patch the home server to reject encrypt requests, and only federate unencrypted rooms

for directory service, LDAP and web SSO are both supported

roblabla• 7 years ago

Matrix doesn't E2E encrypt everything, only certain rooms. It's not currently possible, but I suppose there's no technical roadblock to making a server prevent access to E2E encrypted rooms to meet FOIA requirements.

As for user management, synapse has identity servers and password providers to form a complete authentication solution. mxisd[0] is a service that uses both to offer LDAP authentication, although it seems dead with no real replacement.

[0] https://github.com/kamax-matrix/mxisd

mxuribe• 7 years ago

Agreed on the matrix comment. In fact the government of France thinks so too! https://matrix.org/blog/2018/04/26/matrix-and-riot-confirmed...

kazinator• 7 years ago

That's kind of ironic. I was doing paid contract work in 1994 (on Linux!) that involved customizing the open source CERN httpd server (before Apache existed, or loadable modules in a web server).

sytse• 7 years ago

It is awesome to see how CERN is supporting open source. They have been long time users of our open core GitLab with 12,000 users https://about.gitlab.com/customers/cern/

jobvandervoort• 7 years ago

I visited CERN once in my role at GitLab. The scale of CERN and the level of people there is really impressive.

Many teams had implemented very complicated pipelines doing all sorts of things. Including using GitLab to design, validate and eventually produce hardware that was used in ATLAS.

panpanna2• 7 years ago

CERN had a major role in making kicad what it is today. they implemented tons of complex stuff that you normally only find in really expensive CAD tools.

Hope they can repeat that with a few more FOSS projects!

sigstoat• 7 years ago

and you can donate to KiCad development via CERN: https://cernandsocietyfoundation.cern/projects/kicad-develop...

turbinerneiter2• 7 years ago

Aisler.net, a German boardhouse, makes a donation to kicad for every kicad designed pcb you order there and it's my favourite way to pay for open source.

They literally get paid the moment value is created from using their software. It's beautiful.

xvilka• 7 years ago

Would be nice to push up also the other parts of the physical design - LibreCAD[1] and FreeCAD[2], Yosys[3] and Symbiflow[4], Chisel/FIRRTL[5], and many others.

[1] https://librecad.org/

[2] https://www.freecadweb.org/

[3] http://www.clifford.at/yosys/

[4] https://symbiflow.github.io/

[5] https://chisel.eecs.berkeley.edu/

MisterTea• 7 years ago

CERN has really done wonders to revive the project and get it organized again. Before that, the 2012 build became the last sane build from the project team for a few years. The project appeared to have lost steam and began to fragment. You had to download "latest" windows builds from some random server. Linux distributions stopped packaging at the 2012 build making it appear as if kicad was dead. This forced you to find a 3rd party package or build from source. Then there were a few forks to add features started by random individuals which were always unstable. Component names were renamed with stupid and confusing names to the point where I stopped using it. It almost looked like the project was dead at that point and this was 2014/15.

I recently picked it back up to investigate and was pleasantly surprised. It felt like the team dearly missed the sanity of 2012 version, continued from there and modernized it. Thank you CERN.

Polyisoprene• 7 years ago

Since people don't read the article:

"The Microsoft Alternatives project (MAlt) started a year ago to mitigate anticipated software license fee increases. MAlt’s objective is to put us back in control using open software."

"A prime example is that CERN has enjoyed special conditions for the use of Microsoft products for the last 20 years, by virtue of its status as an “academic institution”. However, recently, the company has decided to revoke CERN’s academic status, a measure that took effect at the end of the previous contract in March 2019, replaced by a new contract based on user numbers, increasing the license costs by more than a factor of ten. Although CERN has negotiated a ramp-up profile over ten years to give the necessary time to adapt, such costs are not sustainable."

ethbro2• 7 years ago

Ouch. Foot meet gun.

Nice to see some of the old guard is still alive and well within the Microsoft walls.

C1sc0cat4• 7 years ago

Really CERN does not qualify !

budu3• 7 years ago

Wow. Heads will surely roll once this reaches Satya's desk.

passer_byer• 7 years ago

...or the person gets promoted.

I previously worked for an enterprise analytics software provider whose success hinged on low-cost license agreements to schools and universities. Once the company became more successful commercially, it took the same tact--substantially raising these fees in search of profitability. Today, the firm is under serious pressure from open source analytical libraries. The point being, what looks to be optimal in the short run may not be optimal in the long run, as future consequences could be anticipated but short term revenue growth was too tempting.

hanniabu• 7 years ago

I'm surprised Gates would be for this

Animats• 7 years ago

"Ten years"? Come on.

gen3• 7 years ago

Also see the comments last time this was posted: https://news.ycombinator.com/item?id=20166070

and what I said last time:

I'm honestly pretty happy about this. I'm hoping that by aiming to replace the commercial products they use with opensource alternatives, the alternatives leave with a better polish and user experience. I also see CERN as an institution that is willing to hire the devs needed to maintain / support a project.

shmerl• 7 years ago

Are they using Linux instead of Windows already?

dukwon7• 7 years ago

There's a mix of linux, windows and mac OS machines, depending on the use-case and/or user-preference. This won't change.

whatshisface• 7 years ago

Public funding of open source software is one of the most direct ways that government spending can grow the supply side of the economy. Taxpayers will directly benefit from the public domain technologies developed to replace the contracts CERN is moving away from. Software and science share an important economic trait, that once you produce them you can maximize their total impact by spreading them around as much as possible. That's an advantageous situation for public funding without IP restrictions, which is the model of academia.

remir• 7 years ago

Migrating away from Microsoft won't be easy. Despite high licensing costs, Windows, AD and Exchange are still great solutions with millions of people familiar with them, good documentation and support.

I agree that only public funding and massive donations would allow the development of true alternative to these products.

The building blocks are already there: desktop Linux, openLDAP, FreeRADIUS, LibreOffice, etc.

What is needed is lots of refinements, improvements and UI/UX polish to have a package that small shops to Fortune 500 companies could deploy.

giancarlostoro• 7 years ago

You dont need to get rid of Windows to support Open Source software though. My favorite Open Source tools work no matter what OS I am using be it Mac, Windows or Linux.

lowdose• 7 years ago

I think everybody should move away from Windows & other software assets. In 2019 it is crazy to make end users responsible for the security of the OS. I have for example never questioned the security of my iPhone.

A child that has a chromebook will never understand why you need a virus checker, firewall, that the data really is gone if a ssd breaks or have to install a special program to make a fysical print. The average training time to make this migration from MS Office to Google Docs for employees at Forbes 500 companies is 1 hour.

Chrome OS is such an undervalued solution for the end user in a traditional business.

In tech we all use GSuite. On stackshare.io a whopping 17.200 companies use G-suite in their tech stack. Microsoft Office 173.

You can't get hired if you answer MS Office as your main document driver in an interview. This discussion is not even a thing that gets discussed at a sane technology company. G Suite is awesome and it just works.

Kids use Google Docs as a chat app. A traditional document program is a typewriter and a google doc is a laptop with internet connection.

G Suite API's create so much automation option plug&play out of the box that 95% of your automation can be arranged with SAAS.

Both Zapier & IFTT supply a polished UI, where anybody can connect G Suite with 600 other SAAS API providers. Nobody outside technology has ever started a conversation about these two.

https://stackshare.io/stackups/g-suite-vs-microsoft-office-3...

Nightshaxx• 7 years ago

Counter point: the more the government gets into funding OSS, they will essentially become a competitor to other software companies. Now the government both has the power and incentive to hinder private companies with rules and regulations.

I think that if the government needs OSS software for actual use by the government, then that's fine. Don't make the government become a competitor to private industry. Look how that turned out with Uber vs. Taxis.

OSS is just like anything in the free market. If people and companies want it that much, they will spend time and energy on it.

usefulcat• 7 years ago

Now the government both has the power and incentive to hinder private companies with rules and regulations.

What incentive? In the scenario you describe, I don't see an obvious mechanism for the government to receive additional tax revenue as a result of funding OSS, and that's the usual mechanism for such an incentive.

I mean, I guess you could argue that tax revenues will tend to increase as the economy grows, but that seems like a case where the interests of the government are aligned with the interests of the nation(s) as a whole, which is generally considered a good thing.

cma• 7 years ago

The same is true of the government using proprietary software thus funding it against open source. The work can be outsourced to open source companies, etc. as well, just like it effectively is with proprietary.

WWLink• 7 years ago

As long as they don't forbid companies from using OSS inside of (or to produce) a commercial product, I don't really see the problem? It'd actually give companies a competitive advantage.

Speaking of that, it's absurd how many private companies benefit from Linux! I bet only a tiny percentage of those companies contribute back into the open source projects they benefit from.

crb• 7 years ago

While this post is referring to desktop/server software, CERN are also using open source platforms in some of their experiments: they recently re-performed some of the 2012 analysis that led to the discovery of the Higgs boson, on top of Kubernetes.

Adam and I spoke to a computer scientist and two physicists from CERN for our podcast, and you might like to listen if you like physics or software.

https://kubernetespodcast.com/episode/062-cern/

ridaj• 7 years ago

Good idea but

Needless to say, isolated initiatives will waste effort and resources.

If the centralized procurement approach is what put everyone in the current mess, what indication is there that a centralized approach to open source won't produce the same issues?

I personally wonder why commit to open source in a centralized manner vs commit to interoperable standards (but not the specific tools used to speak to the standard).

cat199• 7 years ago

jury probably still out, and no direct insight, but the overall incentives/philosphy driving acquisition for each could theoretically be drastically different (e.g. optimizing for openness, community, xyz instead of vendor size, ability to negotiate discounts, etc)

lone_haxx0r2• 7 years ago

Wait. Why the fuck is there a cern TLD? Are ICANN naming TLDs after specific organizations now?

dukwon7• 7 years ago

Basically yes: https://icannwiki.org/Brand_TLD

I agree with the move away from .ch, since it's an international organisation, but .int would have been the correct choice IMO

lone_haxx0r2• 7 years ago

I expect nothing, and I'm still let down.

nullify88• 7 years ago

I came across a white paper years ago (2012/2013 ish) on how they used Icinga (An early but rapidly growing Nagios fork) and mod-gearman to monitor their ATLAS computing farm. At the time, as a newly appointed sysadmin, I was looking in to possible solutions that could scale well myself and their usage was very inspiring. Eventually deploying something similar where I'm still working now.

https://cds.cern.ch/record/1455464/files/ATL-DAQ-PROC-2012-0...

user59944612• 7 years ago

CERN is not a university and has no student, it's entirely justifiable for Microsoft to not give them a university discount.

Universities have thousands of students that do not generate any revenue. The standard licensing model per person or per computer totally breaks down in that case, asking for a huge bill based only on the sheer amount of users. Education discounts are adjusting for that, large user bases with no money to pay.

That being said. The CERN is a government entity. They should argue to be given government discount.

dukwon7• 7 years ago

Universities have thousands of students that do not generate any revenue. The standard licensing model per person or per computer totally breaks down in that case, asking for a huge bill based only on the sheer amount of users.

There are thousands of students, postdocs and academics with CERN computing accounts (way outnumbering employees). It's not like they're generating revenue either.

They should argue to be given government discount.

They did, IIRC, but their status as an IGO didn't cut it with Microsoft.

user59944612• 7 years ago

IGO: https://en.wikipedia.org/wiki/Intergovernmental_organization

Honestly, it looks like the terms and licensing are being dictated by some dude(s) from Microsoft in the US, based on US norms. The way academy/university/research is organized and operated in Europe is fairly different. Bet Microsoft doesn't mind the added money either.

If the CERN were playing by US rules, they would be suing Microsoft to be recognized as a government or academic entity.

sh-run• 7 years ago

I'd like to see more done in the open networking space. Cumulus Linux was talked about a ton just a few years ago, but it seems like I haven't heard about it as much lately. Probably because Juniper and Cisco have (mostly) managed to catch up on the automation front.

I couldn't find anything concrete on the CERN website, but in their virtual DC tour I noticed HP Procurves as top-of-rack switches. I was really hoping to see whitebox switches.

duxup• 7 years ago

I worked support some hardware CERN used once. Fun guys to talk with / work with.

They call me up one day to note that one of their devices I supported had an accident and they were concerned they didn't know why. So they sent me photos.

This device had rows of modular cards installed in it. In the center of the device with two cards pulled out you could see that something had burred and even melted some of the surrounding cards. But it didn't look like any given card had failed as much as there was some sort of really hot fire ... that had been in the air between the cards or something. Now keep in mind this was just what a handful of photos looked like, so who really knows. Makes no sense that there was something floating in the air between the cards hot enough to do that thing ... but that is what it looked like.

Anyway it was like a good 100k+ in hardware burned up, possibly MUCH more as the full chassis held a lot more than that. So I promise them a new chassis and such and tell them to pack it up nicely and we will have a courier come and get it and send it to our QA team. The CERN guys promised not to expose the equipment to anymore micro black holes ;)

The process to send stuff to the QA team in strange situations like this was a painful series of steps. The QA team was BRUTAL about process (even if they never followed it themselves...). They also were a real pain to even email, but that was part of the process. Amusingly enough when I sent them the photos and explained it was CERN even the QA ultra dry guys cracked some good X-files references ;)

Still wonder what the hell happened to that equipment.

dgudkov• 7 years ago

...increasing the license costs by more than a factor of ten

When I hear stories like that I always wonder what was the thought process of the sales people that managed that account. Probably something like that:

- Hey, do you remember our old customer CERN, a world-famous scientific non-profit organization that pushes the boundaries of human knowledge about the universe?

- Yeah, what about them?

- Let's charge them 10 times more for our software licenses

- Can't see anything wrong about it, go ahead

cernguy3• 7 years ago

Disclaimer: I worked at CERN in IT for 5 years as staff.

CERN has never been a Microsoft organisation excepted in the head of management that was pushing for Microsoft solutions everywhere without success.

- Scientific computation are done at CERN under Linux with the Root framework.

- Most (all?) scientists uses OSx or Linux.

- All computing clusters runs SC Linux or centos.

- Most internal softwares: indico, EDH, landb, and other are running under Linux and are web based.

- All DBs are OSS or oracle

- All storages system are home made (EOS, Castor) or ceph based and run under Linux.

- Data distribution is home made and based on a framework named xrootd under Linux.

- Software distribution is also Linux based and run as a fuse module (cernVMFS)

- Most systems services are UNIX C++ and Java for the control part.

- CERN uses Openstack for virtualisation After the management pushed for Hyper-V and failed miserably.

- Management pushed for SharePoint for years before the entire website switched to PHP and Drupal.

There si not real "Microsoft" at CERN excepted AD and phones. It is however a study case of bad management decisions in IT.

chipperyman573• 7 years ago

Why was management pushing for it so hard? Better (existent) salespeople for microsoft vs OSS? Or were there actual reasons they thought microsoft would be better

cernguy3• 7 years ago

Why was management pushing for it so hard? Better (existent) salespeople for microsoft vs OSS?

It's not clear to me. But it seems to have been part of a vision to "servicify" everything. Where everything should be a service "maintained" / "paid" but never "developed" to reduce costs.

Something that backfired beautifully

l0b0• 7 years ago

I'm not proud of it, but I worked on SharePoint at CERN IT. Some examples of how dysfunctional it was:

- When asking about version control I was told that if I really wanted to use it I could have Visual SourceSafe 6.0 or something from way back in the nineties. It was basically CVS 0.1 alpha with a GUI, and I ended up learning Git instead.

- I made sure all my changes worked on Firefox, because that's what I knew the physicists were using. My boss wanted me to support IE only, and after a heated discussion the quote which always haunted me was "We don't care about the physicists!"

- A colleague lamented about having spent about a year developing some uber-flexible interface in SharePoint which would've been an order of magnitude simpler in other content management systems.

The rest of CERN is completely different, as indicated by cernguy.

I left a year into a two-year contract, and I'm happy to say I've not worked on a Microsoft platform ever since!

cernguy3• 7 years ago

The rest of CERN is completely different, as indicated by cernguy

Yes, that is also a thing at CERN.

Many other teams re-develop things internally (Linux based) because they could not trust or rely on IT management to do the right things.

That had a very perveted effect of internal duplication across teams and experiments.

wilsonpie• 7 years ago

Well, now they are conCERNed

anthony_barker• 7 years ago

How widespread is oracle? What features are they using that Postgresql(or commercial derivative) couldn't handle?

rkeene2• 7 years ago

cernVMFS sounds like AppFS [0] with fewer features. Some of those features, like writability or setuid files, might be useful for CERN users.

[0] http://AppFS.net/

CERN migrates to open-source technologies

Related Stories