Red Hat and CentOS systems aren’t booting due to BootHole patches

https://arstechnica.com/gadgets/2020/07/red-hat-and-centos-systems-arent-booting-due-to-boothole-patches/

by thg • 6 years ago

113 60 6 years ago

fomine3• 6 years ago

It seems that it's really critical issue. How does this patch pass Red Hat's test?

chousuke2• 6 years ago

Bugs get through testing quite frequently, and this one was probably somewhat fast-tracked because it's a security fix; it happens.

FWIW, I tested upgrading a few (test) CentOS virtual machines at work to see if I can trigger this bug, but they worked fine, so perhaps the bug only triggers with a configuration they happen to have not tested.

cesarb4• 6 years ago

I tested upgrading a few (test) CentOS virtual machines at work

Were the virtual machines using EFI? Most virtual machines I've seen boot through legacy BIOS, not EFI.

fideloper• 6 years ago

Ubuntu had this as well, but got a release out quickly so it seems to have not been too huge an issue.

Bug report: https://bugs.launchpad.net/cloud-init/+bug/1877491

Description/remediation: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2Secu...

gruez• 6 years ago

Currently, we populate the debconf database variable grub-pc/install_devices by checking to see if a device is present in a hardcoded list [1] of directories:

Is there a reason why this patch was silently applied? For something as risky as breaking the boot process, you'd think you want user confirmation before proceeding. It can obviously be done, eg. https://d11a6trkgmumsb.cloudfront.net/original/3X/f/5/f55e36.... Also, recovering from this might be easy if you're technically inclined, but it could be worse if you have FDE enabled with the boot keys sealed in TPM. Changing the boot loader or the secure boot settings in that case might lead to the TPM refusing to release the disk encryption keys, which could lead to permanent loss of data.

cvwilliams• 6 years ago

As usual, there are prophetic warnings from Linus:

https://yarchive.net/comp/linux/efi.html

BIOS should be simple, because it is buggy anyway. Handing over to a bootloader in the MBR is all that a BIOS should do. Now one is at the mercy of NVRAM, grub2 and loads of gratuitous complexity.

bzb3• 6 years ago

"New software will eventually contain bugs", and then bringing it up 14 years later, seems to bring nothing to the table.

chris_wot• 6 years ago

Hence the “prophetic” bit.

paol• 6 years ago

EFI really was a step in completely the wrong direction. Massive complexity & feature set for no good reason.

Legacy BIOS had to go, it was designed at the time of 8086 and DOS and was really out of step with modern HW and OS needs. The replacement could have been even simpler though, now that OSes really want to take over everything themselves and no longer lean on the BIOS that way that DOS used to. Instead EFI created a monster and now we're stuck with it.

vbezhenar• 6 years ago

BIOS is a good idea. It should have been expanded. For example I should be able to download Nvidia driver, install it into BIOS and then talk to it via standard interface (something like Vulkan). Now every operating system can just utilize that interface and use well tested driver from manufacturer, rather than ask manufacturer to write that driver for every operating system.

6c696e7578• 6 years ago

Not at all like systemd then. I had hoped by now we'd be using something that took the arguably good qualities (simple unit files) and none of the complexity (why does it implement DNS?) and we could have the better subset.

It will continue to bite until enough people petition it's replacement.

Don't get me wrong, some of the features are good, but the kitchen sink is not.

guerby• 6 years ago

I hope RISC-V will get booting right.

https://linuxplumbersconf.org/event/4/contributions/381/atta...

https://www.phoronix.com/scan.php?page=news_item&px=UEFI-RIS...

Early discussions (cannot find a trace online, from my mail archives):

From: ron minnich Subject: Re: [sw-dev] SBI extension proposal v2 Date: Sat, 10 Nov 2018 08:46:07 -0800

At Google and other places, we've been struggling now for years with overly complex firmware that is implemented incorrectly, enabling exploits and other bad things. The list of things vendors get wrong in firmware, both enabling exploits and enabling others to enable exploits, is long and it continues to this day. There is an unbelievable amount of money out there all involving firmware exploits, very little of it involving nice people.

I'm currently working on deleting all use of the x86 version of M mode, i.e. SMM. There are many proposals out there for deleting SMM from the architecture. I've also shown at a talk in 2017 how we could redirect SMM interrupts back into the kernel. We're also removing all use of callbacks into UEFI on x86. We're almost there.

Which is why I'm a bit unhappy to see this (to me) cancerous growth in proposals for M- mode code. PPP in firmware? Really? multiple serial devices? really? We've been here before, in the 1970s, with something called the BIOS. If you're not familiar with it, go take a look, or you can take my word for it that these proposals implement that idea. We spent over 20 years freeing ourselves from it on x86. Why go back to a 50 year old model on a CPU designed to be in use for 50 years?

My early understanding of M mode was that it was an Alpha PALCode like thing, enabling access to resources that were behind a privilege wall. I did not like it that much, but I was OK: it was very limited in function, and the kernel could replace it, or at least measure it. I also accept that every cpu vendor uses m mode like things (e.g. ARM TF) for reasonable purposes and also (let's be honest here) for dealing with chipset mistakes. But that does not mean you need to recreate BIOS.

The SBI should be hard to add to, deliberately. It should be used only when there are no possible alternatives. It needs to be open source and held in common. It should be possible for a kernel to replace or at least measure it. And, further, there needs to be some work done on why you add to it, and why you don't, with bias against adding to it. This proposal works against those ideals, as it explicitly enables vendor-specific forks of the SBI. Sure, this can happen, but why make it so easy?

see https://github.com/riscv/riscv-sbi-doc/pull/12 for other thoughts.

Also, I've had discussions with some security folks in our firmware community about the fact that the PMP can be used in a way that the kernel can not measure the SBI, since SBI might read-protect itself. This is a real step backwards, FYI. Not sure if it can be changed at this point.

ron p.s. For interleaving debug and console output firmware, use the oldest trick in the book: ASCII is 7 bits. Since console out is 8 bits, reserve 128 values for console out, and 128 for debug stream, and if the debug stream needs 8 bit for some words, you know what to do. It's very easy and doesn't require that we add multiple UART support to SBI.

wizzwizz4• 6 years ago

XSFI, the Extremely Simple Firmware Interface (currently in its second draft) is a good solution. https://www.alm.website/misc/specs/xsfi

It could be implemented in UEFI, and bundled with disks as a compatibility layer, for machines that support UEFI but not XSFI.

rhn_mk1• 6 years ago

Few will remember but Intel came up with something simpler: Simple Firmware Interface.

Wikipedia only mentions the mobile platform, but it was also what the card-based Xeon Phi was using (own research).

twic• 6 years ago

This is part of the mess that Oxide Computer wants to clean up, right? Are they going to use EFI at all? Anyone from Oxide around?

badrabbit2• 6 years ago

I really do miss the simplicity of grub 0.97,keeping it as long as I can on my older boxes.

kd9132• 6 years ago

Grub simple?

If you want simple use systemd-boot. It's a hell of a lot leaner than grub ever was. The config is sane and doesn't require a billion additional modules to be installed.

It should be the default at this point, especially after all of this fiasco.

trasz• 6 years ago

How is EFI responsible for a security bug in open source bootloader that's not even EFI-specific?

zabardasth• 6 years ago

It's a lot simpler on embedded devices. U-Boot for example is extremely simple and much more robust than GRUB or LILO.

CSDude• 6 years ago

Like the biggest attack vector was BIOS, they wanted to secure that with UEFI, it was nothing but headaches with my dual boot installs.

hnarn• 6 years ago

I’m not saying Linus isn’t extremely competent, much more so than I and probably anyone reading this will ever be, but in general terms I think anyone arguing against complexity in software design, while in a way “doing the Lord’s work” even in my own opinion, will never be wrong: in the cases where their prophecies never come true, the opinion will still be considered by most to be “correct in theory”, and if something does break down the line, the opinion can be referred to as prophetic.

So, while I agree that we should keep things simple and modular, it’s a thankless job trying to solve issues and being forced to add complexity. Nobody defends complexity, and maybe we shouldn’t, in order to stay on our toes, but defending simplicity is also about the safest thing you can do.

Jonnax2• 6 years ago

Is Secure Boot on Linux actually secure?

I remember reading it was like a signed loader and that's it. But I presume that's incorrect?

rst• 6 years ago

"Is it secure?" is an incomplete question -- it has to be "Is it secure against...?". And then the answers will depend on whatever threat model winds up in the dots.

That said, all secure boot even tries to assure is that the software that's booting is the same thing you thought you installed. If that is, say, a Linux distribution running a webapp which has problems, well... the boot mechanism can't save you from those.

Jonnax2• 6 years ago

Ah yes. The question I should have asked is: Is it more secure than not using it, and in what way?

Like for example, the Linux kernel isn't signed, right?

diffeomorphism• 6 years ago

I remember reading it was like a signed loader and that's it.

That is literally what secure boot is, nothing more:

https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_In...

cpncrunch4• 6 years ago

We have a centos 8 server that has been bricked, and there doesn't seem to be any solution right now. Downgrading the packages shows "lowest version already installed, cannot downgrade it.", and I can't manually install the old packages because there doesn't seem to be any way of getting them. Someone says to use http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/... but that seems to have the very latest packages and even after updating the affected ones it still fails to boot.

asicsp• 6 years ago

See if this helps: https://www.reddit.com/r/linux/comments/i1hlay/red_hat_and_c...

cpncrunch4• 6 years ago

It just links to the redhat info which doesn't work on centos 8.2 (error message is that those packages are already at lowest versions).

cesarb4• 6 years ago

Someone says to use http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/... but that seems to have the very latest packages

At least for me, that page seems to have both shim-x64-15-13.el8.x86_64.rpm from 2020-07-29 22:10 and the older shim-x64-15-11.el8.x86_64.rpm from 2020-05-07 19:53; the older one should work. Worst case, you could manually copy the shim executables from a working server to the EFI partition of the broken server (from what I have read at https://bugzilla.redhat.com/show_bug.cgi?id=1861977, in the RHEL/CentOS case it's the shim executable which is broken, so you don't have to do anything to the grub executables).

cpncrunch4• 6 years ago

Yes, I see I needed to look for the 81 version numbers. I have now installed those packages (and confirmed I have the older grubx64.efi in /boot/efi/EFI/). However it is still giving the same error on reboot.

I installed these rpms (which are the affected ones for my system):

grub2-common-2.02-81.el8.noarch.rpm grub2-efi-x64-2.02-81.el8.x86_64.rpm grub2-pc-2.02-81.el8.x86_64.rpm grub2-pc-modules-2.02-81.el8.noarch.rpm grub2-tools-2.02-81.el8.x86_64.rpm grub2-tools-efi-2.02-81.el8.x86_64.rpm grub2-tools-extra-2.02-81.el8.x86_64.rpm grub2-tools-minimal-2.02-81.el8.x86_64.rpm

Is anything else needed? I'm thinking the easiest solution is for me just to wait a few days and do a "yum update" in rescue mode once an update fix is available. Luckily this is a non-critical server.

CUViper• 6 years ago

I see both the old and new packages on mirrors.kernel.org:

https://mirrors.edge.kernel.org/centos/8.2.2004/BaseOS/x86_6...

ginko• 6 years ago

I still don't understand why this even needed to be fixed. Finding a way to circumvent UEFI DRM seems to be a good thing.

Wowfunhappy3• 6 years ago

Secure boot can usually be turned off if its unwanted. I agree its not necessary on every device—but if it is enabled, the OS should assume the user wants security at that level of the chain. It certainly shouldn't just circumvent it as a matter of course.

cesarb4• 6 years ago

Secure boot can usually be turned off if its unwanted.

Emphasis mine. The main objection to secure boot is that, some time in the future, it will be mandatory; that already is the case for Windows devices with ARM CPUs (https://www.softwarefreedom.org/blog/2012/jan/12/microsoft-c...).

fortran772• 6 years ago

That's the great thing about open source, though. You guys can fix it yourself. Me, I'm stuck on Windows 10.

overcast• 6 years ago

There's probably more people capable of fixing Windows 10 at Microsoft, then there are people capable of fixing bootloaders in Linux in the world. Just because something is open source, doesn't mean anyone can just jump in there and change the oil.

fortran772• 6 years ago

Exactly my point.

qalmakka• 6 years ago

I'm kind of out of the loop - is BootHole a UEFI or GRUB2 vulnerability?

enzanki_ars• 6 years ago

BootHole is a GRUB2 vulnerability that could allow a local attacker to bypass Secure Boot protections. The patches for RHEL/CentOS lead to failures in booting due to a crash in the UEFI loader for GRUB2.

noahbliss• 6 years ago

Hey all, just wanted to make you aware of the mortar project here: https://github.com/noahbliss/mortar

It takes a comprehensive approach rather than piecemeal like a lot of these patches, leveraging technology already in your system to build a conceptually airtight and fully audited system. Happy to get some of your opinions on it, constructive criticism, and pull requests!

nullc• 6 years ago

Number of users saved from attack by secure boot: 0 Number of systems bricked by fixes to secure boot vulnerabilities: Many

Sad to see TSA in software form.

benttoothpaste• 6 years ago

The most secure boot is the one that does not take place ;) So maybe there is a method in this madness?

cpncrunch4• 6 years ago

Fixes now available for RHEL, still waiting for CentOS.

Red Hat and CentOS systems aren’t booting due to BootHole patches

Related Stories