systemd 100% cpu hang? – Proxmox Support Forum

https://forum.proxmox.com/threads/systemd-100-cpu-hang.124767/

by lionkor • 3 years ago

180 141 3 years ago

forum.proxmox.com

lionkor9• 3 years ago

We just spent multiple hours in a team of multiple people debugging this issue. Systemd doesnt work. Checked all disks, fstab, recovery system, etc. and there was nothing clearly wrong.

Turns out something in proxmox (maybe a service?) doesn't understand daylight savings time (Dublin).

The only way to know was to google "proxmox systemd 100% cpu" and find above post.

Christ.

Edit: The fix, of course, is `ln -sf /usr/share/zoneinfo/Etc /etc/localtime`

Edit 2: Looks like that just unsets the timezone. It's too late for me to find the real fix, but you basically want to set the timezone to something else (like UTC).

gjvc3• 3 years ago

servers should always be UTC

LeoPanthera2• 3 years ago

It is entirely reasonable for a personal server to be set to the timezone it is physically located in.

berkle4455• 3 years ago

upvoting because i fiercely agree with you. servers should be UTC.

throw0101b8• 3 years ago

servers should always be UTC

All of the servers I run are located in Toronto, Canada: why do I care about UTC? What would having all of them being UTC get me?

A few jobs ago I took care of servers that were in: California, Ontario, Quebec, Switzerland, Singapore. UTC made sense there.

limaoscarjuliet• 3 years ago

They are UTC - the actual time on the system is measured in seconds since midnight Jan 1 1970 (https://en.wikipedia.org/wiki/Unix_time). This time is then converted to local time per the OS timezone setting. A lot of apps (e.g. databases) will also have their own time zone conversions based on context (e.g. per database connection settings).

As far as practical use of UTC vs local time: I am a developer that happens to work with customers directly. It is difficult enough to get customer to provide timestamps for issues they report in local time, I cannot imagine they would be able to convert to UTC.

Love the local time!

anonymousiam2• 3 years ago

Servers (and every other Unix/Linux box) ALWAYS use UTC. The time zone is only specified for the purpose of DISPLAYING the time in some locale where the system resides. The only things that should care about the time zone are those that generate output for human consumption (such as log entries), and this is usually automated.

For a system to not boot in a particular time zone is a serious bug, but you should not blame the problem on the time zone, you should blame the problem on whatever is broken in some critical piece of code that probably should not be using local time.

skrtskrt• 3 years ago

this position is getting a lot of flak, but I agree since we don't live in magic world where all software can handle time zones well.

I strongly prefer everything in UTC and changing time zone is an issue for only the vary top layer of the display/user interface/frontend whatever if absolutely needed - generally libraries designed for displaying have much more robust timezone handling than just random daemons and backend software.

akira2501• 3 years ago

pid 1 shouldn't choke because of a timezone issue.

jiggawatts• 3 years ago

We share this planet with people that don't use UTC. That's just the way it is. If some computers are set to time zones other than UTC, then non-UTC time zones must be correctly handled, even "servers".

There are terminal servers, for example, which are basically workstations. There are servers configured by other people for local time, and we may need to configure our own servers into its network.

There is server software out there that uses the host timezone. Heck, huge vendors do this regularly. This happens in log files, in schedulers, in notification emails, and so on. People want to see their log timestamps in their local time, they want to schedule in local time, and they want their emails to show events in their local time.

Yes, optimally, all times should always stored in UTC or UTC+offset formats, then displayed in local time only "at the end client". Generally, this is actually what happens. Windows, MacOS, UNIX, Linux, and Android all store the system clock as UTC and all system APIs use this.

So in some sense, we are all already using UTC on our servers. The time zone setting doesn't change the system clock to something other than UTC, it just tells user-mode software how the human users expect to see time formatted for display.

The problem is that server software especially is terrible at this. Just... bad. The worst offenders are text-based log files, which almost always get a formatted timestamp with an unspecified time zone. Could be UTC, could be the server's local time, could be Mars time, who knows?

In summary: don't blame system administrators for correctly configuring server settings. Blame the lazy software developers who can't be bothered to use ISO timestamps that specify time zones unambiguously, irrespective of the time zone setting.

Next thing you'll tell me to stop using space characters in file names...

m-p-3• 3 years ago

The server's clock should indeed be UTC, with the timezone translation done according to the user profile configuration.

CaliforniaKarl• 3 years ago

Don't you mean /usr/share/zoneinfo/Etc/UTC ? If I remember correctly, /usr/share/zoneinfo/Etc is a directory, and /etc/localtime should point to a zoneinfo file.

lionkor9• 3 years ago

Hm, not sure, already closed the ssh. But it did work for us.

The correct fix would be timedatectl, but as you can guess, that's broken together with systemd.

starkparker• 3 years ago

Thread is specific about linking to Etc. Also from the linked thread:

I've noticed systemd (PID=1) is running at 100% CPU utilization. That's not normal. I ran this command:

   strace -f -p 1

> The process was reporting continuously attempt to access /etc/localtime file. That's also unusual.

I managed to get the output I was receiving on the console while tracking systemd process.

    stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0
    stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0
    stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0

Maybe if it's endlessly failing to stat the target, any valid target might help.

somat2• 3 years ago

Are you unsetting your timezone, I hesitate to sound authoritative about because I really don't know, and have not found the correct man page yet. but usually you symlink /etc/localtime to the tzfile you want as your local time. what happens when you symlink it to a directory? in your case /usr/share/zoneinfo/Etc

My guess it that it effectively unset it, in your case that probably does nothing(system time is utc and dublin is in utc) what was it set to before?

edit: all I can find is localtime(5) and it only says that /etc/localtime is the local time zone file.

lionkor9• 3 years ago

Yes, it basically makes it useless, as far as I can tell. I don't know what the proper fix is, because timedatectl was broken.

kodah3• 3 years ago

Sorry this happened to you, these kind of problems tend to be vague and difficult to patch up.

I've advocated for over a decade as a systems engineer not to set system time to anything other than UTC. Not all languages or applications compensate for DST very well, and depending on your applications relationship with time it's a non-trivial problem to solve. What is far more trivial to do is to ship logs to a log server and let the UI of the log server translate time for the viewer of the logs. Almost every time I have argued with executives, managers, and SWEs without SE experience and lost something unexplainable and detrimental happens around DST.

Edit:

For the uninitiated I'll try to paint a clearer picture. All system time in Linux is tracked in seconds past Jan 1, 1970. The problem occurs in the translation between UTC and local time, which is handled by a library/package/module in your application. If your application is time sensitive and not looking for a literal time traveling event then things can get real weird, real fast. If there is a bug in that library/package/module things will also get real weird, real fast.

lionkor9• 3 years ago

Apps shouldnt use anything but UCT for their timekeeping, there is no reason to do anything in the timezone's time.

Setting the timezone should have no such effect.

joshspankit• 3 years ago

All system time in Linux is tracked in seconds past Jan 1, 1970.
If your application is time sensitive and not looking for a literal time traveling event...

One of my least-favourite facts is that these statements are misleading. Since Linux uses UTC and not TAI, it no longer tracks an absolute measurement of seconds since the unix epoch as it takes “leap seconds” in to account. As well, because of leap-seconds you can absolutely have time-travelling events.

https://developers.redhat.com/blog/2015/06/01/five-different...

cmurf• 3 years ago

This bug seems familiar. But I'm a little surprised the fix for this wasn't already in the version of systemd chosen for the release.

March 2022 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/19668...

March 2021 (Fedora) https://bugzilla.redhat.com/show_bug.cgi?id=1941335

systemd upstream fix https://github.com/systemd/systemd/pull/19075

AndyMcConachie• 3 years ago

Almost all software bugs come from managing state. Which is why hoarding state management in PID 0 is a dumb idea. I said this when systemd was gaining traction and I'm still saying it now.

When Debian switched to systemd in the default was when I began to seriously use OpenBSD and I've never regretted the decision.

bitwize• 3 years ago

Which is why hoarding state management in PID 0 is a dumb idea.

Because the Unix process API is unreliable and unsafe ( http://catern.com/process.html ), managing processes is difficult to impossible in a general way under Unix systems including Linux. Linux gained some features to help with this, but for the longest time, PID 1 was the only process that could manage its child processes in a way that wasn't fraught with race conditions -- meaning systemd or something like it was the only way to have safe, effective process management because it tracked all that state in PID 1.

Blame Unix being broken, not systemd, for the problems systemd solves.

kaba02• 3 years ago

Because the litany of shittily written bash scripts don’t hoard state, or what exactly are you saying?

The only thing I dislike about systemd is that it is written in C, but otherwise it is a single purpose solution to the surprisingly complex problem of managing services during boot and running. I fail to see how state is relevant here — if it were written in Haskell it would also have state, almost by definition it has to handle external state. No Monad would help you there, there is just correct and not correct software.

LeoPanthera2• 3 years ago

I guess it can't be all DST timezones, because here in PDT nothing broke during the transition two weeks ago.

lionkor9• 3 years ago

Seems to be (as of now) Ireland?

EvanAnderson• 3 years ago

Same here. I am adjacent two vendor-supported Proxmox clusters (Motorola Emergency CallWorks 911 phone system) and there was no outage / issue last fall moving from EDT to EST, nor two weeks ago flipping back. We’re a 24x7 operation and would have heard about any outage (either planned to address the issue before it occurred, or triage during an unplanned outage).

mastazi• 3 years ago

Many places in the Southern Hemisphere have been on daylight savings time over the past few months, I'm wondering how come this bug did not surface earlier.

lionkor9• 3 years ago

Hugged to death, archive: https://web.archive.org/web/20230325234412/https://forum.pro...

EDIT: Seems to be back.

rchard2scout• 3 years ago

Since this only seems to happen in Ireland, could it be because Ireland technically has summer time as its standard time, and winter time as negative DST? Not sure how that's implemented in tzdata, but it could very well be that some applications can't handle that.

bombcar3• 3 years ago

If that resulted in a backwards time it wasn’t expecting ..,

justinclift• 3 years ago

Heh, database corruption expected in -1... -2... -3...

idlephysicist• 3 years ago

As far as I recall, Ireland is the only country to have its timezones setup that way.

bxparks• 3 years ago

I see 3 countries with negative DST in their RULE records:

* Eire after 1971: https://github.com/eggert/tz/blob/main/europe#L526

* Morocco after 2019: https://github.com/eggert/tz/blob/main/africa#L928

* Namibia between 1994 and 2017: https://github.com/eggert/tz/blob/main/africa#L1167

Symbiote• 3 years ago

It should be easy enough to test -- set the system to Europe/London or Europe/Lisbon.

dTP90pN• 3 years ago

Here's the tzdata: https://github.com/eggert/tz/blob/71faa2a55db2c9f21f4099b58c...

Looks like the negative DST has already caused problems in the past and applications that can't handle it (ICU & OpenJDK) have to build tzdata as per the rearguard section / ziguard.awk.

mjrpes• 3 years ago

I have to say these time zone files are fascinating. Any place/time when a jurisdiction has changed the definition of time is in those files, along with a detailed history of why.

Example, California had an energy crisis soon after WWII, and changed when day light savings started/ended. There are time zone rules that track this history exactly, including a note about how clocks lost 6 minutes a day because PG&E changed AC from 60 to 59.5 Hz:

https://github.com/eggert/tz/blob/71faa2a55db2c9f21f4099b58c...

j16sdiz• 3 years ago

I am very surprised ICU can't handle it. ICU is the de facto standard of handling crazy localisation data

jeron• 3 years ago

I love proxmox but there’s a reason it’s rarely used in enterprise production…

mappu• 3 years ago

I think Proxmox adoption (both enterprise and SMB) has increased a lot in the last 6-12 months re VMware's new pricing changes.

LeBit• 3 years ago

References?

dgl• 3 years ago

https://bugs.launchpad.net/bugs/1966800 looks like a potentially related Ubuntu bug, although the underlying bug is in systemd. The commit that fixed it was in 2021: https://github.com/systemd/systemd-stable/commit/a8b66ca9af8...

The Ubuntu bug is from 2022.

Curious that proxmox is hit in 2023. Proxmox must happen to have a particular systemd timer event setup by default that triggers this.

throw0101b8• 3 years ago

Proxmox runs Debian under the hood. Proxmox 7.x is based on Debian 11.x:

* https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_7.0

happymellon4• 3 years ago

This is just another one of those reasons I just can't use Debian responsibly.

Backport hacks always come back to bite you in the end when your distro is running unsupported software, and it feels like I see something at least annually about Debian's "ensure everything is old so all bugs are predictable" ends up just causing pain.

sashk2• 3 years ago

I ran into issue, where I shutdown system with Proxmox before start of EDT, shipped it to co-worker and when he powered it on after daylight saving time was in effect, time was off by 1 hour for Proxmox and all LXC containers.

Fix was to re-symlink `/etc/localtime` to proper time zone.

Didn't want or cared to dig into this deeper, but interesting to see that there are more issues about this.

WirelessGigabit• 3 years ago

I don't understand why someone would want their servers running on anything but UTC and only use time zones in the application itself.

sashk2• 3 years ago

Agree. In this case it was technical debt, unfortunately.

elpocko• 3 years ago

Is the extra "s" after "daylight saving" canon yet?

NoZebra120vClip• 3 years ago

It's like advertising how to purchase Advanced Tickets to a concert or event.

Because who wants to be bothered with Novice or Intermediate Tickets?

wkdneidbwf2• 3 years ago

why are your servers not UTC? sheesh.

riceart• 3 years ago

All systems in common use, use UTC for the underlying representation.

The concept of setting the time zone is a high level library thing already and all these systems, their standard time functions simply provide a time since some fixed epoch.

Ie see C time(), gettimeofday, clock_gettime, GetSystemTime. They’re all “UTC” based.

If a piece of server software is going out of its way to deal with local time APIs and blows up because of it - well it either legitimately needs to deal in local time, or it’s so poorly designed that simply keeping the system locale set to UTC isn’t somehow a magic fix. That’s just an adhoc assumption.

tiernano• 3 years ago

We had a requirement from a Government agency to store the date/time in the system's local time (Ireland, in our case). All certs issued had to have the time it was issued, and they had to be local.

snvzz• 3 years ago

I hope a complaint has been filed about this policy.

wkdneidbwf2• 3 years ago

haha, amazing! i’ll put my pitchfork away :)

chillfox• 3 years ago

Why wold you use daylight savings on a server, let alone a hypervisor?

ehutch79• 3 years ago

System time should be utc. Your software should use a timezonedb or offset when displaying it to show you YOUR local time

eqvinox• 3 years ago

Yes I also run my servers on UTC.

But.

It's 2023. Setting a non-UTC timezone shouldn't break your system (or applications). Bugs happen and this is one of them, but "set your system to UTC" isn't an appropriate response here.

yarg• 3 years ago

It's not a fix, it's a mitigation.

It's perfectly appropriate to provide a workaround before the problem has actually been resolved.

citrin_ru2• 3 years ago

Bugs should be fixed indeed but using anything but UTC on a server is swimming against the tide. Bugs get fixed in old software over the time but new software written every day without consideration that time can jump around. And even old software from time to time is being rewritten in new languages with old knowledge / workarounds thrown away with bathwater (old code). So I don't believe we will run out of bugs related to DST switch anytime soon.

Vanit• 3 years ago

100% expect this to do with mixing Unix timestamps and timezones. Constant recurring footgun in my career.

skee8383• 3 years ago

i hate systemd it's always getting hung up on shutdown. "a stop job is running for blah blah" yea no. when i'm ready to shut down, it needs to shut down. immediately.

happymellon4• 3 years ago

No one else does.

When I shutdown I would like a shutdown sig to be sent to my database and then continue once it gets a successful response. No one wants that to just be killed because that would cause data loss.

When I shutdown I want my filesystem synced so things mid-write don't experience data loss.

The problem is badly written scripts that keeps sending a "one more min" response. But you could override that if you actually cared instead of just whining.

gpderetta• 3 years ago

If your database loses data when killed it is not worth the bits it is made of.

CoolCold• 3 years ago

To my understanding systemd does proper job here, as it supposed to be, highlighting the issues with other subsystems - be it badly written scripts or misorder of shutdown sequence, those parts were ignored by "traditional" init systems before.

mqus• 3 years ago

Also, systemd comes with a timeout, after which it kills the app anyways, so I think you can surely configure that 90s timeout to be something like 5s or so, which then is much faster.

But as you wrote, If someone really cares, they would fix it.

PS: But what usually gets me about these 90s waits whenever I get them is that the message does not say which thing (unit, etc) is the issue. THAT is something worth criticising.

sycren• 3 years ago

This should be the CEST Austria Daylight Savings change tonight. https://www.visitingvienna.com/visitorinfo/time/

Proxmox is based out of Vienna, Austria.

So a Timing glitch.

shrubble2• 3 years ago

How is this anything other than a systemd bug?

For servers I manage, I absolutely refuse to use an OS with systemd. There have been too many unpredictable cases.

zx8080• 3 years ago

Do you mind sharing some? I would need some real examples to show to the team.

shrubble2• 3 years ago

The above, of course.

The DNS related issues seen (have they now been resolved?) where it takes over the DNS resolver functions and refuses to honor what you have put in e.g. /etc/resolv.conf ; a web search for "systemd breaks normal dns resolv.conf" will give you a lot of results ; unsure if this is now fully fixed.

Strange things with LXC based VMs moved/migrated between hosts, even if an offline move; I think in 1 case I just ended up creating a new VM from scratch; then rsync'ing everything over from the host node into the VM's directory.

Helmut10001• 3 years ago

Ah, this is why I saw an update for `tzdata` this morning in my proxmox (?).

pshirshov• 3 years ago

There is a similar problem with Dublin timezone in Davx5 / iCal4j: https://github.com/bitfireAT/davx5-ose/discussions/265 (and noone cares)

liv-io• 3 years ago

Luckily, it did not happen to me. I manage the "timezone" by default.

For anyone interested, I manage around 20+ Proxmox machines with these Ansible roles:

https://github.com/liv-io/ansible-roles-debian

Example playbook:

https://github.com/liv-io/ansible-playbooks-example/blob/mas...

greatgib• 3 years ago

Just the usual reminder that systemd is crap and that is terrible for us that it was forced shoved to everyone in Debian and co.

The main issue was always this one, having side functions like dealing with timezones breaking your pid 1. In such a case it is almost impossible to resolve the problem from the system itself you have to mount the drive with another OS or reinstall when possible.

lionkor9• 3 years ago

im sure any init script has similar issues if (like systemd in this case?) not updated for a while

tlamponi• 3 years ago

An updated systemd package that includes a fix [0] is in progress on getting rolled out, currently on the pvetest repository.

With that the following reproducer is fixed:

    TZ=Europe/Dublin faketime 2023-03-26 systemd-analyze calendar --iterations=5 'Sun *-*-\* 01:00:00'

[0]: https://git.proxmox.com/?p=systemd.git;a=commit;h=e7eb2b5864... (just noticed that I wrote Ireland/Dublin instead of Europe/Dublin in the commit message)

[1] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_t...

systemd 100% cpu hang? – Proxmox Support Forum

Related Stories