I wish systemd logged information about the source of “transactions”

https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdTransactionSourceWish

by zdw • 5 years ago

127 77 5 years ago

utcc.utoronto.ca

usr11067• 5 years ago

While generally the systemd documentation is pretty good (Start with reading the original blog series as a primer, later search for "systemd directives" using your favourite search engine) I have always found the transactions concept underdocumented. Does anyone have a good link?

Edit: Are transactions and jobs the same thing? Both are mentioned in the documentation here and there, without having an own page or chapter AFAIK.

egberts12• 5 years ago

“Documentation” is “Pretty Good?

“Start with” the “blog”?

Clearly, the two does not connect well … for systemd.

jeppesen-io• 5 years ago

... as a primer

It's worked very well for me. Not seeing your disconnect here. Blog for concepts, man pages for specifics

usr11067• 5 years ago

It's open source, you get what you pay for. Actually, you get much more, but you need to fill any gaps with your own efforts. Accessing the spread out documentation is one of them.

While they are not comparable entities I'd say systemd documentation is in better shape than Linux kernel documentation. (Not to negate the efforts of those who do work with kernel documentation, but to stress the huge areas of no or pretty obsolete documentation).

Of course you can buy Linux (incl. user space) from the commercial players. I have only worked in one small project in my career that did, but I did not notice that better documentation was worth paying for.

usr11067• 5 years ago

Clearly, the two does not connect well … for systemd

It's linked from the man page. https://man7.org/linux/man-pages/man1/systemd.1.html

Unfortunately the reading will still need to do ourselves. Not trying to be snarky here, it happens to me all the time that I have not read something and complain it's hard to understand.

FriedrichN• 5 years ago

Is there a good resource for people who are used to non-systemd systems (something like a gotcha list)? I keep on running into weird situations where I end up finding out systemd is somehow responsible for my woes. Last time that happened was when I changed /etc/fstab but somehow old mounts kept on being remounted, I wasted an hour before I found out I had to reload some systemd service.

mkipper• 5 years ago

I'm not familiar with the term transactions, but it doesn't sound like it's the same as jobs.

Jobs are externally visible. You can see them easily (e.g. systemctl list-jobs), and systemd provides an interface for them over D-Bus[1]. There's no similar interface for anything called a transaction.

From the documentation that does mention transactions, it sounds like transactions are internal to systemd. When systemd starts a unit, it works out the dependency graph and spins up a job for each unit that need to be started before the originally requested unit can start. That would all be considered a single transaction, but it might spin out into dozens of separate jobs that get queued up.

As an example, when systemd starts on boot, all it really wants to do is successfully reach some target (e.g. multi-user.target). systemd starts from there and works backwards, building a dependency graph with every single unit that needs to start up as a part of the boot sequence to reach that target. You could probably consider that a single transaction, but the full dependency graph would probably pull in hundreds of jobs.

I don't work on systemd or anything so this isn't canon.

[1] https://www.freedesktop.org/wiki/Software/systemd/dbus/

Ono-Sendai• 5 years ago

Ran into a similar problem, trying to work out why systemd was stopping my service.

avian• 5 years ago

systemd just has so many reasons to kill processes, and invents new ones with new releases. Timeouts for things it thinks it should be short-lived, resource limits, service isolation and sandboxing settings, etc. They are mostly documented, but you need to know where to look. While I came to like some aspects of systemd, debugging why things die for apparently no reason after systemd was upgraded has eaten many of my workdays.

buserror2• 5 years ago

New one as of this morning is to kill STOP processes it doesn't like. I have no idea why, or how to revent it. All I did was update my 'sid' debian and here goes

TimWolla3• 5 years ago

A peer of mine was exiting their SSH sessions with 'exit'. One time apparently they already typed 'systemctl', probably in an attempt to check the status of a service, changed their mind and then later wanted to close the session using 'exit', actually executing 'systemctl exit'. This translated into a shutdown of the machine in question.

After being able to piece together what happened with the machine's logs and the bash history I recommended to simply exit all programs/sessions with Ctrl+D. It works almost everywhere and would have prevented this exact issue.

Denvercoder9• 5 years ago

I recommend not using root privileges unnecessarily. `systemctl status` and most other querying commands don't need root, while the dangerous things like `systemctl exit` do.

usr11067• 5 years ago

Sure, but some systems do poweroff without being root. Has happened to me, don't remember the details. Maybe a polkit thing because users are supposed to shut down their own laptop?

TimWolla3• 5 years ago

I can't say what they had in mind when typing `systemctl`. Even they couldn't. Because of the delay in the shutdown to cleanly stop the services, they had already forgotten that they just exited a SSH session and thus the connection between the machine being dead and typing `exit` was not obvious.

Maybe it was `systemctl status`. Maybe it was intended to be a `reload` (which would require elevated privileges).

usr11067• 5 years ago

I did not even know that systemctl exit exists. Tye man page says it's equivalent to poweroff (for the system manager not running in a container, i.e. the most common case).

Having 2 alternative commands for the same functionality is not a good design decision IMHO. But not the most central design decision for systemd.

dharmab2• 5 years ago

there are already multiple commands for turning the computer off without systemd anyway

iforgotpassword• 5 years ago

Almost entirely unrelated, but lately I worked with old photos that had missing, incomplete or wrong exif data. While trying to assess and automate fixing the collection via scripts, I used the command line utility "exif" a lot. You can't imagine how many times I typed "exit somefile.jpg" and flinched when the terminal window just closed. Guess I should just have created an alias but that's like resigning to your own stupidity. ;-)

mike_hock• 5 years ago

Also, spam Ctrl+C if you're gonna issue a new command after being AFK for a while.

lathiat• 5 years ago

I can do one better for exit. The Solaris kernel debugger KDB can be used at runtime for inspecting some stats and also to change some global configurable variables.

For whatever reason if you type a variable/symbol it assigns it the value 0. If you type exit nothing happens immediately but as soon as the next process exits usually a few seconds to 10s of seconds later the entire system kernel panics on a null pointer de-reference.

Kernel paniced our production ZFS filer twice before I cottoned on. Newer releases special cased “exit” not to do that.

heavenlyblue• 5 years ago

That sounds like the most amazing UX I have ever encountered :)

For those familiar with Solaris, is there any reason they did it this way?

How can you possibly set the default behaviour of assigning any symbol a value of 0 by default?

cyberpunk• 5 years ago

Remember how on sparc machines if you turned off your laptop with the console cable connected it powers the servers down too?

That was a fun lesson…

zorr• 5 years ago

Years ago I had the habit of shutting down my laptop with "sudo shutdown -h now" at the end of the workday. Until one day I did that accidentally in a live SSH session.

Since then I always shut down my machine using the GUI and I have Tmux configured with different colors for SSH sessions.

thanatos519• 5 years ago

There's a package called molly-guard which can help with that.

MattConfluence• 5 years ago

One time I was using my laptop to remote desktop into a computer I was hundreds of km away from physically, I discovered that if I hit the power key on the laptop (yes, the laptop had a key on the keyboard for power, not a separate power button) while having the remote session window focused, it sent the signal over the wire and put the remote host to sleep instead. Whoops.

After that I looked up how to enable wake-on-lan and open up a port to be able to do that remotely.

yencabulator• 5 years ago

I've done the opposite, back in the day. I used server A's keyboard & monitor to SSH to admin server B. When I was done with the system upgrade on server B, I rebooted it with control-alt-del. Except I rebooted server A.

ASalazarMX• 5 years ago

Happened to me years ago. Fortunately that server had a power saving bug that most times rebooted it instead of powering it down.

genewitch• 5 years ago

On average, systems are poo. The median modes around.

I prefer openRC, and I'll wave a flag or whatever but to each their own.

WinNT4 was goat.

ungamed• 5 years ago

I'll just get off your lawn.

danesparza• 5 years ago

... eyeing you suspiciously and backing away ...

koheripbal• 5 years ago

What's wrong with just closing the putty window?

gruez• 5 years ago

alt-f4 is slightly more efforts to do (requires hand contortions) comapred to writing out a word and typing enter. Also, control-D is easier.

dec0dedab0de• 5 years ago

Maybe they're not using a GUI ssh client, and wanted to return to their local shell.

dharmab2• 5 years ago

ctrl+D/exit doesn't close the window in all cases. eg if you're sshed to a remote it returns you to local.

corney91• 5 years ago

Ctrl+D is great, but will still run a command if it's been typed out so I always Ctrl+C and then Ctrl+D.

TimWolla3• 5 years ago

It does not for me. If the current command line is not completely empty it will not do anything (both with bash and fish). So if the terminal does not close after pressing Ctrl+D I will know that something is wrong and check more carefully.

ploek• 5 years ago

Similarly I discovered yesterday that the systemd service definition for auditd includes the `RefuseManualStop` option for this exact reason. When stopping (and thus also when restarting) the service via systemd, auditd is unable to log who shut it down, so it just disallows being stopped. (https://linux-audit.redhat.narkive.com/3weoVaZE/rational-beh...)

The workaround is to use the service command instead. Manually I usually do that anyway, muscle memory etc. But Ansible's service module will default to systemctl if it finds systemd. So there I had to add a "use: service".

AndyMcConachie• 5 years ago

systemd is why I love OpenBSD.

Datagenerator• 5 years ago

The acronym POLA translates to peace where systemd creates havoc and burns many hours globally having people investigate what the heck is going on in this almost binary blob of spaghetti.

chasil• 5 years ago

Guess what?

https://www.itsfoss.net/initware-as-a-systemd-fork-on-openbs...

guilhas• 5 years ago

A very slim down fork would possibly not be that bad

gunapologist992• 5 years ago

Don't forget Artix, Void, MX Linux, FreeBSD, Alpine, Gentoo/Funtoo..

ufo• 5 years ago

Does OpenBSD log what source caused the machine to reboot?

OJFord2• 5 years ago

It seems odd to me to go to the effort to tweet & blog about 'wishing' for something like this, but not to open a GitHub issue requesting it: https://github.com/systemd/systemd/issues

(At least, I can't find it searching there, and OP doesn't link one, so I assume there isn't one.)

dorianmariefr• 5 years ago

Good point, created one https://github.com/systemd/systemd/issues/21497 (I'm not the author though)

dijit• 5 years ago

I'm probably going to get a lot of hate for saying this; because anything anti-systemd tends to attract weird people who claim that you want bash scripts back but: Making issues on systemd's github page is often met with stoicism and reluctance on the part of the systemd developers.

There's countless examples of the systemd maintainers refusing to fix bugs or acknowledge that bugs exist.

Making a public statement could be more effective.

OJFord2• 5 years ago

Maybe, personally though (if that were my experience and I was OP) I'd try an issue first, and then if it was shot down I could tweet & blog 'unfortunately, doesn't look like it's going to happen' or whatever, referencing my attempt.

It just seems like they went to about the same amount of effort, but on something with a much lower (IMO) probability of being actioned. (Not least because if it were it'd probably go though roughly the same process anyway, such as through the issue now opened by a sibling comment to yours. Maybe PR without issue, but which OP could (skills permitting) also have done.)

blueflow• 5 years ago

Related incident: "Systemd killing processes each minute at second 27" (German):

https://blog.uberspace.de/systemd-anzuenden/

gunapologist992• 5 years ago

Aside from the maintenance and uptime issues, OP is actually raising a very real security concern; often an attacker will reboot the machine to restart all of the logging processes or to load in a LKM. Knowing why, and when (and which process) just forced a reboot is a very real requirement.

Systemd also seemingly randomly attacks my processes, and it's almost impossible to actually figure out why. (At least the kernel OOM killer actually logs "Out of memory, killed this process."[0])

0. https://linuxwheel.com/oom-killer-explained/

I wish systemd logged information about the source of “transactions”

Related Stories