Working with Systemd Timers

https://yieldcode.blog/post/working-with-systemd-timers/

by skwee357 • 1 year ago

80 78 1 year ago

It took a minute to setup, but using a combination of rsync and timers to backup system files has done wonders to decrease my anxiety around upgrades on arch

bhaney8• 1 year ago

Using a modern CoW filesystem lets you level this up to just a snapshot command in a pacman pre-upgrade hook.

tombert3• 1 year ago

I know everyone hates on systemd, but I've generally been happy with it, in no small part because I find the timers to be pretty sensible.

They are, in all ways that I care about, simply better than cron, and especially in NixOS they're really easy to set up.

linsomniac3• 1 year ago

I'm convinced that the people who hate systemd are a small minority. They also happen to be fairly vocal, which I think makes them seem outsized.

throwaway314155• 1 year ago

One might even go so far as to say they're a vocal minority.

jauntywundrkind• 1 year ago

I'd be excellent to see their complaints collected and elaborated. Because God Fucking Damn, these seem like some incredible whining Advanced Persistent Threats against anything getting better. If I were a pro-discord disruptor, I'd pay these people to keep at their endless grudges against every single person having to figure out ever single fucking capability de-novo.

Systemd makes so so so many incredibly good service administration capabilities easy & on tap. The main thing I want from systemd haters is to show how how how folks did things well before. Mostly I think this is a rebellion of the losers, people who hate having to do a good job, people who hate learning the many many many many ways we could run services better, who valorize each service figuring out their own unique special bespoke ways of running their stuff. Its fallen as fuck. We need need need the systemd socialization of running stuff well, controlling permissions & access. There's near zero precedent from the past of anyone else being responsible. That's just the situation. Systemd has drastically elevated what sysops can do, orthogonally to the service by service init launching of the past that got us no where.

MyOutfitIsVague• 1 year ago

I think in general it's a shrinking minority. I was outspokenly critical of systemd in the past, but much about it has improved, and I have to admit it has made my life significantly easier in many ways. I'm still not a big fan of some things being systemd projects that don't really need to be under the umbrella (why is resolved even a systemd project? Or systemd-timesyncd?) but service definition and management, transient services, sockets, timers, and many many more things are pretty great. At this point, I pretty much love it. Even things that initially annoyed me, like timers, I've grown to love, not least because my cronjob entries are manageable files that I can commit and backup, and I can individually enable and disable them without having to deal with stupid commenting and such.

Checking service status and managing services is also significantly better with systemd than any other init system I've used, and I've used a lot over 15 years of systems administration.

I still find journald a bit annoying, though, and I still am not completely sold on binary log files. I understand the benefits, but working with them is still much more opaque than text logs.

Cyph0n4• 1 year ago

+1 for NixOS. I used to find systemd a bit intimidating, but working with NixOS has made it all fit together somehow.

As an example, this is a systemd timer I have that periodically runs rclone to sync photos to Backblaze:

    systemd.services.rclone-photos-sync = {
      serviceConfig.Type = "oneshot";
      path = [ pkgs.rclone ];
      script = ''
        rclone \
          --config ${config.age.secrets."rclone.conf".path} \
          --bwlimit 20M --transfers 16 \
          sync /mnt/photos/originals/ photos:
      '';
      unitConfig = {
        RequiresMountsFor = "/mnt/photos";
      };
    };
    systemd.timers.rclone-photos-sync = {
      timerConfig = {
        # Every 2 hours.
        OnCalendar = "00/2:00:00";
        RandomizedDelaySec = "5m";
        Persistent = true;
        Unit = "rclone-photos-sync.service";
      };
      partOf = [ "rclone-photos-sync.service" ];
      wantedBy = [ "timers.target" ];
    };

SoftTalker3• 1 year ago

That would all be one line in a crontab.

tombert3• 1 year ago

I had to write a systemd timer to rescan PCIe ports about ten seconds after the computer starts. I was able to add that to my NixOS config and it worked how I wanted, and it was great because if I had broken anything all I would have to do is reboot and choose an older generation.

freedomben• 1 year ago

I like so much about Nix and the Nix philosophy, but I'm a bit stuck on the perceived ugliness of the Nix language. It just feels so ... icky to me. I don't typically feel that way about languages. Even Erlang (while being bizarre) didn't/doesn't bother me that much.

Does anybody else have this problem? How do you get over it? (it seems like such a ridiculous reason to avoid picking up an otherwise good tech)

bitwize3• 1 year ago

Systemd hater here. My stance is and always was: if you love systemd, go nuts with it. I'm not looking to reach into your bathroom to set your shower temperature.

But I don't want to have to run it or even think about it. Lennart has other plans, hence his campaign to get distros to require it and software up the stack to hard-depend on it. I don't like its design, I want to opt out of it in favor of something better, but one of the development effort's goals is making that extremely difficult for modern Linux systems. That's my issue with it.

But really it macht nichs for me right now because I run Void, btw.

eth0up• 1 year ago

I was feeling a bit Kafka-ish with all the gushing systemd support until I saw this comment. And same here; I don't care what other folks are enamored with, but them "other plans" you mention, they concern me and I wish it concerned others too.

Vilian• 1 year ago

I don't like its design, I want to opt out of it in favor of something better

Others design were already tested extensively, and the one that stuck was systemd, don't you already stopped to think that the "better design" is the one that systemd are currently using?

paulddraper2• 1 year ago

distros to require it

Distros requires a lot of stuff, e.g. libc. This does not seem inappropriate given what a distro is.

software up the stack to hard-depend on it

What does this refer to?

I can see this for session/user/service management programs. But not normal single-user programs.

stephenr2• 1 year ago

I was initially against systemd (around the time Debian started discussing using it) for non-technical reasons: the project seemed to have some very negative aspects in terms of how feedback and concerns are addressed. I don't know for sure but I strongly suspect this was the core issue a good chunk of those "against" were concerned about: the way the project is run.

Having used it for several major releases of Debian technically I'm happy to have it - it's an improvement over the sysv init & crontabs. I'm not completely in love with journal compared to regular logs but I can see some advantages.

SuperNinKenDo• 1 year ago

I actually love systemd like 96% of the time.

The only time I don't is when it just cannot do something, and then you need to layer the old Unix way on top of it using the wrong semantics, i.e., services that are actually complicated bash scripts which are used to do something for which systemd has specialised units, like an (auto)mount. Then it's all the pain of brittle Unix scripts, with misleading semantics, and an extra layer of abstraction and set of limitations to keep in your head.

But no, I genuinely think that systemd is great these days.

I do understand why people become irked by it becoming a dependency of programs which really should be able to function without it though.

One thing that makes it difficult is how scattered between all the different manpages the documentation is. It took me quite a lot of time and frustration to build up a decent mental model of what everything is for, when it needn't have if (ironically) the docs were just written in a less Unixy way.

But making my system feel more event-driven and less imperative has been a pleasure.

timrichard• 1 year ago

When I started using systemd timers, I really liked the systemd-analyze calendar facility, to calculate n trigger times for a given calendar expression.

For example, show the next five trigger times for the end of the last day when the month has 31 days :

systemd-analyze calendar --iterations=5 '*-*-31 23:59:59'

bhaney8• 1 year ago

I like systemd's timers when they're appropriate, but I really think the author's use case here is better suited to cron. The "issues" he listed for cron aren't very good either.

If you want to execute pre/post commands you have to do it inside the script itself

So?

There are no built-in logs

Every cron implementation I can remember using logs each run to syslog and emails me the output of the run by default

There is no built-in status monitoring

I can't think of any built-in status monitoring that systemd has for timers that's materially different from cron's logging/emailing

If the system is down when the cron needs to run, the cron will be missed

Some cron implementations support this and some don't. Most modern ones that I'm aware of do.

Much more significantly, the amount of setup involved in a systemd timer is way higher than putting a line in a crontab, especially for the author's case of just running a backup script.

Cron only involves running `crontab -e` and adding the line "@daily /path/to/script.sh" (which also handles the author's issue of cron "skipping" runs if the system was powered off, assuming the cron implementation uses something modern like anacron)

Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.

burnJS2• 1 year ago

I pair crons with healthchecks.io

chatmasta• 1 year ago

It would be cool if someone made a crontab interface to systemd timers. Use crontab syntax but generate systemd boilerplate with sane defaults.

bhaney8• 1 year ago

https://wiki.debian.org/systemd-cron

1una• 1 year ago

IMO Systemd Timers provide much better control over how the cron job is run.

Need to distribute lots of cron jobs evenly to avoid overloading the system? Use RandomizedDelaySec.

Some cron jobs are flaky and you want to re-run if it fails? Add Restart=on-failure to corresponding service.

Some cron jobs conflict with each other? Set Conflicts=foo.service or maybe Before & After.

Sure, all above are possible with shell scripts. But systemd provides a standard, reliable way to define these properties.

bhaney8• 1 year ago

Systemd Timers provide much better control over how the cron job is run

Yes, they do. And as the very first sentence in my comment says, I like systemd timers when they're appropriate. None of the features you've mentioned were used in the author's solution, which is the sole thing I'm arguing against.

Ferret74465• 1 year ago

Much more significantly, the amount of setup involved in a systemd timer is way higher than putting a line in a crontab

Counterpoint: this requires root, while you can edit and run systemd timers without root. Thus it's also generally more secure, both when creating the job and every time it runs.

Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.

You could write a script to automate it, if it's such a big deal. Creating timers/cronjobs isn't something that needs to be done often enough for this to matter.

If it is, another counterpoint: systemd supports creating transient timers, which you could do programmatically.

And final counterpoint: you can make systemd understand crontab and convert it into timers (systemd-crontab-generator)

bhaney8• 1 year ago

Counterpoint: this requires root

Counter-counterpoint: No it doesn't? Running `crontab -e` as a non-root user will edit that user's crontab, and running it as root will edit the system crontab. Cron can be configured to deny users their own crontabs, but every common distro I'm aware of defaults to allowing user crontabs.

You could write a script to automate it, if it's such a big deal

Or I could not bother with that and just use cron?

another counterpoint: systemd supports creating transient timers

What is this a counterpoint to? That the author's particular use case of running a backup script once a day is a task better suited to cron than systemd timers? I'm not sure how transient timers are even relevant here, much less a counterpoint.

burnJS2• 1 year ago

This is cool and after learning of this I'm tempted to convert my crons. I currently use systemd for running a redis queue and it's worked great for years.

notepad0x902• 1 year ago

There is one huge advantage of cron that is usually missed with such comparisons. cron is dead-easy to create and maintain. Systemd timers make sense in certain situations, but in 90%+ of my use cases so far, the added complexity really adds up the amount of time I'm administering my systems. I've been in a situation where I needed to troubleshoot why a systemd timer wasn't triggering, and I didn't like that experience at all. To me, it is something I would use if I actually needed it, it isn't a 1-to-1 replacement for cron.

packetlost• 1 year ago

I actually disagree with simplicity. cron does weird things with the environment and it can be hard to correctly reproduce that environment when creating a job and debugging. Conversely, systemd timers are very easy to manually trigger and otherwise behave as a normal oneshot service.

SoftTalker3• 1 year ago

My approach to cron is to assume that you have an empty environment. Just set everything you will need.

encoderer4• 1 year ago

I’m the developer of crontab.guru and Cronitor.

We have a free tool called CronitorCLI that includes a cron-like shell. You can run and test your scripts in an environment that matches how they will be run by cron itself.

Ferret74465• 1 year ago

There is also one huge disadvantage of cron: it requires root. Users can create and run systemd timers without root/sudo.

And quite frankly, your experience is likely due to your personal knowledge bias. Systemd timers are quite easy to debug, in fact easier than cron in my opinion. systemctl list-timers lists all the timers, when they last ran, when they will run next, and you can use other commands to inspect each timer in detail and all associated logs.

In contrast, I find cron much harder to debug. Starting with the first problem, you must first figure out what cron is running, as different implementations have different behavior!

raincom2• 1 year ago

One can set up cron jobs under non-root.

vaylian• 1 year ago

I found the implicit column-based configuration layout of cron rather cubersome. I prefer the key-value configuration layout of systemd timers.

Tractor8626• 1 year ago

So you find troubleshooting cron jobs easier?

Cron has no diagnostics whatsoever. Most asked question "How do I force run cron job for debugging" and answer "You don't"

You just add script and pray it works. If it doesn't work there is nothing to work with. No logs, no checks. Nothing.

It not 1-to-1 replacement. It is superior replacement. Cron is just unusable.

Frotag• 1 year ago

You just add script and pray it works.

All the cron issues I've had boiled down to a difference in env vars. So testing the cronjob means reproducing the environment like this SO answer explains.

https://stackoverflow.com/questions/2135478/how-to-simulate-...

kinglawrence• 1 year ago

Really liked this read. Is anyone able to explain how the backup.timer runs the backup.service? It wasn't obvious to me where the trigger was defined. I guess it's just inherent to what that unit type does, and the fact that both units are named backup? What is the name for that "package" of units that make up the whole backup program?

bhaney8• 1 year ago

I guess it's just inherent to what that unit type does, and the fact that both units are named backup?

Correct. If there's no `Unit=` specified in the timer unit, it defaults to the service unit of the same name. See https://www.freedesktop.org/software/systemd/man/latest/syst...

This is also a common pattern for the socket units of socket activated services

akeck• 1 year ago

I used systemd user timers, a shell script, and an HDHomerun to make a simple VCR.

rs_rs_rs_rs_rs• 1 year ago

I use them all the time, my only wish is they get rid of the .service file and make it possible to define the service inside the .timer file.

xyse53• 1 year ago

I initially had a similar reaction, but there are so many cases like this for unit files. I think a higher level DSL for generating them would be useful.

paulddraper2• 1 year ago

The separation of concerns is legitimate.

merpkz• 1 year ago

Minor nitpick - shouldn't you first define the service and only then a timer for it? Otherwise since you enabled timer and are still trying to figure out how to write service, systemd won't have anything to run when timer triggers. Maybe I am wrong, but that just feels like logical order. Anyways, after years on hating on systemd I also started to embrace it and porting my cron jobs to systemd timers and I must admit it's really nice, the overall overview with list-timers, next execution timestamp, total execution time, ordering of services so one can run after another is completed and of course the logging in journal so I can filter output and keep track of everything it's just wonderful experience.

EDIT: yea, the email reporting is certainly missing, but it was hard to control it since whole STDOUT was shipped, which is not what I wanted most of the time anyways. It would be good to come up with some way to still have small overview emails sent about important jobs done, maybe a dependency service which starts when important job finished and just sends an email about that

dmd• 1 year ago

I would love to see something like https://github.com/isd-project/isd but for systemd timers.

Working with Systemd Timers

Related Stories