2025-07-09 systemd timers for the Butlerian Jihad

I was reading a post about the good parts of systemd (systemd has been a complete, utter, unmitigated success) and started wondering about the use of timers instead of cron jobs.

systemd has been a complete, utter, unmitigated success

The benefits:

each job is a complete script
each job can be sandboxed
each job has output in the log instead of mailing it to me

And if every job is a complete script, then I can also add more stuff, like lowering the limits if load starts to overshoot again.

So here we go.

I have four jobs:

watch-active-autonomous-systems
watch-expensive-end-points
watch-nobots
watch-attempted-edits

Each of them gets a service and a timer.

Since I wanted the service to protect a good part of the system, I had to move things around a bit.

`asncounter`

I had to install asncounter in `/usr/local`. Since I already had a copy of it all I just moved things around, but things to work out next time, my `root` user has `PIPX_HOME` set to `/usr/local/pipx`. This is very messy. I hope that `asncounter` makes it into Debian, soon. ❤️

asncounter

So:

mv ~/.local/pipx /usr/local

I also edited the first line of `asncounter` to read as follows:

#!/usr/local/pipx/venvs/asncounter/bin/python

`/etc/butlerian-jihad`

I created this directory for the scripts and its dependencies.

10min-access-log

2h-access-log

asn-networks

I also create a `data` directory here for the `pyasn` data. This is not the default location so we must always pass `--cache-directory /etc/butlerian-jihad/data` when running `asncounter`.

scripts

The scripts all exclude some more IP numbers in their call to `2h-access-log` (my home IP addresses, my server's IP addresses) so I don't accidentally ban myself, as well as the `social` subdomain which is where my fedi instance is. In addition to that, my IP addresses are also in the allow-list that I use for various things (fediverse servers I am connected to, friends using dubious internet service providers).

`watch-active-autonomous-systems`

#!/usr/bin/sh
export PATH=/etc/butlerian-jihad:/usr/local/pipx/venvs/asncounter/bin:$PATH
export XDG_CACHE_HOME=/etc/butlerian-jihad/data
LOAD=`cat /proc/loadavg | cut -d' ' -f1`
if test 1 = `echo "$LOAD > 30" | bc`
then LIMIT=100
elif test 1 = `echo "$LOAD > 20" | bc`
then LIMIT=150
elif test 1 = `echo "$LOAD > 10" | bc`
then LIMIT=200
elif test 1 = `echo "$LOAD > 5" | bc`
then LIMIT=250
else LIMIT=300
fi
echo "Load $LOAD, limit $LIMIT"
2h-access-log \
    | awk '{print $2}' \
    | asncounter --top 50 --no-prefixes 2>/dev/null \
    | awk '/^[0-9]/ && $1 > '$LIMIT' { print $3 }' \
    | ifne xargs asn-networks \
    | ifne xargs fail2ban-client set butlerian-jihad banip

`watch-expensive-end-points`

#!/usr/bin/sh
export PATH=/etc/butlerian-jihad:/usr/local/pipx/venvs/asncounter/bin:$PATH
export XDG_CACHE_HOME=/etc/butlerian-jihad/data
LOAD=`cat /proc/loadavg | cut -d' ' -f1`
if test 1 = `echo "$LOAD > 10" | bc`
then LIMIT=3
elif test 1 = `echo "$LOAD > 5" | bc`
then LIMIT=4
else LIMIT=5
fi
echo "Load $LOAD, limit $LIMIT"
10min-access-log \
    | egrep '\baction=(rss|rc|contrib)\&|\bsearch=|/random-pick' \
    | awk '{print $2}' \
    | asncounter --top 50 --no-prefixes 2>/dev/null \
    | awk '/^[0-9]/ && $1 > '$LIMIT' { print $3 }' \
    | ifne xargs asn-networks \
    | ifne xargs fail2ban-client set butlerian-jihad banip

`watch-nobots`

#!/usr/bin/sh
export PATH=/etc/butlerian-jihad:/usr/local/pipx/venvs/asncounter/bin:$PATH
export XDG_CACHE_HOME=/etc/butlerian-jihad/data
LOAD=`cat /proc/loadavg | cut -d' ' -f1`
if test 1 = `echo "$LOAD > 10" | bc`
then LIMIT=3
elif test 1 = `echo "$LOAD > 2" | bc`
then LIMIT=4
else LIMIT=5
fi
echo "Load $LOAD, limit $LIMIT"
10min-access-log \
    | awk '$10 == 410 {print $2}' \
    | asncounter --top 50 --no-prefixes 2>/dev/null \
    | awk '/^[0-9]/ && $1 > '$LIMIT' { print $3 }' \
    | ifne xargs asn-networks \
    | ifne xargs fail2ban-client set butlerian-jihad banip

`watch-attempted-edits`

This watches "attempted edits" -- bots requesting the edit page and never posting anything. A clear sign of bot activity, if you ask me.

#!/usr/bin/sh
export PATH=/etc/butlerian-jihad:/usr/local/pipx/venvs/asncounter/bin:$PATH
export XDG_CACHE_HOME=/etc/butlerian-jihad/data
LOAD=`cat /proc/loadavg | cut -d' ' -f1`
if test 1 = `echo "$LOAD > 10" | bc`
then LIMIT=4
elif test 1 = `echo "$LOAD > 2" | bc`
then LIMIT=7
else LIMIT=10
fi
echo "Load $LOAD, limit $LIMIT"
2h-access-log \
    | attempted-edits \
    | awk '{print $2}' \
    | asncounter --top 50 --no-prefixes 2>/dev/null \
    | awk '/^[0-9]/ && $1 > '$LIMIT' { print $3 }' \
    | ifne xargs asn-networks \
    | ifne xargs fail2ban-client set butlerian-jihad banip

`attempted-edits`

#!/usr/bin/env perl
use v5.40;
my %get;
my %post;
while (<>) {
  my ($host, $ip, $path, $upload, $id) = /^(\S+) (\S+) [^"]* "GET ([^?; ]*)\?action=edit;(upload=1;)?id=([^; ]*) HTTP\/[0-9.]+" 200/;
  if ($host && $ip && $path && $id) {
    $get{"$host/$ip$path"} = $_;
    next;
  }
  ($host, $ip, $path) = /^(\S+) (\S+) [^"]* "POST (\S*)/;
  if ($host && $ip && $path) {
    $post{"$host/$ip$path"} = 1;
    next;
  }
}
for my $key (keys %get) {
  print $get{$key} unless $post{$key};
}

`*.service`

Each script gets a service file. I'm only going to post `watch-active-autonomous-systems.service`. The only thing that changes from service to service is the `Description` and the `ExecStart` naming the script to run.

[Unit]
Description=Watch active autonomous systems
RequiresMountsFor=/var/log
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/etc/butlerian-jihad/watch-active-autonomous-systems

# Priority has to be higher than the regular web services so that banning can still happen.
# See systemd.exec(5) for more.
Nice=9
IOSchedulingClass=best-effort
IOSchedulingPriority=3

# asncounter needs to download new route view data
# PrivateNetwork=true

# asncounter needs to save the data somewhere
ReadWritePaths=/etc/butlerian-jihad/data

LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

`*.timer`

Each script gets a timer file. I'm only going to post one of them. The only thing that changes from service to service is the `Description` as by default the timer applies for the service with the same name. Win!

I'm adding a `RandomizedDelaySec` of two minutes, hoping that over time the three timers will start to diverge.

[Unit]
Description=Watch active autonomous systems

[Timer]
OnCalendar=*:00,10,20,30,40,50:00
RandomizedDelaySec=120

[Install]
WantedBy=timers.target

systemd cheat-sheet

# install service or timer (only once)
systemctl enable --now ./the-job.service
systemctl enable --now ./the-job.timer
# check how it went
systemctl status the-job.service
systemctl status the-job.timer
# run it again
systemctl start the-job.service
systemctl start the-job.timer
# check the logs (use --follow for watching)
journalctl --unit the-job.service
journalctl --unit the-job.timer

#Administration #Butlerian Jihad #systemd