Paraelectrobombus

Following my inability to bring the new server online based on a copy of the whole filesystem (see 2026-02-26 Clone a virtual machine from inside it), I'm going to be bringing it online manually, domain name by domain name, service by service. And as in previous years, I'm going to try and document it here. For my future self. To channel my disappointment.

2026-02-26 Clone a virtual machine from inside it

Looking at my computer naming scheme, I picked a new name for the server: Paraelectrobombus.

my computer naming scheme

Locale

Add `de_CH.UTF-8` and `en_GB.UTF8`.

Time

Pick `Europe`/`Zurich`.

Software to install

Add backports:

Install `asncounter`:

Yay!

Environment

Switch to the `fish` shell:

Copy stuff for root:

Limit the logs

Hostname

Rename a computer on the Debian wiki.

Rename a computer

SSH

Facilitate the connection to the old server:

Copy and this line and append it to `~/.ssh/authorized_keys` on the old server.

On the new server:

`sshd`

Prevent login using passwords. Make sure you have authorized the necessary keys! If you ever loose access to them, use the rescue system or a VNC connection to allow passwords again.

Also less printing when I log in. I do it a lot. And switch the ssh port so that the logs aren't too busy.

Just to be sure, allow the following users only:

Apache

Remove default site, remove CGI scripts, enable `mod_md`.

There is no point in restarting the server just yet.

What's important is that I use `mod_md` to manage the certificates for my sites.

Thus, whenever Apache makes changes, I might have to update copies of the certificates elsewhere. This copy of `/etc/apache2/hook.sh` includes services that haven't migrated to the new server, yet.

The `conf-site` directory contains config files that need to be included for some locations.

This is a file that locks and unlocks sites. The default is for the site to be unlocked. A timer starts a service every few minutes that decides whether to switch it on or off.

The bot-check configuration is also documented on the Butlerian Jihad page.

Butlerian Jihad

This requires a form:

This is a file to block a lot of bots by user agent:

Garbage

The previous section uses `/run/garbage` for the bots. Here's where that comes from.

Build garbage and install it as `/usr/local/bin/garbage`.

garbage

Use a system user.

Socket:

Service:

You can use any text file you want for the garbage generation. I downloaded a copy of Moby Dick. Sorry!

Test the socket:

Monit

This uses the korero.org certificates, as shown by the `hook.sh` file above.

The config file, with a different password, of course.

Restart and check the log:

And now `https://korero.org:2812/` should work.

Munin

Setup an Apache config file:

Notice how the above relies on a CGI module.

Get the password file:

Enable it:

Fix the hostname, add my own modules and their config:

my own modules

Using the korero.org domain that's set up below: Munin.

Munin

Butlerian Jihad including fail2ban

Note that GoToSocial hasn't migrated, yet. That's why the allow-list can be empty.

Double-check:

Install the watchers and their timers but not the services to publish the `butlerian-jihad-week` fail2ban jail and not the `gotosocial-prepare` service.

Gomphotherium, rss-bot

There is a system user of the same name that owns the directory.

Service:

Timer:

The script itself is from the Gomphotherium repository.

Gomphotherium repository

Since `rss-bot` needs the same credentials, I'm going to put the service here, too. Install it from the repo into `/usr/local/bin`.

the repo

The service needs to call all commands and return an error if any one of them failed. Since there is no rush, run them one after another using `&&` in a shell:

Timer:

Since it writes into `~/.local/share/gomphotherium`, we need to make sure it can create this file. In this case, `/srv/gomphotherium` belongs to `root` so the process cannot create the necessary directories. Do it manually:

OddÎĽ

`flying-carpet.ch`

Copy the service and socket units from the old server and rename them. Set up a socket at `/run/flying-carpet.socket`.

Set up a service that uses it.

Start them both:

Test it:

Copy the website setup:

A typical setup with port 80 redirected to port 443 which uses SSL; the domains managed by `mod_md`. All requests except for the `.well-known/` directory are passed on to the socket.

The password file only needs to contain the user `claudia`:

Site, based on the config file above:

Now we're ready to switch the IPv4 and IPv6 of the domain. Verify that the change has taken hold via `ping` and then restart the web server:

If it works, go back to the old site, Sibirocobombus:

Add monitoring via `/etc/monit/conf.d/flying-carpet.conf`:

korero.org

I'm going to drop the spell checking and voice synthesis. This domain is just for email.

The Apache config file is short:

Switch the DNS to the new site and enable it:

`transjovian.org`

Install the Patched Satellite.

Regarding the permissions:

The Oddmu service:

Edit socket location:

The Apache config file, after editing the socket location and the document root:

The patched Satellite service:

The Patched Satellite configuration:

Change the DNS entries and enable it all:

Test it:

`alexschroeder.ch`

All the wiki files are in `/srv/alexschroeder/www/wiki`.

Create a user.

Apache config:

Service for Oddmu:

The last line is to exclude my dad's pages from the regular search results on my site.

Socket:

The `bookmark-feed` creates a feed for the long list of bookmark pages. You can install it from the repo into `/usr/local/bin`.

from the repo

Service:

Timer:

Make sure that the Patched Satellite can read the files:

And that the Patched Satellite knows about the domain:

Notify Antenna about new posts in Gemini format.

Service:

Timer:

Install Markdown Gopher. Since Gopher doesn't know about virtual hosts, unlike Gemini, this service is for exactly one domain: mine.

Markdown Gopher service:

For Radicale, I copied config and data, installed the package and chowned the files.

Looked over the config files and restarted it. It seems to work.

For Git, I copied the data directory, created a new user, chowned the files. It seems to work.

`git` needs to be a regular user with `/srv/home` as home directory so that we can have a file to record authorized accounts. Since I set it up the wrong way, I had to `deluser`, first.

Ensure that `/srv/git/.ssh/authorized_keys` has the right keys.

`orientalisch.info`

This is an Oddmuse wiki. Figuring out how to set it up will be important for the remaining Oddmuse wikis (Emacs Wiki, Campaign Wiki, Community Wiki and Oddmuse Wiki itself).

I created a system account to own all the files in `/srv/orientalish`.

Sadly, I was unable to make Unix domain sockets work and so the wiki still uses its own port instead of a socket.

The service:

The wrapper mounts the real wiki on `/wiki`:

The real Mojolicious server:

This is the back-end. The web server acts as a reverse proxy:

For monit:

`communitywiki.org`

This domain hosts two Oddmuse wikis and Mojolicious application.

The directories under `/src/communitywiki` are `www` for the web root, `data` and `mark` for the wiki data directory, `trunk` for Trunk. There's a copy of Oddmuse here, called `wiki.pl`.

Create a system user that will own `/src/communitywiki` and all it contains:

The Community Wiki Apache config file, which is linked to `/etc/apache2/sites-available` and then linked from there to `sites-enabled` using `a2ensite communitywiki`.

Community Wiki service making sure that the wiki is listening on port 4019:

Community Wiki wrapper script:

And the script that mounts this as `/wiki`:

The same goes for Mark.

Service:

Wrapper:

Mount:

The web application, Trunk, same thing:

Service:

The application itself is in the `trunk` directory, so only a mount is required:

The source code is here.

here

For Monit, all three web applications:

For Munin, I added the three to the `[alex_multips*]` category.

`emacswiki.org`

Directories:

We need a real user because it needs a `.ssh` directory containing a private/public key pair for git. Or perhaps we could have gotten the same effect by setting the `HOME` environment variable in the service?

Service listening on port 4002:

Wrapper script:

Mount:

Monit config:

For Munin, I added `emacswiki` to the `[alex_multips*]` category.

Emacs Wiki has a git working directory in `/srv/emacswiki/data/git`.

There is a service to push commits to the Emacs Mirror every now and then:

Timer:

Oddmuse also wants regular maintenance. This is a short script:

And a service:

And a timer:

`chat.campaignwiki.org` and `talk.campaignwiki.org`

The benefit of having separate subdomains for different services is that they can be migrated independently. The `chat` and `talk` subdomains host an instance of The Lounge, one public and one private.

I had to install The Lounge from a stand-alone deb file:

I created a user and a directory and give him ownership of all the files at the end:

Moving everything to `/srv` instead of `etc` mean that we needed to set the environment variable in the service and enable it again.

and the public one runs on a different port:

The virtual hosts for Apache do a lot of extra stuff:

Like always, the file is linked into `/etc/apache2/site-enabled`, enabled using `a2ensite thelounge`, and then apache is restarted. What follows is the strange dance of getting the DNS changes to work, to get Let’s Encrypt to work, to switch off the old The Lounge instances, transfer all the data (`users`, `logs`, `config.js`), start it on the new site and hope of the best.

If the DNS doesn’t spread, the site won’t work. It took me a while to get there.

For monit:

Since the uploads never expire, we need a job.

Timer:

Service:

Script:

`oddmuse.org`

This is a regular Oddmuse site with a web app to download from the tarballs.

Service:

Wrapper to mount it as `/wiki`:

Actual web app, using a local copy of `wiki.pl`:

Apache site:

The tarball app has a service:

Wrapper:

`tarballs.pl` itself is found in the `scripts` directory of the Oddmuse repository.

Monit:

`search.indieblog.page`

Xobaque is a search engine I run for other people.

Xobaque

I use one user to run all instances of the search engine:

This is the familiar setup. Web server communicates with socket.

Apache config:

Socket:

Service:

It updates on a timer:

This runs in the background, as nice as possible. It runs for a very long time.

Monit:

`search.emacslife.com`

This Xobaque setup is just like the previous one.

Apache config:

Socket:

Service:

Update timer:

Update service:

Monit:

`xobaque-campaignwiki`

This Xobaque setup is not quite like the previous two because the Apache web server config is part of the `campaignwiki.org` setup. For reference, this is what the `VirtualHost` says:

Socket:

Service:

Monit now refers to a path that is mentioned in the Apache config:

Update timer:

Update service:

And then once a month there is a full update!

And the service:

`campaignwiki.org`

Apache config:

Oddmu socket:

Oddmu service:

Oddmuse service:

Oddmuse mount:

Oddmuse script:

The directory structure is what we already know from elsewhere:

`data` and `www/wiki` are owned by the `campaignwiki` user.

Various directories in `www` are owned by the `planet` user: `rpg`, `osr`, `indie`, `jdr`, `podcast`, `podcast-de`, `podcast-fr`.

Monit for both:

Names

The service is called via the `ProxyPassMatch` directive of the `campaignwiki.org` Apache config.

Create a directory under `/run` such that any sockets created therein are all writeable by the `www-data` group:

The service creates a socket in the run directory created above:

The `run` directory:

The wrapper script:

`names.pl` is the actual Mojolicious application.

Monit:

Traveller

The service is called via the `ProxyPassMatch` directive of the `campaignwiki.org` Apache config.

Create a directory under `/run` such that any sockets created therein are all writeable by the `www-data` group:

The service creates a socket in the run directory created above:

The `run` directory:

Wraper script:

Install the app itself:

Monit:

Character Sheet Generator

Install the software from a tarball, as shown for Traveller, above.

Create a directory under `/run` such that any sockets created therein are all writeable by the `www-data` group, as shown for Traveller, above.

Wrapper script:

Service:

The `run` directory:

Config file to tell it about the face generator:

Monit:

Text Mapper

This uses the same setup.

Install the software from a tarball, as shown for Traveller, above.

Create a directory under `/run` such that any sockets created therein are all writeable by the `www-data` group, as shown for Traveller, above.

Wrapper script:

Service:

The `run` directory:

Monit:

Face Generator

Install the software from a tarball, as shown for Traveller, above.

Create a directory under `/run` such that any sockets created therein are all writeable by the `www-data` group, as shown for Traveller, above.

Wrapper script:

Service:

The `run` directory:

Config file, with passwords for the various users:

Monit:

Hex Describe

Install the software from a tarball, as shown for Traveller, above.

Create a directory under `/run` such that any sockets created therein are all writeable by the `www-data` group, as shown for Traveller, above.

Wrapper script:

Service:

The `run` directory:

Config file to tell it about Text Mapper and Face Generator:

Monit:

INN2 (net news)

See my peering notes.

peering

When I moved virtual machines my INN installation was borked. Specifically, when I used `rtin -A` to connect, with `~/.newsauth` set, subscribed to a few newsgroups (using `S` and a few patterns), and tried to enter them, it told me, "no articles". But when I used `ls /var/spool/articles/…` I saw articles!

Interestingly, I could only get the old (!) articles via telnet. The newer ones don't show up.

All the files are `-rw-rw-r-- 1 news news` so I'm thinking: is there some sort of index that I need to regenerate? And of course there is.

This is a Debian stable server, with very little fiddling of the settings. Most importantly, `/etc/news/storage.conf` says that the storage method is `tradspool` for all newsgroups. This installation uses `tradindexes` and `find /var/spool/news/overview/ -name "*.IDX" | xargs ls -l` shows that some of the newsgroups I care about have size 0. Time to read up on `tdx-util`.

Trying to rebuild an index with `/usr/lib/news/bin/tdx-util -R /var/spool/news/articles/grenzland/test -n grenzland.test` (should have used `sudo -u news`) gives me an error for every single article: `tdx-util: cannot find article <…> in history`. Time to read up on `makehistory`.

`sudo -u news /usr/lib/news/bin/makehistory -O` and I'm getting another error: `makehistory: cannot open /run/news/.rebuildoverview: No such file or directory`. And I'm not sure how to take that since I stopped the `inn2.service` before doing this. And with that, I'm not surprised that stuff in `/run` is gone. As `/run/news` is the `pathrun` option in `inn.conf` I'm assuming it's safe to create it while `inn` is not running.

Now it reports non-zero sizes for the things I care about. It seems to work!

News web application

This should be a different user than `news`.

We still keep the files in `/srv/news`.

Service:

The `run` directory:

Monit:

`stunnel4` for INN2

Configure:

This uses the web server certificates created by `mod_md`.

`ngircd`

This is the IRC server that is connected to two other servers.

Planet Jupiter

There are a number of planets this runs: the blogs I follow, RPG Planet, Old School RPG Planet, Indie RPG Planet, RPG Podcast Planet, Planète des JDR, Podcast Planète des JDR, Rollenspiel Podcast Planet.

the blogs I follow

RPG Planet

Old School RPG Planet

Indie RPG Planet

RPG Podcast Planet

Planète des JDR

Podcast Planète des JDR

Rollenspiel Podcast Planet

The OPML files are available from the respective sites.

Download all the blog feeds:

Timer:

Service:

Assemble them into web sites using the templates I wrote for them (not linked):

Timer:

Service:

Building a dependency that wasn't written by me makes me want to avoid `root`. Here's the installation:

Transmission

In `settings.json`, set the `download-dir` property to `/srv/campaignwiki/www/1pdc/`.

Norn

User:

Installing Norn was a pain. See Build AnyEvent::Discord on Debian for how to build it all. Install the Debian files; then install App-Norn-1.tar.gz by unpacking it in `/srv/norn/perl`:

Build AnyEvent::Discord on Debian

Service:

Monit doesn't know if Norn works or not. It might be interesting to write up an IRC connection to see if the user `norn` is online?

Norn writes its files into `files` so create a symlink:

Moira

User:

Service:

Monit doesn't know if Norn works or not.

Moira knows where to write the XML file because it's part of the Discord server configuration:

The local `osr-discord.xml` file is the template and `/srv/campaignwiki/www/files/osr-discord.xml` is the target file.

Roll

The dice roller offers dice rooms. This is a server component.

Service:

The `run` directory:

Monit:

TODO

campaignwiki.org:

Other: