Filespooler Append-Only Queues
Purpose of Append-Only Queues
In the Using Filespooler over Syncthing[1] example, we synced the entire `b64queue` directory. This is unnecessary.
1: /using-filespooler-over-syncthing/
It would be better to sync only the `jobs` subdirectory of it, to prevent it looking like a valid queue for processing on the sender.
Implications of Append-Only Queues
The Filespooler Reference[2] discusses append-only queues. An append-only queue will allow commands that append (`fspl queue-write`) and commands that inspect (for instance, `fspl queue-ls`) to succeed. Commands that require a sequence number, such as `fspl queue-process`, will fail.
The idea is that often times, the `jobs` directory will be synced. It would be convenient to be able to write to this directory on the sender side, while still preventing the sender from accidentally trying to process the queue. That is exactly the point of an append-only queue.
Set up
1. On the sender, use `fspl queue-init --append-only` to create an append-only queue.
2. On the receiver, use `fspl queue-init` *without* `--append-only` to create a regular queue.
3. Both queues will contain a `jobs` subdirectory. Tell your software to sync just that directory.
--------------------------------------------------------------------------------
Links to this note
- Using Filespooler over Rclone and S3, Rsync.Net, Etc.[3]
3: /using-filespooler-over-rclone-and-s3-rsync-net-etc/
You can use Filespooler with a number of other filesystems and storage options. s3fs, for instance, lets you mount S3 filesystems locally. I can't possibly write about every such option, so I'll write about one: rclone.
- Only Syncing the Jobs/ Subdirectory With an Append-Only Queue[4]
4: /using-filespooler-over-syncthing/
Filespooler[5] is a way to execute commands in strict order on a remote machine, and its communication method is by files. This is a perfect mix for Syncthing[6] (and others, but this page is about Filespooler and Syncthing).
- Filespooler[7]
Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing[8], NNCP[9], ssh, UUCP[10], USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler.
Filespooler is particularly suited to distributed and Asynchronous Communication[11].
11: /asynchronous-communication/
More on www.complete.org
(c) 2022-2024 John Goerzen