Dealing with bots

I kept working on this page on how to deal with bots, and I'd be happy
to add more ideas, or refine the existing sections. If anybody is
interested in how to get fail2ban to work with their server, I'd be
happy to add pages explaining how to do this if you provide me with a
log file snippet, for example.

=> //transjovian.org:1965/gemini/page/Dealing%20with%20bots

Feel free to copy and use elsewhere.

Feel free to mail me directly with comments.

On 2021-10-27 08:40, Alex Schroeder wrote:
> I kept working on this page on how to deal with bots, and I'd be happy
> to add more ideas, or refine the existing sections. If anybody is
> interested in how to get fail2ban to work with their server, I'd be
> happy to add pages explaining how to do this if you provide me with a
> log file snippet, for example.
> 
> => //transjovian.org:1965/gemini/page/Dealing%20with%20bots
> 
> Feel free to copy and use elsewhere.
> 
> Feel free to mail me directly with comments.

Good job!

> ### Banning IP numbers is problematic
> It’s true. Perhaps there’s a shared server at that IP number. One of
> the users on that server writes a misbehaving bot and all are
> punished. If you are concerned about that, your server needs to move
> the dynamic content behind a client certificate requirement. There is
> no other way to identify particular users using Gemini.

I'm concerned about that, so I have Tor exit nodes explicitly ignored
from blocking. That's what I have in my jail.d files (except for sshd):

> ignorecommand = /bin/grep <ip> /etc/tor/torbulkexitlist

/etc/tor/torbulkexitlist is updated daily by a cronjob from
https://check.torproject.org/torbulkexitlist

Anna “CyberTailor” <cyber@sysrq.in> writes:

> I'm concerned about that, so I have Tor exit nodes explicitly ignored
> from blocking.

I added a link to gemini://transjovian.org/gemini/page/fail2ban and
there I described your setup. Thanks again!

-- 
Fingerprint: DF94 46EB 7B78 4638 7CCC  018B C78C A29B ACEC FEAE

I've stumbled upon a clever way to stop bots from accessing dynamic
content and infinite loops: a simple captcha!

 > gemini://topotun.hldns.ru/cgi-bin/lock.cgi
 < 10 Protection against bots. What is 2+2?
(translated ru-en)

It's so obvious but I've never thought about it lol.

This article has /brilliant/ captcha ideas by the way:
=> https://nearcyan.com/you-probably-dont-need-recaptcha/

My all-time favorites are:

* Gentoo Wiki captcha
=> https://wiki.gentoo.org/wiki/Special:CreateAccount

* Mathematical analysis captcha (it won't look good in input prompts
  though, however unicode has enough math symbols)
=> https://math.stackexchange.com/questions/2266227/solving-math-captcha-in
volving-a-limit-and-sin1-x/2266239
=> https://lurkmore.to/Матановая_капча

On Thu, 2021-10-28 at 04:51 +0500, Anna “CyberTailor” wrote:
> I've stumbled upon a clever way to stop bots from accessing dynamic
> content and infinite loops: a simple captcha!
> 
>  > gemini://topotun.hldns.ru/cgi-bin/lock.cgi
>  < 10 Protection against bots. What is 2+2?
> (translated ru-en)

So the bots will index gemini://capsule.example/locked-content?4

I think the better way to handle bots stuck in an infinite loop is to
ban them _temporarily_, or ban them just from the recursive links.

-- 
DJ Chase
They, Them, Theirs