Looks like my tarpit has managed to serve about 31,000 pages to alvus.nl

I had a post a while back about crawlers that ignore robots.txt:

bbs.geminispace.org/s/Geminispace/37243

So when I made my personal capsule at buffering.party I created "the tarpit" - a part of the site listed in robots.txt that just returns never-ending links that get slower and slower as you go.

buffering.party/tarpit/

Looking at the logs I see good ol alvus.nl is still chugging away at page #31,229.

I'm just getting a kick out of knowing how much time that's wasting due to not honoring robots.txt.

Posted in: s/Geminispace

🎮 jprjr

Feb 19 · 3 months ago · 👍 drh3xx, norayr · 😄 1