Comment by 🚀 stack
Re: "Made myself a personal capsule"
My site returns the index page on error, and all the 280 links are relative. Just happened that way. someone has been crawling it pointlessly for weeks now.
Feb 06 · 3 months ago
6 Later Comments ↓
🎮 jprjr [OP] · Feb 06 at 21:42:
I created a script on my site at /tarpit/. Each page ends with a new "next page" link, and it just goes on forever. Responses also get slower and slower as you go. I made sure to list it in my robots.txt so, hoping it wastes bad crawler's time.
At a certain point of following the never-ending links you start getting chapters of The Damned by Algernon Blackwood. The first chapter's at the 64th link, the second is at the 4,096 link, and so on.
I think in theory it will generate 64^990 links (though there's only 9 chapters, so once you hit link 18014398509481984 you'll have all the chapters.)
In the old days, you had to break a sweat to get crawled by bots, now you have to put in effort to not get crawled...
I love the client server interaction example on your gemlog. "let's fucking go!"
I'm thinking of adding a "crawler trap" for my https mirror of my Gemini capsule that's running on Web 1.0 hosting. Oh yeah, I shoudl share that on my next post
Only one post in your gemlog - post more!
I'll make sure to post more for this month.
Original Post
Made myself a personal capsule — Hoping to get back into doing some writing! I've also got a fun surprise for crawlers that ignore robots.txt.