Comment by 🍀 gritty
TIL what bash.org is.
Jan 10 · 4 months ago
1 Later Comment
🛸 bluesman [OP] · Jan 11 at 14:15:
On the off chance the admin sees this, it's good that you're checking robots.txt but it needs to be queried on each crawler pass. I recently added robots.txt to exclude a new gemini streaming resource. The crawler did check once for robots.txt weeks ago but it hasn't looked for it since. Without the exclude directive, the search engine tries to download an "infinite" audio stream (it shouldn't do that anyway, IMO). I had to kick the connections after many hours on several occasions. I blocked the crawler IP for the moment.
Original Post
Alvus Search — Anyone know who runs the Alvus search engine? I'd like a word. [gemini link]