Comment by 🛸 bluesman
On the off chance the admin sees this, it's good that you're checking robots.txt but it needs to be queried on each crawler pass. I recently added robots.txt to exclude a new gemini streaming resource. The crawler did check once for robots.txt weeks ago but it hasn't looked for it since. Without the exclude directive, the search engine tries to download an "infinite" audio stream (it shouldn't do that anyway, IMO). I had to kick the connections after many hours on several occasions. I blocked the crawler IP for the moment.
Jan 11 · 4 months ago
Original Post
Alvus Search — Anyone know who runs the Alvus search engine? I'd like a word. [gemini link]