Meta Information About Gemini Crawlers

Here is some info about different Geminispace crawlers for people, who look into their log files every now and then and want to know what’s going on. Ideally, I would like to have the respective crawler owner publish this information on their crawler’s sites prominently in one place, so I can delete this page.

While the amount of indexed pages and capsules is not strictly necessary, as a user I would appreciate this information to make an informed choice about which search engine to use. It is omitted here, because the numbers may change daily.

The contact info is from the linked sites excactly how it is written there. Contact me, if you want your personal info to be changed or removed, or if you have a crawler and want to publish missing information.

There are virtual user-agents defined for Gemini for crawlers and proxies, and, like on the web, they should follow it. But especially most of the web proxies listed below don’t. They expose Gemini content on the web, effectively republishing it in a space, where the content creator might not want to publish of be found.

robots.txt for Gemini

Note: `pubnix-aware` means `example.com/~foo` and `example.com/~bar` are treated as different sites with each having its own `robots.txt`. That allows for `example.com/~foo` being equivalent to `foo.example.com`. Additionally, this allows for telling pages and people’s Gemini capsules apart to have better statistics.

Last update: 2026-04-24

Indexers

Kennedy

IPv4: 64.149.155.184
IPv6: 2600:1700:1731:d0f:35a7:42d4:c71f:a02b
robots.txt user agents: indexer, *
index update date: gemini://kennedy.gemi.dev/stats
pubnix-aware: ?
content types: text/gemini, text/plain, images
content size limit: 10 MB
crawl frequency: monthly
crawl delay: 1.5 s
rate limited: yes
redirects: yes
removal after unavailability: immediately with the next crawl
filtered URIs: yes
API: no
contact: acidus at gemi dot dev
repository: https://github.com/acidus99/Kennedy

TLGS

IPv4: 23.88.52.182
IPv6: 2a01:4f8:c0c:8d14::1
robots.txt user agents: tlgs, indexer, *
index update date: gemini://tlgs.one/statistics
pubnix-aware: ?
content types: text/gemini, text/plain, text/markdown, text/x-rst, plaintext
content size limit: 2.5 MB text
crawl frequency: ?
crawl delay: ?
rate limited: ?
redirects: yes, max. 5
removal after unavailability: 1 month
filtered URIs: yes
API: limited, gemini://tlgs.one/doc/api
contact: pilot \at tlgs.one
repository: https://github.com/marty1885/tlgs

Freeshell Hashtags

IPv4: 116.202.128.144
IPv6: 2a01:4f8:231:482b::2
robots.txt user agents: hashtags, indexer, *
index update date: gemini://freeshell.de/hashtags/stats.gmi
pubnix-aware: no
content types: text/gemini
content size limit: none
crawl frequency: once
crawl delay: 5 s, but from a randomized queue of all indexed pages
rate limited: on “44 SLOW DOWN” no more pages will be requested
redirects: not handled correctly
removal after unavailability: no
filtered URIs: yes
API: no
contact: jbanana at dreamwidth dot org
repository: not open source

AuraSearch

IPv4: dynamic
IPv6: n/a
robots.txt user agents: indexer, *
index update date: gemini://aurasearch.ddns.net/stats/
pubnix-aware: ?
content types: ?
content size limit: ?
crawl frequency: ?
crawl delay: 2 seconds
rate limited: yes
redirects: ?
removal after unavailability: ?
filtered URIs: ?
API: no
contact: ?
repository: not open source

Alvus

IPv4: 77.161.107.142
IPv6: ?
robots.txt user agents: ?
index update date: ?
pubnix-aware: ?
content types: ?
content size limit: ?
crawl frequency: ?
crawl delay: ?
rate limited: ?
redirects: ?
removal after unavailability: ?
filtered URIs: ?
API: ?
contact: ?
repository: ?

Archivers

DeLorean Time Machine

(like Kennedy above)
robots.txt user agents: indexer, archiver, *
content types: anything
crawl frequency: weekly

Researchers

Lupa

IPv4: 193.70.85.11
IPv6: 2001:41d0:302:2200::180
robots.txt user agents: researcher
index update date: gemini://gemini.bortzmeyer.org/software/lupa/stats.gmi
pubnix-aware: no
content types: anything
content size limit: none
crawl frequency: ?
crawl delay: ?
rate limited: ?
redirects: yes, but not for the current crawl
removal after unavailability: 46 days
filtered URIs: yes, but not public
API: no
contact: stephane+gemini@bortzmeyer.org
repository: https://framagit.org/bortzmeyer/lupa

Web Proxies (From the Web to Geminispace)

Please contact me, if you know any proxies not listed here

Project Segfault

IPv4: 110.227.202.171
IPv6: n/a
robots.txt user agents: no
contact: ?
repository: ?
note: makes Gemini content linkable on the web

Obsessive Facts

IPv4: 24.181.150.210
IPv6: n/a
robots.txt user agents: no
contact: henriquez@protonmail.com
repository: ?
note: makes Gemini content linkable on the web, replicates are browser-like interface

Wobbly

IPv4: 109.228.177.104
IPv6: n/a
robots.txt user agents: no
contact: bjorn.warmedal@gmail.com, ew0k@tilde.team
repository: ?
note: replicates are browser-like interface

Aggregators

Antenna

IPv4: 109.228.177.104
IPv6: n/a
robots.txt user agents: ?
index update date: manual through user submissions
feed support: Gemini page subscriptions, Atom, RSS, twtxt
content size limit: ?
crawl frequency: manual feed submission with a maximum of 5 minutes until the feed is being crawled
crawl delay: ?
redirects: ?
removal after unavailability: n/a
filtered URIs: user-configurable
API: no
contact: bjorn.warmedal@gmail.com, ew0k@tilde.team
repository: gone
note: feed has to be submitted every time it should be crawled

Ucant Antenna

(like Antenna above)
IPv4: 178.62.35.244
IPv6: n/a
note: like Antenna (backup)

DSN Antenna

(like Antenna above)
IPv4: 137.184.25.127
IPv6: n/a
note: like Antenna, but only non-computer related submissions

CAPCOM

IPv4: 167.88.113.156
IPv6: 2604:180:1:cad::36da
robots.txt user agents: ?
index update date: no, but there are post dates
feed support: Gemini page subscriptions, Atom
content size limit: ?
crawl frequency: every 8 hours for 100 randomly selected feeds which change monthly
crawl delay: ?
redirects: ?
removal after unavailability: gemini://geminiprotocol.net/news/2023_06_28.gmi
API: no
contact: none officially published
repository: https://git.sr.ht/~solderpunk/capcom
note: feed has to be submitted once

gmisub aggregate

IPv4: 95.111.237.17
IPv6: 2a02:c207:2038:2970::1
robots.txt user agents: ?
index update date: no, but there are post dates
feed support: Gemini page subscriptions
content size limit: ?
crawl frequency: twice a day
crawl delay: ?
redirects: ?
removal after unavailability: ?
API: no
contact: callum@calcuode.com
repository: https://git.sr.ht/~callum/gmisub
note: feed has to be submitted once, no Ed25519 certificate support

Nytpu’s Comitium Subscriptions

IPv4: 71.19.146.159
IPv6: 2605:2700:0:3:a800:ff:fee7:3f74
robots.txt user agents: ?
index update date: yes
feed support: Gemini page subscriptions, Atom, RSS, and JSON
content size limit: ?
crawl frequency: ?
crawl delay: ?
redirects: ?
removal after unavailability: ?
API: no
contact: alex [at] nytpu.com
repository: https://git.nytpu.com/comitium/

Juntaletras

IPv4: 45.157.179.215
IPv6: n/a
robots.txt user agents: ?
index update date: no, but there are post dates
feed support: ?
content size limit: ?
crawl frequency: every two hours
crawl delay: ?
redirects: ?
removal after unavailability: ?
API: no
contact: ?
repository: ?

Cosmos

Meta-aggregator of BBS, CAPCOM, Antenna, nytpu’s comitium subscriptions, gmisub aggregate, bot en deriva, The Midnight Pub, Smol Pub, Flounder, Station, Geddit (as of 2023-05-22). Some sources, especially the Spacewalk ones, spawn other gemlog-type sources that are then remembered and fetched separately.
IPv4: dynamic with a long lease
IPv6: n/a
robots.txt user agents: ?
index update date: no, but there are post dates
content size limit: ?
crawl frequency: 60-90 minutes (randomized to spread the load over time)
crawl delay: ?
redirects: ?
removal after unavailability: ?
filtered URIs: yes
API: no
contact: jaakko.keranen@iki.fi
repository: not open source