[Tech] Useless statistics of the day: status codes

In the database of the Lupa crawler
<gemini://gemini.bortzmeyer.org/software/lupa/>, I find the following
status codes (by order of decreasing frequency):

(The status codes are documented in
<gemini://gemini.circumlunar.space/docs/specification.gmi#fragments_could_b
e_useful_to_indicate_the_exact_section>.)

20 : OK
51 : Not found, because many links no longer work, already, or may be
never worked
31 : Permanent redirection (Lupa adds the target URI to its database
but keeps the original one)
30 : Temporary redirection
10 : Input, there are 310 URI replying by asking for input
50 : Permanent failure, something wrong in the server
40 : Temporary failure, server overloaded?
60 : Client certificate required, 40 URI require this authentication
42 : The ultimate answer (no, actually, CGI error)
53 : Proxy request refused (typically a name pointing to an IP address
where the server is not configured for this virtual host)

The rest is marginal, less than ten URI:

49 : First status code I find which is not in the official
specification. Anyone has an idea? Two URI.
2 : Old server with the deprecated one-digit status codes?
39 : Not in the official specification. Anyone has an idea? Two URI.
59 : Bad request (no idea why my client sends bad requests)
99 : Not in the official specification. Anyone has an idea? Two URI.
58 : Not in the official specification. Anyone has an idea? Two URI.
29 : Not in the official specification. Anyone has an idea? Two URI.
43 : Proxy error. Reason unknown.
61 : Certificate not authorised. Since I do not send client
certificates, this is surprising.
41 : Server not available (maintenance, for instance).
52 : Gone. Like 51 but final. Just one URI,
<gemini://gemini.conman.org/no-longer-here/> so it's probably a test.
22 : Not in the official specification. Anyone has an idea? One URI.
69 : Not in the official specification. Anyone has an idea? One URI.
19 : Not in the official specification. Anyone has an idea? One URI.

The number of status codes not registered in the specification may
mean that some servers are extending the Gemini protocol with unused
status codes. But some are also in test capsules like
<gemini://gemini.conman.org/> or <gemini://egsam.glv.one/>.

Note that Lupa did not find even one occurrence of:

11 : Sensitive input
44 : Slow down (may be because the crawler is nice)
62 : Certificate not valid (which makes sense since I don't send
certificates)

On Mon, 4 Jan 2021 11:53:41 +0100
Stephane Bortzmeyer <stephane at sources.org>:

> In the database of the Lupa crawler
> <gemini://gemini.bortzmeyer.org/software/lupa/>, I find the following
> status codes (by order of decreasing frequency):
> 
> (The status codes are documented in
> <gemini://gemini.circumlunar.space/docs/specification.gmi#fragments_could
_be_useful_to_indicate_the_exact_section>.)
> 
> 20 : OK
> 51 : Not found, because many links no longer work, already, or may be
> never worked
> 31 : Permanent redirection (Lupa adds the target URI to its database
> but keeps the original one)
> 30 : Temporary redirection
> 10 : Input, there are 310 URI replying by asking for input
> 50 : Permanent failure, something wrong in the server
> 40 : Temporary failure, server overloaded?
> 60 : Client certificate required, 40 URI require this authentication
> 42 : The ultimate answer (no, actually, CGI error)
> 53 : Proxy request refused (typically a name pointing to an IP address
> where the server is not configured for this virtual host)
> 
> The rest is marginal, less than ten URI:
> 
> 49 : First status code I find which is not in the official
> specification. Anyone has an idea? Two URI.
> 2 : Old server with the deprecated one-digit status codes?
> 39 : Not in the official specification. Anyone has an idea? Two URI.
> 59 : Bad request (no idea why my client sends bad requests)
> 99 : Not in the official specification. Anyone has an idea? Two URI.
> 58 : Not in the official specification. Anyone has an idea? Two URI.
> 29 : Not in the official specification. Anyone has an idea? Two URI.
> 43 : Proxy error. Reason unknown.
> 61 : Certificate not authorised. Since I do not send client
> certificates, this is surprising.
> 41 : Server not available (maintenance, for instance).
> 52 : Gone. Like 51 but final. Just one URI,
> <gemini://gemini.conman.org/no-longer-here/> so it's probably a test.
> 22 : Not in the official specification. Anyone has an idea? One URI.
> 69 : Not in the official specification. Anyone has an idea? One URI.
> 19 : Not in the official specification. Anyone has an idea? One URI.
> 
> The number of status codes not registered in the specification may
> mean that some servers are extending the Gemini protocol with unused
> status codes. But some are also in test capsules like
> <gemini://gemini.conman.org/> or <gemini://egsam.glv.one/>.
> 
> Note that Lupa did not find even one occurrence of:
> 
> 11 : Sensitive input
> 44 : Slow down (may be because the crawler is nice)
> 62 : Certificate not valid (which makes sense since I don't send
> certificates)

My server may be the cause of the 39 codes. When I have read the
specs I missed the appendix so I assumed all 3x code were fine for
redirecting and I chose 39.  While looking why Amfora client was
complaining about "unknown status core 39" on irc #gemini, I've
been told to look at the appendix and I realized my mistake.

> On Jan 4, 2021, at 11:53, Stephane Bortzmeyer <stephane at sources.org> wrote:
> 
> 44 : Slow down (may be because the crawler is nice)

I use that one! :)

But perhaps not exactly for the intended purpose.

It goes like this:

(1) An URL is submitted for analyzes
(2) The service responds with a 31 redirect to where the result will be 
(3) If the results are not there yet, the service responds with a 44 (try again later)
(4) Eventually, once processed, the service responds with 20 (ok, or some 
errors when appropriate)

Kind of an asynchronous API of sort.

? ???

It was thus said that the Great Stephane Bortzmeyer once stated:
> 
> The rest is marginal, less than ten URI:
> 
> 49 : First status code I find which is not in the official
> specification. Anyone has an idea? Two URI.
> 2 : Old server with the deprecated one-digit status codes?
> 39 : Not in the official specification. Anyone has an idea? Two URI.
> 59 : Bad request (no idea why my client sends bad requests)
> 99 : Not in the official specification. Anyone has an idea? Two URI.
> 58 : Not in the official specification. Anyone has an idea? Two URI.
> 29 : Not in the official specification. Anyone has an idea? Two URI.
> 43 : Proxy error. Reason unknown.
> 61 : Certificate not authorised. Since I do not send client
> certificates, this is surprising.
> 41 : Server not available (maintenance, for instance).
> 52 : Gone. Like 51 but final. Just one URI,
> <gemini://gemini.conman.org/no-longer-here/> so it's probably a test.
> 22 : Not in the official specification. Anyone has an idea? One URI.
> 69 : Not in the official specification. Anyone has an idea? One URI.
> 19 : Not in the official specification. Anyone has an idea? One URI.
> 
> The number of status codes not registered in the specification may
> mean that some servers are extending the Gemini protocol with unused
> status codes. But some are also in test capsules like
> <gemini://gemini.conman.org/> or <gemini://egsam.glv.one/>.

  The Gemini Client Torture Test [1] is responsible for the following
erroneous status codes:

	99, 29, 39, 49, 58, 2, and 0.

  I know there's another client test out there, so it could be that as well.

  -spc

[1]	gemini://gemini.conman.org/test/torture/

Indeed, egsam.glv.one will return 2, 222, hi, 19, 29, 39, 49, 59, 69,
and 99 - https://github.com/pitr/egsam/blob/master/main.go#L83-L104

On Mon, Jan 4, 2021 at 8:16 PM Sean Conner <sean at conman.org> wrote:
>
> It was thus said that the Great Stephane Bortzmeyer once stated:
> >
> > The rest is marginal, less than ten URI:
> >
> > 49 : First status code I find which is not in the official
> > specification. Anyone has an idea? Two URI.
> > 2 : Old server with the deprecated one-digit status codes?
> > 39 : Not in the official specification. Anyone has an idea? Two URI.
> > 59 : Bad request (no idea why my client sends bad requests)
> > 99 : Not in the official specification. Anyone has an idea? Two URI.
> > 58 : Not in the official specification. Anyone has an idea? Two URI.
> > 29 : Not in the official specification. Anyone has an idea? Two URI.
> > 43 : Proxy error. Reason unknown.
> > 61 : Certificate not authorised. Since I do not send client
> > certificates, this is surprising.
> > 41 : Server not available (maintenance, for instance).
> > 52 : Gone. Like 51 but final. Just one URI,
> > <gemini://gemini.conman.org/no-longer-here/> so it's probably a test.
> > 22 : Not in the official specification. Anyone has an idea? One URI.
> > 69 : Not in the official specification. Anyone has an idea? One URI.
> > 19 : Not in the official specification. Anyone has an idea? One URI.
> >
> > The number of status codes not registered in the specification may
> > mean that some servers are extending the Gemini protocol with unused
> > status codes. But some are also in test capsules like
> > <gemini://gemini.conman.org/> or <gemini://egsam.glv.one/>.
>
>   The Gemini Client Torture Test [1] is responsible for the following
> erroneous status codes:
>
>         99, 29, 39, 49, 58, 2, and 0.
>
>   I know there's another client test out there, so it could be that as well.
>
>   -spc
>
> [1]     gemini://gemini.conman.org/test/torture/