File size issues

Two quick points with regard to the fact that Gemini currently does not
convey file sizes to users at any point:

* Sean has pointed out in one of his RFCs that this means there is no
  way for a client to know whether or not a download completed
  successfully or was interrupted due to an accidentally dropped or
  even a maliciously severred connection

* I've received an email from somebody watching the Gemini design unfold
  with interest, who is concerned about Gemini clients with limited
  system resources unwittingly downloading large files (such as PDFs of
  scanned documents) which they aren't even capable of opening.  While I
  quite like the idea of Gemini being friendly to low-end systems, I do
  wonder whether or not the TLS requirement makes this a little moot.

Anyway, the question is do we want to change anything to address these
issues and if so how do we want to do it?

I'll quickly note in pasing that both of these problems also exist in
exactly the same form for Gopher, but I've never once heard Gopher users
complain about them.  Which isn't to say we should cheerfully ignore
them, but it does establish that they cannot reasonably be considered as
fatal flaws.

One possibility, as proposed by Sean, is to add file size to the
response header, with it optionally appearing after the MIME type.  I'm
not hugely fond of this myself, simply because it complicates parsing of
the response header.  Remember that the MIME type can have multiple
components specifying encodings etc.  If you just split the META part of
the header on whitespace, the number of components is variable, so
recognising whether or not an optional filesize is present requires
actually inspecting the parts and looking for a number.  In fairness to
Sean, at the time of writing of his RFC the spec spec said META was
separated from STATUS by a tab (whereas now it is just whitespace), so
tacking something after META with another tab was unambiguous, assuming
nobody put tabs in their MIME types...

Another possibility ties into another request I got from somebody very
early on - it would be nice if there was some way to query a Gemini
server for the time a resource was last modified, so that Gemini
equivaents of tools like moku pona could avoid needlessly fetching
unchanged resources over and over again.  At that point I started
wondering about giving Gemini some equivalent of HTTP HEAD, although I
abandoned it pretty quickly when I realised that substantial TLS
overhead probably made making a whole second request to check if a
resource had changed not such a worthwhile idea.  But, we could possibly
bring this idea back, as the response to such a request could naturally
include the file size as well.  The real question is how to *make* such
a request, ideally in a way which doesn't open the door to a half dozen
other new "methods".

At this point I'm not necessarily convinced that either of these
problems warrant changes to the protocol.  It's not that I can't see how
it would be nice to work around them, but at some point one has to
accept that protocols which are simple and minimalist by design are
inevitably going to have some rough spots as a result of that design.
But if anybody has particularly elegant solutions, I'm all ears.

Regarding ways to enable something like a HEAD request without changing
the request format to include a method field - I'm not quite sure
whether using a fixed URL fragment, like #meta, on requests would be a
kosher way to do this.  Does metadata count as "some portion or subset
of the primary resource, some view on representations of the primary
resource, or some other resource defined or described by those
representations" (from RFC3986)?

-Solderpunk

It was thus said that the Great solderpunk once stated:
> Two quick points with regard to the fact that Gemini currently does not
> convey file sizes to users at any point:
> 
> * Sean has pointed out in one of his RFCs that this means there is no
>   way for a client to know whether or not a download completed
>   successfully or was interrupted due to an accidentally dropped or
>   even a maliciously severred connection
> 
> * I've received an email from somebody watching the Gemini design unfold
>   with interest, who is concerned about Gemini clients with limited
>   system resources unwittingly downloading large files (such as PDFs of
>   scanned documents) which they aren't even capable of opening.  While I
>   quite like the idea of Gemini being friendly to low-end systems, I do
>   wonder whether or not the TLS requirement makes this a little moot.
> 
> Anyway, the question is do we want to change anything to address these
> issues and if so how do we want to do it?
> 
> I'll quickly note in pasing that both of these problems also exist in
> exactly the same form for Gopher, but I've never once heard Gopher users
> complain about them. 

  Gopher does address this rather obliquely---text files (and gopher
indexes) are supposed to end with a '.' on a line by itself.  This lets the
client know it received the data correctly, and it says as much in RFC-1436,
section 3.8:

	Note that for type 5 or type 9 the client must be prepared to read
	until the connection closes.  There will be no period at the end of
	the file; ...

  It's not necessarily a pain point about the filesizes not being known
before hand, but it does make displaying a progress bar (for example)
difficult to implement.

> One possibility, as proposed by Sean, is to add file size to the
> response header, with it optionally appearing after the MIME type.  I'm
> not hugely fond of this myself, simply because it complicates parsing of
> the response header.  

  I'm not seeing much of an issue.  Assuming tabs separate the compoents on
the status line, then

	(\d+)\t([^\t]+)(\t([^\t]+))*

would parse the line (I suspect, I'm not a fan of regex but I think the
above would work to parse the status line).  I don't see much of an issue in
parsing any of the following:

	20<HTAB>text/plain; charset=utf-8<HTAB>2123<CRLF>
	20<HTAB>text/plain<CRLF>

> Remember that the MIME type can have multiple
> components specifying encodings etc.  If you just split the META part of
> the header on whitespace, the number of components is variable, so
> recognising whether or not an optional filesize is present requires
> actually inspecting the parts and looking for a number.  In fairness to
> Sean, at the time of writing of his RFC the spec spec said META was
> separated from STATUS by a tab (whereas now it is just whitespace), so
> tacking something after META with another tab was unambiguous, assuming
> nobody put tabs in their MIME types...

  Which could be specified, "don't put tabs in the MIME type section."

> Another possibility ties into another request I got from somebody very
> early on - it would be nice if there was some way to query a Gemini
> server for the time a resource was last modified, so that Gemini
> equivaents of tools like moku pona could avoid needlessly fetching
> unchanged resources over and over again.  At that point I started
> wondering about giving Gemini some equivalent of HTTP HEAD, although I
> abandoned it pretty quickly when I realised that substantial TLS
> overhead probably made making a whole second request to check if a
> resource had changed not such a worthwhile idea.  

  One way would be to query a well-known endpoint (these exist in the HTTP
world---robots.txt is one such file) that contains tiemstamps for various
resources.  Slap a MIME type of text/gemini-timestamp and call it done:

gemini://example.com/	2019-08-15T13:53:00-05:00
gemini://example.com/feed	2019-07-29T00:00:00-05:00
gemini://example.com/other	2019-08-01T00:00:00-05:00

  That's one way.

> But, we could possibly
> bring this idea back, as the response to such a request could naturally
> include the file size as well.  The real question is how to *make* such
> a request, ideally in a way which doesn't open the door to a half dozen
> other new "methods".

  As I mentioned in a private email to solderpunk earlier, one could always
take advantage of the sub-delimeters in the path portion.  I had at one
point mentioned using those to specify the prompt (otherwise the server
would return a status of 10):

	gemini://example.com/search;Search%20for

This could be formalized:

	gemini://example.com/search;prompt=Search%20for
	gemini://example.com/blogfeed;timestamp=2019-08-15T00:00:00Z
	gemini://example.com/wildexample;prompt=Search%20for;timestamp=2019-08-15:
00:00:00Z?query=foo&usename=bar

  So, you have "prompt" and "timestamp".  Others could be propsed.  If the
"timestamp" thing above is accepted, then you might want to have a new
status code meaning "no change" or "okay, but there's no content".

> Regarding ways to enable something like a HEAD request without changing
> the request format to include a method field - I'm not quite sure
> whether using a fixed URL fragment, like #meta, on requests would be a
> kosher way to do this.  Does metadata count as "some portion or subset
> of the primary resource, some view on representations of the primary
> resource, or some other resource defined or described by those
> representations" (from RFC3986)?

  Well, there are RFC-5147 and RFC-7111 that give semantics to the URI
fragment section, but I still think using the sub-delimeter of ';' in the
path portion is the way to go.

  -spc

>   Gopher does address this rather obliquely---text files (and gopher
> indexes) are supposed to end with a '.' on a line by itself.  This lets the
> client know it received the data correctly, and it says as much in RFC-1436,
> section 3.8:

Whoops, true!  In my defence, I think this is very rarely used nowadays.
VF-1 includes no code whatsoever to detect and strip this from files it
downloads and I've never seen one appear on screen.
 
>   I'm not seeing much of an issue.  Assuming tabs separate the compoents on
> the status line, then
> 
> 	(\d+)\t([^\t]+)(\t([^\t]+))*
> 
> would parse the line (I suspect, I'm not a fan of regex but I think the
> above would work to parse the status line).  I don't see much of an issue in
> parsing any of the following:
> 
> 	20<HTAB>text/plain; charset=utf-8<HTAB>2123<CRLF>
> 	20<HTAB>text/plain<CRLF>
> 
>   Which could be specified, "don't put tabs in the MIME type section."

Yes, with sufficient prescription of whitespace practices in response
headers it could be made sufficiently parsable, but it would be nice if
things weren't so brittle.

This also, of course, sets a precendent of "whenever we decide a little
bit of extra metadata would be handy in the header, just append it after
a tab", which over time could bloat our header until it's basically just
a HTTP header in disguise with tabs instead of newlines.

(not a fan of regex either, by the way, and was quite happy to discover
Lua's lightweight alternative system when I first picked it up)
 
>   One way would be to query a well-known endpoint (these exist in the HTTP
> world---robots.txt is one such file) that contains tiemstamps for various
> resources.  Slap a MIME type of text/gemini-timestamp and call it done:
> 
> gemini://example.com/	2019-08-15T13:53:00-05:00
> gemini://example.com/feed	2019-07-29T00:00:00-05:00
> gemini://example.com/other	2019-08-01T00:00:00-05:00
> 
>   That's one way.

I actually quite like this idea.  No need to make it timestamp-specific
either.  We could have a well-known endpoint for general file metadata,
which listed modification time, file size, checksum, MIME type, etc.  It
could accept queries for a specific path, and *that* could be the way to
do an equivalent of a HEAD request.

This would let clients for specific scenarios do the extra work
themselves to work around their problems, e.g. clients with very low
memory or storage space could request the metadata for all files before
attempting to access them and warn the user if the file size exceeds a
threshold; clients on unreliable connections could request the metadata
before downloading and then warn the user if file size and/checksum did
not agree.  Most "normal" clients could do neither and just operate as
they already do.

I think it's kind of neat to keep solutions to edge problems outside of
the protocol itself and push them into things like well-known endpoints
like the above where they can easily be ignored when they are not
needed/wanted.  The downside is that server developers have to do the
work to add support for these things - but it's expected, I think, that
servers are harder to write than clients.  Ease of client implementation
is very important - it leads to a large number of independent clients,
which means unofficial extensions of the standard can only really take
off if a large number of people with presumably diverse opinions can be
convinced they are worthwhile.  And, of course, some server authors can
just choose not to support some of these endpoints, and when queried can
just return status 51 and then the client understands they are on their
own.  All of this can be done without any change to the core Gemini spec
(each well-known endpoint, of course, would need its own spec).

>   As I mentioned in a private email to solderpunk earlier, one could always
> take advantage of the sub-delimeters in the path portion.  I had at one
> point mentioned using those to specify the prompt (otherwise the server
> would return a status of 10):
> 
> 	gemini://example.com/search;Search%20for
> 
> This could be formalized:
> 
> 	gemini://example.com/search;prompt=Search%20for
> 	gemini://example.com/blogfeed;timestamp=2019-08-15T00:00:00Z
> 	gemini://example.com/wildexample;prompt=Search%20for;timestamp=2019-08-1
5:00:00:00Z?query=foo&usename=bar
> 
>   So, you have "prompt" and "timestamp".  Others could be propsed.  If the
> "timestamp" thing above is accepted, then you might want to have a new
> status code meaning "no change" or "okay, but there's no content".

I think I prefer the well-known endpoint over this, but that's right now
more of a gut reaction and not a well thought-out and defencible
position. 
 
>   Well, there are RFC-5147 and RFC-7111 that give semantics to the URI
> fragment section, but I still think using the sub-delimeter of ';' in the
> path portion is the way to go.

Ah, more for the reading list!

-Solderpunk

solderpunk writes:

> This also, of course, sets a precendent of "whenever we decide a little
> bit of extra metadata would be handy in the header, just append it after
> a tab", which over time could bloat our header until it's basically just
> a HTTP header in disguise with tabs instead of newlines.

We are, of course, re-inventing the Content-Length header here, which is
an extremely useful header so that clients can do things like have a
progress bar for downloads. If we support every useful feature, of
course, we end up with basically HTTP 0.9.

> Sean writes:
>>   One way would be to query a well-known endpoint (these exist in the HTTP
>> world---robots.txt is one such file) that contains tiemstamps for various
>> resources.  Slap a MIME type of text/gemini-timestamp and call it done:
>>
>> gemini://example.com/	2019-08-15T13:53:00-05:00
>> gemini://example.com/feed	2019-07-29T00:00:00-05:00
>> gemini://example.com/other	2019-08-01T00:00:00-05:00
>>
>>   That's one way.

IMO you probably don't want to serve the timestamps or file sizes for
every single file/resource in one response, for a number of reasons both
privacy-related and performance-related.

> I actually quite like this idea.  No need to make it timestamp-specific
> either.  We could have a well-known endpoint for general file metadata,
> which listed modification time, file size, checksum, MIME type, etc.  It
> could accept queries for a specific path, and *that* could be the way to
> do an equivalent of a HEAD request.

Yeah, it makes sense to bundle the metadata into one response, but only
serve for one path per request.

> This would let clients for specific scenarios do the extra work
> themselves to work around their problems, e.g. clients with very low
> memory or storage space could request the metadata for all files before
> attempting to access them and warn the user if the file size exceeds a
> threshold; clients on unreliable connections could request the metadata
> before downloading and then warn the user if file size and/checksum did
> not agree.  Most "normal" clients could do neither and just operate as
> they already do.

The one downside I see to that is that once we start seeing more complex
clients written, especially graphical clients, we will start seeing
progress bars, download managers, etc., and there will always be a need
to make a second request to get the metadata, which is relatively
expensive.

> I think it's kind of neat to keep solutions to edge problems outside of
> the protocol itself and push them into things like well-known endpoints
> like the above where they can easily be ignored when they are not
> needed/wanted.  The downside is that server developers have to do the
> work to add support for these things - but it's expected, I think, that
> servers are harder to write than clients.

My experience *so far* is that server implementation is easier. But that
is likely to change, I guess.

>>   As I mentioned in a private email to solderpunk earlier, one could always
>> take advantage of the sub-delimeters in the path portion.  I had at one
>> point mentioned using those to specify the prompt (otherwise the server
>> would return a status of 10):
>>
>> 	gemini://example.com/search;Search%20for
>>
>> This could be formalized:
>>
>> 	gemini://example.com/search;prompt=Search%20for
>> 	gemini://example.com/blogfeed;timestamp=2019-08-15T00:00:00Z
>> 	gemini://example.com/wildexample;prompt=Search%20for;timestamp=2019-08-
15:00:00:00Z?query=foo&usename=bar
>>
>>   So, you have "prompt" and "timestamp".  Others could be propsed.  If the
>> "timestamp" thing above is accepted, then you might want to have a new
>> status code meaning "no change" or "okay, but there's no content".

This is definitely re-inventing HTTP headers with a different format.

--
+----------------------------------------------------------------------+
| Jason F. McBrayer                                jmcbray at carcosa.net |
| The scalloped tatters of the King in Yellow must hide Yhtill forever.|

> We are, of course, re-inventing the Content-Length header here, which is
> an extremely useful header so that clients can do things like have a
> progress bar for downloads. If we support every useful feature, of
> course, we end up with basically HTTP 0.9.

Right, and that's what I'd like to avoid even taking one step toward
to doing, because unless we resist it hard from day one, I worry we'll
inevitably fall down a slippery slope to recreating HTTP.

In some sense, Gemini as-is chooses Content-Type as the "one true
header".  To some extent that needs justification, because there are
certainly other HTTP headers which don't do any evil and are genuinely
helpful.  I guess Content-Type was elevated because in my experience
with Gopher, it was the missing piece of information which caused the
most pain.  The lack of in-band content type signalling in Gopher
requires things like a custom URL scheme to encode that information,
which is a bit gross.
 
> IMO you probably don't want to serve the timestamps or file sizes for
> every single file/resource in one response, for a number of reasons both
> privacy-related and performance-related.

Yeah, putting every hosted resource in one response makes no sense -
that response could easily be larger than the one resource we were
interested in checking the timestamp or file size of.  The sensible
thing is to include the particular path you care about as a query.  If
that is absent, a sensible default might be to list the N most recently
updated resources, or the N most popularly requested?  But, really, I'm
not sure who would be making that kind of request or why.
 
> The one downside I see to that is that once we start seeing more complex
> clients written, especially graphical clients, we will start seeing
> progress bars, download managers, etc., and there will always be a need
> to make a second request to get the metadata, which is relatively
> expensive.

While they are not prohibited, using Gemini to transfer large binary
files (of the kind for which progress bars and download managers are
useful) is not a very smart idea.  There's no way to use compression to
speed things up, and there is no way to resume an interrupted download
with anything like a Range: header.  It's just not the right tool for
the job.  Thankfully, it's very easy for a Gemini document to link to
things hosted on other protocols, like https or even bittorrent, or one
of these newfangled P2P things the kids are using like IPFS, which are
more appropriate for that task.  I would say it makes sense to
recommend not using Gemini for files larger than, I dunno, 10MB?

The second request is definitely expensive, but the idea is not that
every client does this before every "real" request.  This "pseudo HEAD"
is mostly a support mechanism for clients with unreliable and/or slow
connections or with very limited disc space and/or memory.  I expect 95%
of clients to work exactly the way Gopher clients do, by just ignoring
all of this and generally being none the worse for it.

Well, maybe 95% is a bit strong.  Moku pona style aggregators might be
able to make good use of this, too.

Very slick clients could just request a resource directly, keep track of
how much response they've pulled down so far (or how many seconds
they've been reading for) and once some threshold is passed (1MB?  5
seconds?) make another request in a separate thread for the metadata and
then begin displaying a progress bar or ETA prompt or something to
reassure the user that things really are happening.

> My experience *so far* is that server implementation is easier. But that
> is likely to change, I guess.

I guess for a sufficiently simple server which only deals in 20, 40 and
50 that's true.
 
> This is definitely re-inventing HTTP headers with a different format.

It is - while the well-known endpoint for metadata is basically
reinventing a HTTP method with a different format.

Maybe we don't need to worry about this?  There's no shame in not being
able to match every single useful capability that HTTP has.  It should
have been clear from the outset we wouldn't be able to do that.

-Solderpunk

>>>> * Sean has pointed out in one of his RFCs that this means there is no
>>>>   way for a client to know whether or not a download completed
>>>>   successfully or was interrupted due to an accidentally dropped or
>>>>   even a maliciously severred connection

Does TLS not have a native way to handle this? I did a quick search and found
that either the client/server may send a close_notify message before
terminating a TLS connection.

    https://tools.ietf.org/html/rfc5246#section-7.2.1

> Very slick clients could just request a resource directly, keep track of
> how much response they've pulled down so far (or how many seconds
> they've been reading for) and once some threshold is passed (1MB?  5
> seconds?) make another request in a separate thread for the metadata and
> then begin displaying a progress bar or ETA prompt or something to
> reassure the user that things really are happening.

The whole idea is serving metadata in a separate TCP request might work fine
for static files, but it would break down when attempting to serve any type
of dynamically generated content.

Say I want to host an endpoint that does something like return a random quote
from fortune(1). Gopher sites love to serve these little CGI tools because
they're fun and require very little effort :) After I generate the text, I can
easily calculate the size and send along with the response in-line using
something like a response header. But I wouldn't be able to tell you ahead of
time from a separate metadata TCP request.

> Maybe we don't need to worry about this?  There's no shame in not being
> able to match every single useful capability that HTTP has.  It should
> have been clear from the outset we wouldn't be able to do that.

I agree with this sentiment. Knowing the file size would be nice to have, but I
haven't seen any use cases that are compelling enough to justify increasing the
complexity to the protocol for. Even in HTTP land, Content-Length is an optional
response header and there are several scenarios where it doesn't make sense to
include it at all.

- mozz

> Does TLS not have a native way to handle this? I did a quick search and found
> that either the client/server may send a close_notify message before
> terminating a TLS connection.
> 
>     https://tools.ietf.org/html/rfc5246#section-7.2.1

Ah, this is a great detail to have noticed!  I'll need to look into how
hard it typically is for clients to check that they've received this.
If it's straightforward, it's probably a good idea to amend the spec to
say that servers MUST send this close_notify message.  That would then
act as an analogue of Gopher's lone dot.

> I agree with this sentiment. Knowing the file size would be nice to have, but I
> haven't seen any use cases that are compelling enough to justify increasing the
> complexity to the protocol for. Even in HTTP land, Content-Length is an optional
> response header and there are several scenarios where it doesn't make sense to
> include it at all.

Yeah, I think we can leave this for now.  It was a hypothetical concern
that somebody had.  Not necessarily a bad one, but until it's observed
actually creating significant trouble for actual users on actual clients
I think we can just table this issue.  If it does come up as a practical
concern, we can resume discussion of some of the ideas here.

-Solderpunk

It was thus said that the Great solderpunk once stated:
> 
> Yeah, I think we can leave this for now.  It was a hypothetical concern
> that somebody had.  Not necessarily a bad one, but until it's observed
> actually creating significant trouble for actual users on actual clients
> I think we can just table this issue.  If it does come up as a practical
> concern, we can resume discussion of some of the ideas here.

  I had a thought last night in that there does, in fact, exist a way to
include the information without changing the syntax of the protocol.  The
MIME type that is returned can have parameters, and these parameters can act
as small headers, so:

20 text/plain; charset=iso-8859-1; size=1345
20 text/gemini; size=2003; modified=2019-08-18T17:15-05:00

  -spc

>   I had a thought last night in that there does, in fact, exist a way to
> include the information without changing the syntax of the protocol.  The
> MIME type that is returned can have parameters, and these parameters can act
> as small headers, so:
> 
> 20 text/plain; charset=iso-8859-1; size=1345
> 20 text/gemini; size=2003; modified=2019-08-18T17:15-05:00
>

Strictly speaking, those parameters are supposed to be limited to the
ones defined in the RFC for the corresponding type, right?

Not that we're necessarily playing by all the rules here.  text/gemini
is an unregistered, experimental type and so I think strictly speaking
we should be text/x.gemini or text/prs.gemini or something...

Don't tell the IETF!

-Solderpunk

It was thus said that the Great solderpunk once stated:
> >   I had a thought last night in that there does, in fact, exist a way to
> > include the information without changing the syntax of the protocol.  The
> > MIME type that is returned can have parameters, and these parameters can act
> > as small headers, so:
> > 
> > 20 text/plain; charset=iso-8859-1; size=1345
> > 20 text/gemini; size=2003; modified=2019-08-18T17:15-05:00
> >
> 
> Strictly speaking, those parameters are supposed to be limited to the
> ones defined in the RFC for the corresponding type, right?

  Nope.  From RFC-2045, section 5:

	MIME implementations must ignore any parameters whose names they do
	not recognize.

  Also, from the same section:

	The ordering of parameters is not significant.

> Not that we're necessarily playing by all the rules here.  text/gemini
> is an unregistered, experimental type and so I think strictly speaking
> we should be text/x.gemini or text/prs.gemini or something...

  Nope, RFC-6648 deprecates the whole "X-" thing.

  -spc

On Sun, Aug 18, 2019 at 11:36:41AM +0000, solderpunk wrote:
 
> Yeah, I think we can leave this for now.  It was a hypothetical concern
> that somebody had.  Not necessarily a bad one, but until it's observed
> actually creating significant trouble for actual users on actual clients
> I think we can just table this issue.  If it does come up as a practical
> concern, we can resume discussion of some of the ideas here.

Well, I guess this is no longer quite so hypothetical, as Konpeito has
given us large binary files over Gemini.  I noticed that this was an
unpleasant user experience to download via AV-98 (with the client
appearing to "hang" for a long time during the download).  And, in fact,
I did receive an email from one person telling me they liked Gemini a
lot, who mentioned that they had not been able to download the Konpeito
album.  I strongly suspect that, in fact, they just didn't realise their
client was slowly downloading it.

One possible solution to the user experience problem that doesn't
require the client actually knowing the file size is to display, instead
of a progress bar, some kind of "stuff is happening" indicator, e.g. an
animated spinner (you know, of the /, -, \, | kind) accompanied by a 
count of KiB / MiB downloaded so far?  That would at least make it
obvious to users that something was happening.

And a convention of specifying filesizes for large binary files in the
link text would allow rough mental calculation of progress?

Cheers,
Solderpunk