a space case for transparent gemtext compression

Attál gemininauts,

Gemini appeals to me as someone who likes plain text and keeping things
simple. It also evokes a huge nostalgia for the days of gopher, bbs and
slow packet radio links.
Suppose there were a few shielded and solar powered Raspberry Pico based
Gemini servers in orbit with radio transmitter and receiver. The long
distance bandwidth would be a scarce resource and timeouts can happen
frequently. There's HTTP has compression, BBS has zmodem, there's the Squid
proxy. Caching and compression have been discussed before. I want to focus
on compression as it surprises me as the remarks seem to be to dismissive
even though compression was a huge thing in the early days of popular data
networks (fidonet, bbs, simtel).

On March 10, 2021 Bradley D. Thornton remarks "neither of those two things
(Accessibility or gzip compression) have anything to do with, nor have any
place in discussions, for spec changes" and argues:
* "Gemini is intended (expected, rather) to deliver mostly small files"
* "Would compression be nice? - you betcha! but at 1Gbps it's not really
that big of a deal - at least for now"
 and Solderpunk remarked earlier on the 16th of January 2020:
"compression is not a bad thing, but for small content the benefit is not
proportionate to the implementation effort".

I'm not sure about that.
I agree that Gemini is not about efficient transfer of files, there's
bittorrent, IPFS and plain http for that. For me it's about transferring
text as simple and effectively as possible. As Gemini is also about
creativity and focus through limits, I want to to test the Gemini design in
constrained bandwidth situations for example at sea or in space with 1200
bps on a low power Raspberry Pico or perhaps even on the computer and radio
of the Gemini capsule in 1965. Loading twelve Kbyte SF story wouldn't load
instantaneous.
Fortunately Unicode can be compressed substantially. So I wondered if
compression could somehow be implemented easily without breaking the
minimalist design philosophy. Of course one could use Mosh and SSH to start
a terminal Gemini client on a host with higher bandwidth (or use lzo
compression in OpenVPN), but it would be elegant if the decompression
happens transparently in Gemini without using additional ssh or openvpn
servers.
It also would be a sport to get gemtext to the client as fast as
technically possible, ideally below the threshold of noticing there has
been a network transfer at all.

One simplistic idea is to check if the gmi is binary, and assume it has
been compressed with zstd and then automatically decompress it in the
client. It'd be just one extra step in the gemini client.
To assess thethe speed and data savings I compressed a short story by Gene
Wolfe with zstandard and lpaq9, a page or so, representative of an average
gemtext. The compression and decompression speeds are below 30ms here. I
didn't precompress with a dictionary (for added significant size reduction)
as that would mean detecting the language (from for example word
frequency). Ideally the gmi would be compressed one time on the server to
avoid compressing (and straining the solar powered pico on space) for each
request.

11390 xicygnus.gmi
4098 xicygnus.gmi.0.lpaq9
4855 xicygnus.gmi.13.zst
5146 xicygnus.gmi.3.zst

How would people solve such use cases elegantly and within the design
philosophy?

I tried to find out the data rates from the pictures of the Apollo moon
missions, but only found this:
"For a typical Apollo launch, Cape Kennedy is connected directly to the
Mission Control Center, Houston, by NASCOM's Apollo Launch Data System, a
combination of data gathering and transmission systems designed to handle
launch data exclusively. A high speed data (2400 bit per second) line
connects the Cape to Goddard. At Goddard the transmission rate is increased
to 40,800 per second rate from there to Houston."
http://web.mit.edu/digitalapollo/Documents/Chapter8/apollomsfn.pdf

kind regards,
Francis Siefken (NL)
gemini://fsiefken.srht.site

On 17/06/2021 14:24, Francis Siefken wrote:

> How would people solve such use cases elegantly and within the design philosophy?
> 

In my personal opinion, the best solution would be to simply have it as a 
separate MIME-type, something along the lines of "text/gemini+gzip". I 
don't recall how this works with MIME-types, but it should be something like this.

I do support your idea, though. I think compression would be a nice 
addition, but it doesn't belong in the protocol itself, but the file format.

Cheers,
~almaember

-- Unless you're replying to me on the Gemini mailing list,
reply to almaember@almaember.com instead.

Website: https://almaember.com/
Gemini capsule: gemini://almaember.com/
IRC: almaember on Libera.chat and tilde.chat

Hello,

Francis Siefken <fsiefken@gmail.com> writes:

> Attál gemininauts,
>
>snip<

> One simplistic idea is to check if the gmi is binary, and
> assume it has been compressed with zstd and then automatically
> decompress it in the client. It'd be just one extra step in
> the gemini client. To assess thethe speed and data savings I
> compressed a short story by Gene Wolfe with zstandard and
> lpaq9, a page or so, representative of an average gemtext. The
> compression and decompression speeds are below 30ms here. I
> didn't precompress with a dictionary (for added significant
> size reduction) as that would mean detecting the language
> (from for example word frequency). Ideally the gmi would be
> compressed one time on the server to avoid compressing (and
> straining the solar powered pico on space) for each request.
>
> 11390 xicygnus.gmi
> 4098 xicygnus.gmi.0.lpaq9
> 4855 xicygnus.gmi.13.zst
> 5146 xicygnus.gmi.3.zst
>
> How would people solve such use cases elegantly and within the
> design philosophy?

Well, imho you do not neccessarily need any changes to the
protocol or the gemtext specs.

Noone stops you from:

* writing an article (arbitrary size, consider it big for the
  argument)
  => filename.gmi
* compressing this file with xz, say
  => filename.gmi.xz
* adding an abstract which describes the article
  => filename-abstract.gmi
* adding a link to the abstract
  => filename.gmi.xz  read more [compressed]
* the abstract is added to your feed and/or index.gmi

The the user will decide whether to follow the link.
The client software might be able to uncompress the downloaded
file and display it similar to LaGrange displaying image as if
they were "inline".


Please bear in mind that the gemini protocol does not even
announce the length of the content to follow.


I would go the same route for extented text where I want to
control the presentation. Use \LaTeX, produce a .pdf file, offer
a download link in an plain/text abstract file, which is
indexed. No need for "complex machinery".

Cheers,
~ew


>snip<

> kind regards,
> Francis Siefken (NL)
> gemini://fsiefken.srht.site


-- 
Keep it simple!

On Thu, Jun 17, 2021 at 04:34:54PM +0200, ew.gemini wrote:
> Well, imho you do not neccessarily need any changes to the
> protocol or the gemtext specs.
> 
> Noone stops you from:
> 
> * writing an article (arbitrary size, consider it big for the
>   argument)
>   => filename.gmi
> * compressing this file with xz, say
>   => filename.gmi.xz
> * adding an abstract which describes the article
>   => filename-abstract.gmi
> * adding a link to the abstract
>   => filename.gmi.xz  read more [compressed]
> * the abstract is added to your feed and/or index.gmi
> 
> The the user will decide whether to follow the link.
> The client software might be able to uncompress the downloaded
> file and display it similar to LaGrange displaying image as if
> they were "inline".
> 
> 
> Please bear in mind that the gemini protocol does not even
> announce the length of the content to follow.
> 
> 
> I would go the same route for extented text where I want to
> control the presentation. Use \LaTeX, produce a .pdf file, offer
> a download link in an plain/text abstract file, which is
> indexed. No need for "complex machinery".
> 
> Cheers,
> ~ew
> 

This is basically how I'd go, although I'd lean towards gzip over xz for
being much kinder to low-memory systems.  Anything that can't be done on
a 68030 or an 80386SX doesn't belong in any of the core Gemini
standards, or even de-facto standards like how to serve compressed
gemtext. 

See this StackOverflow posting for more details on the justification.

https://stackoverflow.com/questions/6493270/why-is-tar-gz-still-much-more-c
ommon-than-tar-xz

-- 
tidux@sdf.org
SDF Public Access UNIX System - http://sdf.org

Hi ew, thanks for coming up with an interesting alternative:

* writing an article (arbitrary size, consider it big for the
>   argument)
>   => filename.gmi
> * compressing this file with xz, say
>   => filename.gmi.xz
> * adding an abstract which describes the article
>   => filename-abstract.gmi
> * adding a link to the abstract
>   => filename.gmi.xz  read more [compressed]
> * the abstract is added to your feed and/or index.gmi
>

But in this flow there is additional user choice involved to 'read more'.
It's similar to gopher, the text is plain text, anything else such as
compressed text is designated as a binary file.
In my personal preference as a gemini reader, if it's basically just a page
or a few pages of text I'd like to read it all immediately.
Perhaps it's a habit from the www. But it's similar to teletext, when I
choose a number I expect to see the complete teletext page, even though
teletext is just a few lines (which won't really benefit from compression
even in 300 baud situations).


> The client software might be able to uncompress the downloaded
> file and display it similar to LaGrange displaying image as if
> they were "inline".
>

I just tried it, amfora asks me to open or download the
xicygnus.gmi.gz file.
I choose open; the os recognizes the *.gz extension and loads the default
decompression extraction app.
In this extraction app I have to manually press extract, choose an
extraction location and when it's extracted I can opt to open the file
location and then double click the gmi file which it opens in a plain text
editor, not in amfora.
That's not 'keep it simple'. While it's possible that one day there's a
gemini browser which recognizes the gmi.gz and automatically decompresses
it, this is't working now - and why would it? In firefox and chrome it's
the same behaviour unless one would write a javascript that would handle it
automatically - but that doesn't fly in gemini space.
Alternatively one could reconfigure it client side to automatically
decompress and display client side, with the back button or key working -
but this you cannot enforce as the one serving the gemini capsule.

regards,
Francis (NL)
gmni://fsiefken.srht.page

On 18 Jun 2021, at 6:27 am, Francis Siefken <fsiefken@gmail.com 
<mailto:fsiefken@gmail.com>> wrote:
> Alternatively one could reconfigure it client side to automatically 
decompress and display client side, with the back button or key working - 
but this you cannot enforce as the one serving the gemini capsule.

Right. There is nothing strange or wrong about returning compressed data 
with a suitable MIME type, and it is logical for a client to pass the 
binary blob on to the default extractor app, as happened in your test.

To me, this line of the spec rules out decompression as a feature of the 
Gemini client itself:

> Response bodies are just raw content, text or binary, ala gopher. There 
is no support for compression, chunking or any other kind of content or 
transfer encoding.

As with any home-grown extension you can certainly implement it for your 
own purposes. I just don't see a way forward for widespread compression 
support when it's been explicitly scoped out.

Since you mentioned 1200 bps and radios in your original post, I will 
remind the list that Gemini is unsuitable as a protocol for any amateur 
radio operations due to the encryption. I used gopher for that purpose instead.

Cheers, Tom

Get ready for a wall of text.

On Thu, Jun 17, 2021 at 02:24:45PM +0200, Francis Siefken wrote:
> How would people solve such use cases elegantly and within the design philosophy?

Compression could be valuable to users with poor download speeds, 
especially those using an overlay network like Tor with a poor uplink.

I do like the idea of compression being optional; if a client supports 
compression it can get a Gemini file compressed, but otherwise a plain 
version. However, this does create a huge problem: clients would have to 
declare that they support compression, which opens up a can of worms 
(complexity, fingerprinting, etc.) that we should stay away from. 
Almaember's approach is much better:

On Thu, Jun 17, 2021 at 03:47:48PM +0200, Almaember wrote:
> In my personal opinion, the best solution would be to simply have it as 
a separate MIME-type, something along the lines of "text/gemini+gzip". I 
don't recall how this works with MIME-types, but it should be something like this.
> 
> I do support your idea, though. I think compression would be a nice 
addition, but it doesn't belong in the protocol itself, but the file format.

I think the best solution is:

- Have clients optionally support pagination, like what most line-mode 	
clients (gmnlm, cgmnlm, diohsc) do. Show the first N lines/bytes 	instead 
of downloading the whole thing; let the user scroll to trigger 	
downloading the rest. This probably should not be the default setting, 	
but that's up to client developers to decide.
- For big files, have authors link a compressed version: "Dear reader, 	
this gemini file is large. Here's a link to a compressed version:"
- Users can then select the link before they've finished 	
downloading/paging through the file.

This therefore doesn't need to be part of the Gemini "standard" but can 
simply be a recommendation for authors and devs. If it gains a lot of 
traction, perhaps it could be formalized using the word "MAY" in the spec 
("clients MAY also support the mimetype...").

Regarding a compression algorithm to pick: it should be one that's fairly 
common with a lot of libraries/implementations for a variety of platforms 
and programming languages, keeping with the rationale for choosing TLS 
1.2+ over other options for transport-layer encryption. Although I'd love 
to pick Lizard for its speed, it's not universal enough to qualify.

=> https://github.com/inikep/lizard Lizard (formerly LZ5)

Our best options are therefore gzip/DEFLATE and perhaps zstd. Gzip can 
actually get pretty small when compressing statically/ahead-of-time, where 
compression speed is less of a constraint. Tools like Zopfli and 
especially Efficient-Compression-Tool can get a dump of all posts on 
seirdy.one (~100kb) 41.1 kb, compared to 39.9kb with zstd -f19. The 
difference only becomes significant with Gemini files above 200kb.

=> https://github.com/fhanau/Efficient-Compression-Tool

I therefore vote for the following *non*-standard:

- Encourage authors of 50kb+ gmi files to link a text/gemini+gzip 	
somewhere near the top.
- Encourage client developers to consider supporting pagination (can be 	
optional and off by default), so only the first N bytes/lines are 	
downloaded before the user performs an action.
- Encourage client developers to support the text/gemini+gzip MIME type

Only after this gains traction and is well-received by users with 
constrained bandwidth should consider adding a "MAY" statement to the 
Gemini spec describing the text/gemini+gzip MIME type. Compression support 
should never be required.

-- /Seirdy (seirdy.one)

On Thu, Jun 17, 2021 at 08:20:08PM -0700, Rohan Kumar wrote:
> Only after this gains traction and is well-received by users with
> constrained bandwidth should consider adding a "MAY" statement to the Gemini
> spec describing the text/gemini+gzip MIME type. Compression support should
> never be required.

I thought the spec was finalized and not designed to be extended. I can
see clients treating gzip files specially though, no different than
lagrange supporting mp3s.

Lagrange already supports opening Zips and browsing gemtext from it. So... 
there you go. There's your compression.

If you want different compression method, just tell gemini browsers to 
support it. This isn't against the spec, and it's actually very simple to 
use. In fact, gempubs are basically this same thing - they are zips with 
gemtext files inside of them. When you click on a gempub link, lagrange 
will literally have a button saying "Download and Open" and it'll download 
and open it right away for browsing. Other browsers can make this just as easy to do.

Was wondering when that would get brought up. Hadn't spoken because I 
wasn't sure if it hadn't already been looked into.

Gwmpub/gpub is readable by Lagrange, and hopefully other browsers will 
follow suit (especially Ariana or another mobile browser.) 

Is there a reason this wouldn't work though?

I think people would probably want either a different compression, or to 
just compress one file instead of a whole bunch of files. So we'd probably 
need to come up with a different format for compression of just single 
files. And this is actually *easier* to support than supporting zips, etc, 
because those are more like compressed directories that you can navigate within.

Anyways, for people wondering about the gempub project, it's the 
equivalent of both epubs, but with gemtest, and also has the goal of being 
used for gemini capsule archives. The spec for gempubs is here, if anyone 
wants to contribute. You can perhaps suggest this to other browsers to 
support: https://codeberg.org/oppenlab/gempub

I do want to note that the gemini protocol doesn't care about what's send  
over it. This is why mimetypes were added to the protocol. You can send 
any binary data over gemini. This is why compression doesn't need to be in 
the main protocol - because you can send over the compressed stuff just as 
any other binary file can. It's the job of the gemini browser to handle 
the compressed data by reading the mimetype and dealing with that mimetype 
in some way. And if it's too complicated, then tell your browser 
maintainer and they can fix it if they want. If they don't want to, then I 
would consider looking into other gemini browsers that do support what you want.

Finally, the other consideration is whether supporting this within clients 
adds too much complexity. Personally, I do not think so, as there are tons 
of very simple libraries you can use for supporting compression. Golang 
even has these in its standard library.

Some gemini browser maintainers are probably going to argue that it's not 
the purpose of a browser to support opening zips, as a similar argument 
has been used for Gemini Subscription Feeds, Atom Feeds, audio/video 
files, and other such things. I would have to completely disagree. The 
point of a browser is to browse documents - that's the scope of gemini browsers.

I do recognize that there are, however, two different approaches to this. 
Have the "gemini browser" be a full document browser for the gemini 
protocol, or have it just deal with the gemini protocol and pass the 
documents off to another program. The problem, of course, is many 
applications don't support gemini links, which is definitely why I prefer 
the 1st approach, which seems more like the approach Lagrange takes 
(Skyjake can correct me if I'm wrong). The point is, the point of a 
browser is to browse geminispace. Which means you need to either support 
things yourself, or have the ability to pass them off to another 
application that *can* support them properly.

On 2021-06-18 15:22+02:00, Christian Seibold wrote:
> I think people would probably want either a different compression, or to 
just compress one file instead of a whole bunch of files.

> So we'd probably need to come up with a different format for compression 
of just single files.
gzip by itself only ever compresses single files, leading to the common 
.tar.gz combination, where the tape archive (tar) contains any directory structure.

The MIME type text/gemini+gzip has been suggested for this earlier. [1] I 
think that would be a good solution although it is questionable how well 
browsers check the MIME type and if they would mistake "text/gemini+gzip" 
for plain "text/gemini", leading to some confusion. The +gzip suffix is 
registered by IANA [2].

[1] gemini://gemi.dev/gemini-mailing-list/messages/006710.gmi
[2] https://www.iana.org/assignments/media-type-structured-suffix/media-typ
e-structured-suffix.xml

Hi Christian, yes, there's my compression - but it's not transparent: it
takes 4 user actions. So unless a client automatically opens and displays a
gzipped gemtext there is no way to have a near instantly readable and
extracted compressed index.gmi. Even if it did I am not sure if the back
button works at once, as now I also have to hit the back 4 times after
downloading and displaying a gzipped gemtext.

It's not that I want a different compression codec as the usual suspects
(zstd, gz, zip) all have similar ratios (40-50%), although i'd like the
most efficient text compression code with a high speed decompression. For
example ppmd with for example dictionary wrt preprocessing to get to ~25%.

Cheers,
Francis
gemini://fsiefken.srht.site

The last person who emailed on the mailing list was Johann Galle, not me.

Please read my own response again. If you're using zips, Lagrange doesn't 
take 4 user actions, it takes 1. If other browsers don't have this 
functionality, then tell them. Browsers should implement this, not the 
protocol. It's that's simple.

On Fri, Jun 18, 2021 at 08:22:32AM -0500, Christian Seibold wrote:

Hi!

> Finally, the  other consideration is whether  supporting this within
> clients adds too much complexity. Personally,  I do not think so, as
> there are tons  of very simple libraries you can  use for supporting
> compression. Golang even has these in its standard library.

This is how bloated software starts.

>  Some gemini  browser maintainers are  probably going to  argue that
> it's not  the purpose  of a  browser to support  opening zips,  as a
> similar argument has  been used for Gemini  Subscription Feeds, Atom
> Feeds, audio/video  files, and  other such things.  I would  have to
> completely disagree. The point of a browser is to browse documents -
> that's the scope of gemini browsers.  I do recognize that there are,
> however, two different approaches to this. Have the "gemini browser"
> be a full document browser for  the gemini protocol, or have it just
> deal with the gemini protocol and  pass the documents off to another
> program. The problem, of course,  is many applications don't support
> gemini links, which is definitely why I prefer the 1st approach,

So are you  arguing that gemini client should include  a postscript or
PDF interpreter?  Supports HTML?  JavaScript?  There are libraries for
those in the most common languages.

> which  seems more  like  the approach  Lagrange  takes (Skyjake  can
>  correct me if I'm wrong).

Please this  is not the place  to discuss a single  implementation for
gemini standard.

The fact is  that or you are proposing a  standard extension, and just
let me  write that i am  tired that about  60% of the traffic  in this
mailing list is  about proposing extension to the  protocol that still
needs works  to be finalized, instead  of proposing fix (there  is the
gitlab  repo for  that).  If  you  do not  like gemini  propose a  new
protocol and  i likely be  the first  to works on  that if i  think is
good.  Or if  this is not extension  (and i disagree it is  not, to be
clear) it is a discussion  about a single implementation which produce
noise in this ml.

Moreover  i  think  that  promoting a  "fat"  client  promotes  client
mono culture as  well. Complicated clients need  more resources (people
or money) and lead to monopolistic behaviour (even in good faith).

To be  clear. No if  lagrange implements X  this does not  means every
other  client should  follows  the  same path.  In  my  opinion if  an
implementation  follows  the  path  you are  proposing  (bloating  the
software) this,  in my opinion,  is a  reason to not  implements those
(anti)features.

To reuse (badly) a  metaphor that is used often these  days, to me you
are  proposing tho  transform an  old  boring tree  that helps  people
meditating  with  a  shiny  Christmas  tree  with  colored  light  and
decoration which only purpose is to  remember us to spend our money in
the  commercial center  nearby  until  it is  disposed  of, after  the
season! :)

What, in my opinion, needs gemini  now is not extension but fixing the
standard, new contents  a lot of propaganda, i think  help of everyone
is appreciated.

Bye!
C.

PS: as usual, sorry for my bad English!

> This is how bloated software starts.

Who says what is bloated? How much features do we have to add for 
something to be bloated? Was adding TLS bloat? TLS added a bunch of 
complexity to the protocol. What about gemtext? Is gemtext bloat?

This isn't black and white. Bloat is a ratio of various things, including 
complexity added, size added, and performance effects vs. how much the 
feature adds to the program and its usefulness - it's "power to weight ratio"

> 
> >  Some gemini  browser maintainers are  probably going to  argue that
> > it's not  the purpose  of a  browser to support  opening zips,  as a
> > similar argument has  been used for Gemini  Subscription Feeds, Atom
> > Feeds, audio/video  files, and  other such things.  I would  have to
> > completely disagree. The point of a browser is to browse documents -
> > that's the scope of gemini browsers.  I do recognize that there are,
> > however, two different approaches to this. Have the "gemini browser"
> > be a full document browser for  the gemini protocol, or have it just
> > deal with the gemini protocol and  pass the documents off to another
> > program. The problem, of course,  is many applications don't support
> > gemini links, which is definitely why I prefer the 1st approach,
> 
> So are you  arguing that gemini client should include  a postscript or
> PDF interpreter?  Supports HTML?  JavaScript?  There are libraries for
> those in the most common languages.

Nope, I never suggested such a thing. HTML isn't used in geminispace. 
Neither is javascript. Additionally, again, you have to look at the pros 
and cons of things. Javascript adds significant performance and security 
problems. Compression **doesn't**.
Stop trying to compare compression to Javascript and HTML, especially when 
compression is very basic and doesn't add barely any LOC or performance decreases.

Let's look at an lz4 library:
https://github.com/lz4/lz4

It's 49000 MB/s on a Core i7-9700K CPU @ 4.9GHz.
And it's 5699 LOC.

Btw, this is the approach that Solderpunk used to determine what should go 
into the protocol. It's right on the front page of the gemini capsule:
"Strives for maximum power to weight ratio"

Imo, compression fits perfectly with this. Some people will disagree, and 
that's fine. That's not gonna change my opinion on this matter though. All 
I ask is that you stop using personal attacks, because it doesn't help your argument.

> 
> > which  seems more  like  the approach  Lagrange  takes (Skyjake  can
> >  correct me if I'm wrong).
> 
> Please this  is not the place  to discuss a single  implementation for
> gemini standard.

Yes it is. The OP was asking for a specific feature, and I suggested a 
client that had that feature that they can use. I NEVER suggested every 
single client should implement compression, and I actually said the opposite:
"Which means you need to either support things yourself, or have the 
ability to pass them off to another application that *can* support them properly."

Passing things off to other applications is just as valid of a way to deal 
with this, and it's what AV-98 and various other clients do.

> 
> The fact is  that or you are proposing a  standard extension, and just
> let me  write that i am  tired that about  60% of the traffic  in this
> mailing list is  about proposing extension to the  protocol that still
> needs works  to be finalized, instead  of proposing fix (there  is the
> gitlab  repo for  that).  If  you  do not  like gemini  propose a  new
> protocol and  i likely be  the first  to works on  that if i  think is
> good.  Or if  this is not extension  (and i disagree it is  not, to be
> clear) it is a discussion  about a single implementation which produce
> noise in this ml.

I in no way proposed an extension to the protocol. In fact, I proposed the 
exact opposite, that this should stay *out* of the protocol, and be in 
client applications for people who want to support this:
"I do want to note that the gemini protocol doesn't care about what's send 
over it. This is why mimetypes were added to the protocol. You can send 
any binary data over gemini. This is why compression doesn't need to be in 
the main protocol - because you can send over the compressed stuff just as 
any other binary file can."

> 
> Moreover  i  think  that  promoting a  "fat"  client  promotes  client
> mono culture as  well. Complicated clients need  more resources (people
> or money) and lead to monopolistic behaviour (even in good faith).

People are smart enough to decide for themselves what client they want to 
use. I'm allowed to promote a client. I think it's very presumptuous to 
think I'm somehow "propagandizing" people into supporting Lagrange. It's 
ridiculous. People are smart enough to decide for themselves what client 
they want to use.

> 
> To be  clear. No if  lagrange implements X  this does not  means every
> other  client should  follows  the  same path.  In  my  opinion if  an
> implementation  follows  the  path  you are  proposing  (bloating  the
> software) this,  in my opinion,  is a  reason to not  implements those
> (anti)features.

I didn't suggest any such thing, as I've already said above.

On Sat, Jun 19, 2021 at 02:15:45AM -0500,
 Christian Seibold <krixano@mailbox.org> wrote 
 a message of 3 lines which said:

> If you're using zips, Lagrange doesn't take 4 user actions, it takes
> 1. If other browsers don't have this functionality, then tell
> them. Browsers should implement this, not the protocol. It's that's
> simple.

If I understand correctly your proposal, you suggest to change,
neither the protocol nor the gemtext format, but to push browser
authors to recognize text/gemini+gzip and act accordingly.

In that case, I see two issues:

* how the end user of a browser without compression support will know
in advance if the file is compressed or not? From the file extension?
It is not always present.

=> ilovecompression.gmiz I cannot read it
=> plain.gmi I can read it

* today, every browser can read every Gemini file. Your proposal
seems to imply that some users will now see only a part of the
Geminispace, the uncompressed one. This is the same problem we have
when trying to use the Web with lynx or dillo: we can see only a part
of it.

> To reuse (badly) a  metaphor that is used often these  days, to me you
> are  proposing tho  transform an  old  boring tree  that helps  people
> meditating  with  a  shiny  Christmas  tree  with  colored  light  and
> decoration which only purpose is to  remember us to spend our money in
> the  commercial center  nearby  until  it is  disposed  of, after  the
> season! :)

Btw, soft Christmas lights can help many people, including some autistic 
people, as it can be used as a stimulant to calm them down.

What you consider useless or "commercial" some people might consider 
beautiful and a way to connect and relax and spend time with their families.

Finally, light, especially candles, has been used very often across 
history for meditation, for centuries. It's used in Judaism, Christianity, 
Yoga, Islam, various eastern religions, etc.

Again, this is all about perspective.

Hi Christian, on Sa 19 jun. 2021 09:15 Christian Seibold <
krixano@mailbox.org> you wrote:

> The last person who emailed on the mailing list was Johann Galle, not me.

I know, in my previous message replied specifically to your message vr 18
june 14:24. Is there something wrong with that?

> Please read my own response again.
[..]
> If you're using zips, Lagrange doesn't take 4 user actions, it takes 1.

I understood what you meant yesterday. But in my test with Langrange 1.5.2
and gmnisrv it took 4 steps. With the reference gpub (excellent choice) it
takes 3 steps and it opens an addititional tab. With a gmi zip it says:
"xi.zip is a compressed archive. You can save it as a file to your
Downloads folder: press Ctrl+S or select "Save to Downloads" from the
menu." And then a whole lot more steps. I wish it would take zero steps and
a client would for example automatically gunzip a gmi.gz and display it
inline immediately by default.

> Browsers should implement this, not the protocol. It's that's simple

Will it be? For example Asuka, Amfora, Ariane and Lagrange are nice
browsers but if I read the discussion so far I am sure it will be
implemented in the way I envision. Images and sounds are handled
differently and take at least one or a few user actions. What I hope and
wish for is that one day my Gemini capsule can serve index.gmi.zstd or
index.gmi.ppmd transparently. I trust they can be automatically
decompressed inline in the same tab by visiting browsers, either through
the mime-type route, TLS compression or through a Gemini Content-Encoding
and zstd or ppm server side compression.
You mentioned the dogma browsers should implement this and not the
protocol. I am curious as to what your reason for this is other then that
gemini protocol is supposedly set in stone?
Protocol support has been discussed on list before. On March 10, 2021
Bradley D. Thornton had a similar remark "neither of those two things
(Accessibility or gzip compression) have anything to do with, nor have any
place in discussions, for spec changes" and argued:
* "Gemini is intended (expected, rather) to deliver mostly small files"
* "at 1Gbps it's not really that big of a deal"
 and Solderpunk wrote 16th of January 2020:
* "compression is not a bad thing, but for small content the benefit is not
proportionate to the implementation effort".

But both arguments do not apply in low bandwidth situations (in a forest or
in space), even with an 2000 byte mail like mine you get at least a 50%
reduction which in some situations in space or remote places could
theoretically mean life or death. IMAP has compression, HTTP has
compression (for examplee Content-Encoding: zstd, rfc8478). How hard is it
really, a few extra lines in the server?
I'd think it's not much harder then getting mime-type handling consistently
implemented in Gemini browsers (which in my original post means zero steps
for compressed gemtext). TLS 1.3 has compression on board, it's a security
risk in HTTP session hijacking and all HTTP browsers have turned it off.
But this security risk doesn't apply to Gemini or with SMTP or IMAP. This
would be another way to making this compressed gemtext vision work. Even in
Vinton Cerf's interplanetary DTN bundling protocol compression plays a
role, so why not in the space inspired Gemini protocol if it can be done
easily? And if not, of course I appreciate the basic answer of keeping it
simple. But how simple is the proposed alternative really? The Gempub
doesn't load immediately and I suspect there will always be the extra
download and open step. Of course I can send patches, fork browsers or roll
my own gemini client and encourage people to additionally compress their
individual gemtexts with PPMd and their capsule with Gempub.

regards,
Francis Siefken (NL)
gemini://fsiefken.srht.site

> I know, in my previous message replied specifically to your message vr 
18 june 14:24. Is there something wrong with that?

It looked like you were replying to Johann Galle, not me, as I didn't 
mention the specific compression you were talking about - I deliberately 
kept my message more general. I did, however, read about what you wanted.

> The Gempub doesn't load immediately and I suspect there will always be 
the extra download and open step. Of course I can send patches, fork 
browsers or roll my own gemini client and encourage people to additionally 
compress their individual gemtexts with PPMd and their capsule with Gempub.

Yes, **currently** it takes multiple steps for lagrange to open a zip, but 
my point is it CAN be instant without having to implement it in the 
protocol. It absolutely *can* be implemented in client so that it's instant.

Anyways, you can make suggestions to skyjake on Station or on the lagrange 
github (https://github.com/skyjake/lagrange) so that it can be made more 
instant. There will always be 1 extra step (the "download and open" 
button) due to the fact that zips are more like a compressed directory - 
that is unless we figure out how to fix the url handling of things within 
zips. However, compressed single files can absolutely be instantly 
decompressed by a client that implements this.

Also, if TLS 1.3 has compression (thanks for telling me that, I didn't 
know that), then it makes supporting specific compressions in clients 
unnecessary, because we are going to be switching to TLS 1.3 in the 
future, from what I've gathered.

On Sat, Jun 19, 2021 at 03:42:26AM -0500, Christian Seibold wrote:
> > This is how bloated software starts.
>
> Who says what is bloated? How much features do we have to add for 
something to be bloated? Was adding TLS bloat? TLS added a bunch of 
complexity to the protocol. What about gemtext? Is gemtext bloat?

There are reasons why TLS is in gemini and compression not, please reread the FAQ.

> This isn't  black and  white.

I agree it is not.

> Bloat  is a  ratio of  various things,
> including  complexity added,  size  added,  and performance  effects
> vs. how  much the feature adds  to the program and  its usefulness -
> it's "power to weight ratio"

I agree.

>
> >
> > >  Some gemini  browser maintainers are  probably going to  argue that
> > > it's not  the purpose  of a  browser to support  opening zips,  as a
> > > similar argument has  been used for Gemini  Subscription Feeds, Atom
> > > Feeds, audio/video  files, and  other such things.  I would  have to
> > > completely disagree. The point of a browser is to browse documents -
> > > that's the scope of gemini browsers.  I do recognize that there are,
> > > however, two different approaches to this. Have the "gemini browser"
> > > be a full document browser for  the gemini protocol, or have it just
> > > deal with the gemini protocol and  pass the documents off to another
> > > program. The problem, of course,  is many applications don't support
> > > gemini links, which is definitely why I prefer the 1st approach,
> >
> > So are you  arguing that gemini client should include  a postscript or
> > PDF interpreter?  Supports HTML?  JavaScript?  There are libraries for
> > those in the most common languages.
>
> Nope, I never suggested such a thing. HTML isn't used in geminispace.

It is,  because as you  correctly said,  the protocol with  mime types
allow for any type of document.

>Neither is javascript. Additionally, again, you have to look at the pros 
and cons of things. Javascript adds significant performance and security 
problems. Compression **doesn't**.

So are you OK, with PDF and postscript that you left out?

> Stop trying to compare compression to Javascript and HTML, especially 
when compression is very basic and doesn't add barely any LOC or performance decreases.

You you know that compression "barely" add a few effort for every possible 
programming scenario?

> Let's look at an lz4 library:
> https://github.com/lz4/lz4
>
> It's 49000 MB/s on a Core i7-9700K CPU @ 4.9GHz.
> And it's 5699 LOC.
>
> Btw, this is the approach that Solderpunk used to determine what should 
go into the protocol. It's right on the front page of the gemini capsule:
> "Strives for maximum power to weight ratio"

Yes, please reread the FAQ. Compression is not in the standard for reasons.

> Imo, compression fits perfectly with this. Some people will disagree, 
and that's fine. That's not gonna change my opinion on this matter though. 
All I ask is that you stop using personal attacks, because it doesn't help your argument.

You tell me that i am presumptuous and accuse me for personal attacks?

>
> >
> > > which  seems more  like  the approach  Lagrange  takes (Skyjake  can
> > >  correct me if I'm wrong).
> >
> > Please this  is not the place  to discuss a single  implementation for
> > gemini standard.
>
> Yes it is. The OP was asking for a specific feature, and I suggested a 
client that had that feature that they can use. I NEVER suggested every 
single client should implement compression, and I actually said the opposite:
> "Which means you need to either support things yourself, or have the 
ability to pass them off to another application that *can* support them properly."

I quote another your message:

"Please read  my own  response again. If  you're using  zips, Lagrange
doesn't take 4 user actions, it  takes 1. If other browsers don't have
this functionality,  then tell  them. Browsers should  implement this,
not the protocol. It's that's simple."

And a  few lines  above: "...why  I prefer the  1st approach"  [a full
gemini document browser].

> Passing things off to other applications is just as valid of a way to 
deal with this, and it's what AV-98 and various other clients do.

But  you prefer  a full  gemini browser,  right? And  clients *should*
implement a form of decompression one way or another, right?

> >
> > The fact is  that or you are proposing a  standard extension, and just
> > let me  write that i am  tired that about  60% of the traffic  in this
> > mailing list is  about proposing extension to the  protocol that still
> > needs works  to be finalized, instead  of proposing fix (there  is the
> > gitlab  repo for  that).  If  you  do not  like gemini  propose a  new
> > protocol and  i likely be  the first  to works on  that if i  think is
> > good.  Or if  this is not extension  (and i disagree it is  not, to be
> > clear) it is a discussion  about a single implementation which produce
> > noise in this ml.
>
> I in no way proposed an extension to the protocol. In fact, I proposed 
the exact opposite, that this should stay *out* of the protocol, and be in 
client applications for people who want to support this:
> "I do want to note that the gemini protocol doesn't care about what's 
send   over it. This is why mimetypes were added to the protocol. You can 
send any binary data over gemini. This is why compression doesn't need to 
be in the main protocol - because you can send over the compressed stuff 
just as any other binary file can."
>
> >
> > Moreover  i  think  that  promoting a  "fat"  client  promotes  client
> > mono culture as  well. Complicated clients need  more resources (people
> > or money) and lead to monopolistic behaviour (even in good faith).
>
> People are smart enough to decide for themselves what client they want 
to use. I'm allowed to promote a client. I think it's very presumptuous to 
think I'm somehow "propagandizing" people into supporting Lagrange. It's 
ridiculous. People are smart enough to decide for themselves what client 
they want to use.

i  like lagrange  a  lot, this  is neither  about  lagrange nor  about
compression, this is about pushing for complex client that could break
the geminispace.

> >
> > To be  clear. No if  lagrange implements X  this does not  means every
> > other  client should  follows  the  same path.  In  my  opinion if  an
> > implementation  follows  the  path  you are  proposing  (bloating  the
> > software) this,  in my opinion,  is a  reason to not  implements those
> > (anti)features.
>
> I didn't suggest any such thing, as I've already said above.

And i can requote your other message as above.

Please reread the Bortzmeyer reply, he is wiser than me and so is able
to explain where the problem is better than me.

Also i encourage  *everyone* that cares for gemini  (including you) to
helps to finalize the standard,  not advocating for re-implementing web
browser with an unfit protocol.

Bye!
C.

> If I understand correctly your proposal, you suggest to change,
> neither the protocol nor the gemtext format, but to push browser
> authors to recognize text/gemini+gzip and act accordingly.

Ok, so, as you have probably read before :D , there's a divide on whether 
compression should even be put into the protocol or not. There's even a 
divide on whether it should be put into any clients - of course nobody can 
control every client and force them to do what they want it to do. So, 
realistically, I don't think we can get everyone to agree on a decision 
about compression being put into the protocol, especially when the 
protocol is almost finalized.

However, users are allowed to suggest things for browsers, and browser 
developers are allowed to reject suggestions. So, as we cannot get it in 
the protocol, then some client developers might decide to implement it in 
their clients, and it's totally possible to do this. This is really the 
next best step that we could take, assuming we won't reach a consensus on 
putting it into the protocol.

However, if what was said before was correct about TLS 1.3 having 
compression, all of this might be completely unnecessary when we switch to TLS 1.3.


> * how the end user of a browser without compression support will know
> in advance if the file is compressed or not? From the file extension?
> It is not always present.

This is a good point about client design. I think clients should tell the 
user the mimetype of files that are unrecognized by the client. I would 
also personally like that they allow the user to open the file in another 
program, kinda like AV-98's hooks, basically.

> * today, every browser can read every Gemini file. Your proposal
> seems to imply that some users will now see only a part of the
> Geminispace, the uncompressed one. This is the same problem we have
> when trying to use the Web with lynx or dillo: we can see only a part
> of it.

I mean, right. But what else can we do if we cannot reach a consensus on 
putting compression in the protocol? Solderpunk could just make the 
decision anyways, but that's dependent on him. The next best thing is for 
clients that want to provide this functionality to do so, and for clients 
that don't - they might or might not allow you to open it in a different program.

We also kinda already have this problem with various filetypes. There's 
just too many filetypes to support them all in clients, so sending them 
off to be handled by another program can help resolve this.

I'm a lurker, but this discussion requires some commentary from the sidelines:

On 6/19/21 11:24 AM, Christian Seibold wrote:
> Btw, soft Christmas lights can help many people, including some autistic 
people, as it can be used as a stimulant to calm them down.
As a personed diagnosed on the autistic spectrum that cannot *stand* 
"christmas lights", especially the "animated" or blinking kind, for a 
multitude of reasons, I request you refrain from dragging my "tribe" into 
this discussion to support your arguments. It is neither valid as an 
argumentative strategy, nor is it particularly welcome.

> What you consider useless or "commercial" some people might consider 
beautiful and a way to connect and relax and spend time with their families.
Noone thinks of themselves as the villain.

> Finally, light, especially candles, has been used very often across 
history for meditation, for centuries. It's used in Judaism, Christianity, 
Yoga, Islam, various eastern religions, etc.
Go worship in the church of feature-creep then, but leave the atheists 
alone with your "traditions".

> Again, this is all about perspective.

My perspective is:
- Proposing an extension to the protocol is obviously against the core idea of gemini.
- Proposing clients support an additional mime-type will lead to 
fracturing gem-space and is to be avoided. I for one will actively avoid 
any client that does this, it is *exactly* the kind of bloated "useful" 
features absent in gemini I find attractive.

Robert

You are making a bunch of assumptions about what I do or don't want based 
on no evidence and reading into some of the responses that I have given.

I never said clients *must* implement compression. I said it's up to the 
client developers if they want to or not, and that users are allowed to 
make suggestions. If a user doesn't have the features they want, they can 
easily move to a client that does have the features they want.

I respect solderpunk's decision to not include it into the protocol, but 
adding compression to clients in no way "breaks" geminispace, and it 
doesn't disobey the spec either. So your arguments there are completely invalid.

Let me repeat this... adding compression to clients in the way that has 
been described in this thread does NOT break geminispace. Gemini 
deliberately allows you to send whatever you want over. It's a 
file-transfer protocol. Using the mimetype is *exactly* what I proposed. 
Period. End of discussion.

> Also i encourage  *everyone* that cares for gemini  (including you) to
> helps to finalize the standard,  not advocating for re-implementing web
> browser with an unfit protocol.

Compression is NOT a web thing, and it predates the web, so knock that 
crap off. Not everything the web has remotely touched is bad. TLS is used 
by the web. It's not suddenly "re-implementing" the web by using TLS. 
Adding compression is also not "re-implementing" the web.

This is the final message that I will be sending on this thread.

The Gemini protocol restricts what is added to the core protocol. 
Compression is not used there for a reason as outlined by the FAQ, which I 
have read numerous times.

What the spec does NOT do is outlaw which filetypes can be sent over the 
protocol. Zip files can be downloaded over gemini. Same with any other filetype.

The spec says how to interpret both the protocol itself, and gemtext. It 
has nothing to say about handling filetypes, aside from gemtext. Why? 
Because gemini is a file transfer protocol at its core. You transfer files 
over it. The spec also purposefully allows for streaming of files, and 
this has been talked about many times in both a mailing list thread and on 
Solderpunk's gemlog.

What the spec does NOT do is try to censor what files can be downloaded 
over the protocol. Which means compressed files are allowed to be 
downloaded over the protocol, just like any other file. This is NOT 
against the protocol, as shown in the spec:

> Response handling by clients should be informed by the provided MIME 
type information. Gemini defines one MIME type of its own (text/gemini) 
whose handling is discussed below in section 5. In all other cases, 
clients should do "something sensible" based on the MIME type. 
Minimalistic clients might adopt a strategy of printing all other text/* 
responses to the screen without formatting and saving all non-text 
responses to the disk. Clients for unix systems may consult /etc/mailcap 
to find installed programs for handling non-text types.

I will now give various quotes by Solderpunk that show that Solderpunk was 
a fan of a variety of different types of clients that do different things 
- from his article on "A vision for Gemini applications" 
(gemini://gemini.circumlunar.space/users/solderpunk/gemlog/a-vision-for-gem
ini-applications.gmi):

> What this provides is a nice little, dare I say it, "containerised" 
Gemini application experience. You are easily and reliably identified to 
one service and one service only, and no external content can control what 
you do with that identity, and your identity can't accidentally leak out 
to anywhere external.
> 
> Meanwhile, your "everyday" Gemini client, which will let you go anywhere 
you like and follow whichever links you like, does not know the paths to 
any of your client certificates. Maybe it doesn't even support client 
certificates at all! If you roam the dangerous wild internet frontiers 
with this client, and you accidentally follow a malicious link to one of 
your apps running on localhost, as long as that app requires an approved 
client certificate to do anything of consequence, no damage can be done.

Later on:
> Some people may still be thinking that this looks like an ugly 
complication, even if you can wrap these containers up in nice convenient 
shell scripts (and I'm sure some kind of GUI management solution could be 
whipped up for people who want one): "What, now I need *two* kinds of 
client to use Gemini? Gimme a break!". But I think it's a small surface 
complication which yields large simplifications deeper down. The 
containerised identities approach using ultra-slim clients creates two 
clearly separate ecological niches for clients: reading static textual 
content like gemlogs, technical documentation, fiction, news reports, 
weather forecasts, etc. on the one hand and making use of individual, 
certificate-secured dynamic applications on the other. This allows for 
client authors to target one niche only and therefore write simpler clients.

> In short, different tools for different jobs, but with a common 
underlying protocol and markup language.

> Or they can partake of both Gemini experiences. I really hope that this 
approach can ease some of the tensions that are building between people 
with different visions for what Geminispace should be, and make the lives 
of developers easier at the same time.

In summary, and this is the most important one:
> Most Gemini clients will fall into one of two distinct categories
> * "Reading-centric" clients will probably have very little or no support 
for client certificates at all. Instead they will focus on beautiful and 
customisable typography, bookmarking, feed subscription, TOC generation, 
etc. There will be many clients of this kind, taking different approaches. 
Many will be graphical. People will love some and hate others. Plenty of 
room room for all.
> * "App-centric" clients will instead have good support for client 
certificates and will be designed to very securely limit the use of those 
clients to a single app in order to totally avoid the risk of CSRF. There 
will be much less variation amongst these clients because they are all 
just doing the same basic thing. Most of them will be terminal-based, 
because that's just the better interface for interacting with text-based 
applications, but they needn't all be.
> Hybrid clients attempting to address both the document reading and 
secure app niches will surely exist, but to avoid the risk of CSRF they 
will require either slightly clunky interfaces or good awareness and 
understanding from users. These may be the preferred clients of "power users".

Christian Seibold <krixano@mailbox.org> writes:

> Let me repeat this... adding compression to clients in the way that
> has been described in this thread does NOT break geminispace.

I'm not too too sure about this.  As I see it, using *any* file type
other than text/gemini as default choice for the content will break the
geminispace.

Gemini browser have to implement text/gemini[0], not text/gemini+gzip,
text/html or text/x-markdown, and if text/gemini+gz (or whatever) starts
to gain a bit of popularity it *will* break the geminispace IMHO.

> Gemini deliberately allows you to send whatever you want over. It's a
> file-transfer protocol. Using the mimetype is *exactly* what I
> proposed. Period. End of discussion.

and I totally agree with you here.  You can serve any type of content
over Gemini, so what's all this fuss about?  You have a strange use case
where you need compression?  Fine, you can do it.  Good for you.  But
please don't waste our time by telling how much cool would be to change
every browser just to read the usual 1-2K[1] pages.


Cheers,

Omar Polo

P.S.: I don't want to sound too harsh, in fact I have a local patch for
      gmid to allow the `entrypoint' config option inside `location'
      blocks, so that for instance one can do something like

	server "xyz.com" {
		...
		location "*.gmi.gz" {
			entrypoint "/my/gzip/script";
		}
	}

      to automatically generate gzipped files (using an hypotetical CGI
      script in /my/gzip/script) on the fly[2].


[0]: OK, not quite really, gemget/curl-like applications only implement
     the protocol, but I'm talking about more "user-facing" browsers.
[1]: Stephane Bortzmeyer once shared with us the mean size of
     text/gemini pages as computed by lupa IIRC.
[2]: actually, this will allow to generate *any* file on the fly, that's
     why I added it.

> I'm not too too sure about this.  As I see it, using *any* file type
> other than text/gemini as default choice for the content will break the
> geminispace.
> 
> Gemini browser have to implement text/gemini[0], not text/gemini+gzip,
> text/html or text/x-markdown, and if text/gemini+gz (or whatever) starts
> to gain a bit of popularity it *will* break the geminispace IMHO.
> 
> and I totally agree with you here.  You can serve any type of content
> over Gemini, so what's all this fuss about?  You have a strange use case
> where you need compression?  Fine, you can do it.  Good for you.  But
> please don't waste our time by telling how much cool would be to change
> every browser just to read the usual 1-2K[1] pages.
> 

Read the whole thread. I never once suggested compressed gemtext should be 
default. I suggested it can be supported in some clients when clicking on 
a link. Other people are having a problem with this, not me. I fully 
support sending compressed files over gemini, because it's allowed by the spec.

> As a personed diagnosed on the autistic spectrum that cannot *stand* 
"christmas lights", especially the "animated" or blinking kind, for a 
multitude of reasons, I request you refrain from dragging my "tribe" 
into this discussion to support your arguments. It is neither valid as 
an argumentative strategy, nor is it particularly welcome.


It is well-known within the autistic community that not all autistic 
people are the same. some are like you and have problems with light 
sensitivity. Some have other sensory sensitivities.

I request you refrain from speaking for all of the Autistic community, and 
assuming who is or isn't autistic within the mailing list.

> - Proposing an extension to the protocol is obviously against the core 
idea of gemini.

And btw, nobody here was proposing an extension to the protocol.

A bit of a recap and my thoughts:

* Compression is not going to be in the protocol; this is a bad idea for
multiple reasons, already pointed out.
* Authors can of course offer a compressed version of a gemtext document
alongside the uncompressed version. It remains to be seen how many
people would make use of the compressed version (depends, one might
suppose, on people's internet speeds and how long the document in
question is).
* Client software could in theory recognise a compressed gemtext file,
decompress it and display it as normal, giving the user the same
experience as with a normal gemtext file. Probably not many clients will
have this functionality, but who knows.
* Part of the goal with Gemini's simplicity is to avoid implementation
monoculture, so it is reasonablye to say for many users not using the
most featureful client. However, given the great variety in Gemini
users, I think a monoculture will be avoided anyway, and a feature like
automatic handling of compressed gemtext will never become mandatory. I
don't think proposing a feature that clients may implement is a threat
to Geminispace.

Some people should reflect on what they said in this discussion and how
they said it: what parts of it were helpful and what parts weren't.
Maybe they can avoid a similar mess in the future. Moving on...

On 19/06/2021 10:59, Stephane Bortzmeyer wrote:
> * how the end user of a browser without compression support will know
> in advance if the file is compressed or not? From the file extension?
> It is not always present.

I expect authors would indicate this in the link text. So:

=> article.gmi my article
=> article.gmi.gz my article (compressed version)

> * today, every browser can read every Gemini file. Your proposal
> seems to imply that some users will now see only a part of the
> Geminispace, the uncompressed one. This is the same problem we have
> when trying to use the Web with lynx or dillo: we can see only a part
> of it.

First of all, people are already serving PDFs &c via Gemini, which is
also something that Gemini is intended for. So already, not the entire
Geminispace is visible from within the Gemini client.

Secondly, I'd expect that authors offer a compressed version only in
addition to the uncompressed version, not as a replacement. I don't see
this feature becoming nearly popular enough for authors to start doing
the latter.

-- 
pjvm

> Some people should reflect on what they said in this discussion and how
they said it: what parts of it were helpful and what parts weren't.
Maybe they can avoid a similar mess in the future. Moving on...

It doesn't matter, because people are going to cherry pick what someone 
says anyways and try to put their own agenda forward as if it was the one 
intended by Solderpunk when it wasn't.

Anyways, everything you've said I completely agree with, and is precisely 
what I have been saying this entire time.

Normally, I prefer being staying out of stupid arguments, but this email 
just has so much bullshit that I cannot not respond.
>> Neither is javascript. Additionally, again, you have to look at the 
pros and cons of things. Javascript adds significant performance and 
security problems. Compression **doesn't**.
> 
> So are you OK, with PDF and postscript that you left out?

If I say that I love a person, does that imply I don't love anybody else? 
People don't always list everything.
>> Let's look at an lz4 library:
>> https://github.com/lz4/lz4
>> 
>> It's 49000 MB/s on a Core i7-9700K CPU @ 4.9GHz.
>> And it's 5699 LOC.
>> 
>> Btw, this is the approach that Solderpunk used to determine what should 
go into the protocol. It's right on the front page of the gemini capsule:
>> "Strives for maximum power to weight ratio"
> 
> Yes, please reread the FAQ. Compression is not in the standard for reasons.
And nobody is saying it should be, we are simply debating a separate file 
format for compressed gemtext. Nobody said anything about the protocol.
>> 
>>> 
>>>> which  seems more  like  the approach  Lagrange  takes (Skyjake  can
>>>>   correct me if I'm wrong).
>>> 
>>> Please this  is not the place  to discuss a single  implementation for
>>> gemini standard.
>> 
>> Passing things off to other applications is just as valid of a way to 
deal with this, and it's what AV-98 and various other clients do.
> 
> But  you prefer  a full  gemini browser,  right? And  clients *should*
> implement a form of decompression one way or another, right?
What exactly is wrong with advocating for something?

>> I in no way proposed an extension to the protocol. In fact, I proposed 
the exact opposite, that this should stay *out* of the protocol, and be in 
client applications for people who want to support this:
>> "I do want to note that the gemini protocol doesn't care about what's 
send   over it. This is why mimetypes were added to the protocol. You can 
send any binary data over gemini. This is why compression doesn't need to 
be in the main protocol - because you can send over the compressed stuff 
just as any other binary file can."
>> People are smart enough to decide for themselves what client they want 
to use. I'm allowed to promote a client. I think it's very presumptuous to 
think I'm somehow "propagandizing" people into supporting Lagrange. It's 
ridiculous. People are smart enough to decide for themselves what client 
they want to use. > i  like lagrange  a  lot, this  is neither  about  
lagrange nor  about
> compression, this is about pushing for complex client that could break
> the geminispace.
It's not gonna break the geminispace, as long as clients have the 
capability to simply dedicate that functionality to an external software, 
which is absolutely possible with this suggestion.


-- Unless you're replying to me on the Gemini mailing list,
reply to almaember@almaember.com instead.

Website: https://almaember.com/
Gemini capsule: gemini://almaember.com/
IRC: almaember on Libera.chat and tilde.chat

On 19/06/2021 10:59, Stephane Bortzmeyer wrote:
> If I understand correctly your proposal, you suggest to change,
> neither the protocol nor the gemtext format, but to push browser
> authors to recognize text/gemini+gzip and act accordingly.
> 
> In that case, I see two issues:
> 
> * how the end user of a browser without compression support will know
> in advance if the file is compressed or not? From the file extension?
> It is not always present.
> 
> => ilovecompression.gmiz I cannot read it
> => plain.gmi I can read it
The client could display a simple indicator on the UI. I'm pretty sure 
some browsers already display the MIME-type.

This isn't our decision, but the client authors'.
> 
> * today, every browser can read every Gemini file. Your proposal
> seems to imply that some users will now see only a part of the
> Geminispace, the uncompressed one. This is the same problem we have
> when trying to use the Web with lynx or dillo: we can see only a part
> of it.

Progress comes at a price. And, this is an unfinished protocol, at least officially.

-- Unless you're replying to me on the Gemini mailing list,
reply to almaember@almaember.com instead.

Website: https://almaember.com/
Gemini capsule: gemini://almaember.com/
IRC: almaember on Libera.chat and tilde.chat

On Sat, Jun 19, 2021 at 12:51:28PM +0200,
 Omar Polo <op@omarpolo.com> wrote 
 a message of 50 lines which said:

> [1]: Stephane Bortzmeyer once shared with us the mean size of
>      text/gemini pages as computed by lupa IIRC.

gemini://gemini.bortzmeyer.org/software/lupa/stats.gmi

* 10% of the resources are 247 bytes or less,
* 20% of the resources are 339 bytes or less,
* 30% of the resources are 507 bytes or less,
* 40% of the resources are 732 bytes or less,
* 50% of the resources are 1,032 bytes or less, MEDIAN
* 60% of the resources are 1,653 bytes or less,
* 70% of the resources are 2,626 bytes or less,
* 80% of the resources are 4,225 bytes or less,
* 90% of the resources are 7,448 bytes or less,
* 100% of the resources are 2,677,426 bytes or less.

(Yes, there is a text/gemini page of more than 2 MB, 
gemini://gemini.lost-frequencies.eu/posts/archive/articles-date/articles-da
te-2021/articles-date-2021-03/articles-date-2021-03-28/c941e49fc8990c8f20e4
b5d58a5bc8fd-The_Society_of_the_Spectacle.gmi)

Hi Christian, on Sa 19 jun 2021 11:48 you wrote:

> you can make suggestions to skyjake on Station or on the lagrange github
[..]
> compressed single files can absolutely be instantly decompressed by a
client that implements this.
>
> Also, if TLS 1.3 has compression (thanks for telling me that, I didn't
know that),
> then it makes supporting specific compressions in clients unnecessary,
because we
> are going to be switching to TLS 1.3 in the future, from what I've
gathered.

I am sorry I misinformed you, TLS 1.2 has compression (for example
deflate), but in TLS 1.3 it has been removed. But it introduces certificate
compression to save handshake time. Although it's understandable because of
the various exploits it's unfortunate as in my opinion there are use cases,
like SMTP or Gemini where it makes sense and where it's not a risk. I'll
make suggestions for browser handling of compressed individual Gemtexts.
Just like the TLS 1.3 design committee thinks about saving time with the
handshake, and compressing a certificate the size of Gemtext.  I dream
about saving time with optional zstandard or ppmd compression in the Gemini
protocol. Making it a user or browser choice is second best. Perhaps Gemini
can stay on TLS 1.2 forever?

Kind regards,
Francis Siefken (NL)
gemini://fsiefken.srht.site

On Sat, Jun 19, 2021 at 01:13:20PM +0200,
 PJ vM <pjvm742@disroot.org> wrote 
 a message of 52 lines which said:

> > * how the end user of a browser without compression support will
> > know in advance if the file is compressed or not? From the file
> > extension?  It is not always present.
> 
> I expect authors would indicate this in the link text. So:
> 
> => article.gmi my article
> => article.gmi.gz my article (compressed version)

As an user, my personal opinion is that it is not great UX.

On Sat, Jun 19, 2021 at 02:08:18PM +0200,
 Almaember <almaember@disroot.org> wrote 
 a message of 34 lines which said:

> I'm pretty sure some browsers already display the MIME-type.

They cannot, you don't know it until you've actually send the request
(and Gemini has no HEAD method so you have to retrieve everything).

* Saturday, 2021-06-19 at 11:34 +0200 - Francis Siefken <fsiefken@gmail.com>:

> TLS 1.3 has compression on board, it's a security
> risk in HTTP session hijacking and all HTTP browsers have turned it off.
> But this security risk doesn't apply to Gemini or with SMTP or IMAP. This
> would be another way to making this compressed gemtext vision work.

If I understand correctly, TLS 1.3 actually *removed* compression.

RFC 8446:
> legacy_compression_methods:  Versions of TLS before 1.3 supported
>  compression with the list of supported compression methods being
>  sent in this field.  For every TLS 1.3 ClientHello, this vector
>  MUST contain exactly one byte, set to zero, which corresponds to
>  the "null" compression method in prior versions of TLS.  If a
>  TLS 1.3 ClientHello is received with any other value in this
>  field, the server MUST abort the handshake with an
>  "illegal_parameter" alert.  Note that TLS 1.3 servers might
>  receive TLS 1.2 or prior ClientHellos which contain other
>  compression methods and (if negotiating such a prior version) MUST
>  follow the procedures for the appropriate prior version of TLS.

So TLS compression does not seem to be a viable option (we don't want to 
downgrade to 1.2, for various reasons).

P.S. here's how one could handle gzipped gemtext in diohsc, causing it to 
be presented as if it were uncompressed text/gemini:
set geminator text/gemini+gzip gzip -dc -

Hi Folks,

Just a couple of thoughts about the compression...

Even though it looks pretty harmless I think is going to add unnecessary complexity.

As Author I have to decide if to use or not use compression, HTML/HTTP 
doesn't use compression so far I know.

As Client I have to ask to the server if it serves compressed GMI file. 
The easy way I think is using a sibling protocol, hence I use 
geminiz://my.url: if the server server supports it I receive the 
compressed page otherwise it will use the default Gemini protocol.

On the other hand the Server might try to serve before compressed page and 
see if the client accepts those and then deciding which content serving.

Anyway everything looks quite uncertain; compressing pages is an unwanted 
behavior that as author I would not to deal with. I would like to transfer 
this task to my server or to a daemon that belongs to my server, hence it 
would a bargain on the shoulder of the implementers. 

I consider that many authors and implementers would be surely unhappy 
about this feature. Compressing content implies a choice which eventually 
is not trivial; for instance, knowing myself, I will surely ended up using 
a compressed method to squeeze up any bit of performance more.

The fact that such common feature was not included in the specification is 
exactly to avoid complexity and ambiguous behavior; However it would be 
smart if an offline client would be able to download an entire capsule and 
store it conveniently as compressed GMI.

In such scenario compressing GMI pages would be totally useful, under 
control without hurting or annoying anyone.

My best,

Gnuserland

On June 19, 2021 7:00:04 AM EDT, gemini-request@lists.orbitalfox.eu wrote:
>Send Gemini mailing list submissions to
>	gemini@lists.orbitalfox.eu
>
>To subscribe or unsubscribe via the World Wide Web, visit
>	https://lists.orbitalfox.eu/listinfo/gemini
>or, via email, send a message with subject or body 'help' to
>	gemini-request@lists.orbitalfox.eu
>
>You can reach the person managing the list at
>	gemini-owner@lists.orbitalfox.eu
>
>When replying, please edit your Subject line so it is more specific
>than "Re: Contents of Gemini digest..."
>
>
>Today's Topics:
>
>   1. Re: a space case for transparent gemtext compression
>      (Christian Seibold)
>   2. Re: a space case for transparent gemtext compression
>      (Christian Seibold)
>   3. Re: a space case for transparent gemtext compression (Omar Polo)
>

I kind of skimmed over this surprisingly heated debate on compression, so 
I will try to keep my thoughts short:

* I agree compression is unnecessary. Given what resources we have at our 
disposal (storage space, processing power, network bandwidth), Gemini is 
already far lighter than what our systems could handle. With compression 
enabled, we will gain little in speed or storage.

* I agree that compression is cool. Yes, I like transferring 90kb instead 
of 300kb. It jut seems neat, and for end users might still provide a 
noticeable speed improvement for larger documents.

* Adding compression to my site would break compatibility with a lot of 
Gemini browsers currently out there. This is probably not worth it. Gemini 
derives strength from being fixed standard, as its impressive levels of 
adoption and software development have affirmed.

In the past I tossed the idea around with a friend on the idea of making a 
Gemini capsule downloadable for offline use in a single, compressed 
archive. This eliminates a massive amount of overhead at the protocol 
level, since I could provide a compressed archive of 100 Gemini pages to 
the client in seconds compared to what would potentially take a very long 
time for the client to make 100 individual requests. This could greatly 
improve the efficacy of crawlers / search engines and people who for some 
reason would like to read my entire capsule in one go.

I think rather than bothering to compress individual files, we should be 
thinking about compressed archives/mirrors.

Ben

-- gemini://kwiecien.us/

Hi Gnuserland, you wrote today:


> As Author I have to decide if to use or not use compression, HTML/HTTP
> doesn't use compression so far I know.
>
> The HTTP protocol supports compression; for example "Content-Encoding:
zstd"
=> https://datatracker.ietf.org/doc/html/rfc8478#section-3
If both the client and server support gzip or zstd compression a compressed
stream is established. Yes this causes a developer overhead, that's why I
was happy to read compression is in TLS 1.2 and dismayed that it was
removed in TLS 1.3. While it's not removed in HTTP it warns against it's
use in certain situations, which do or do not apply in Gemini context:

=> https://datatracker.ietf.org/doc/html/rfc2616 HTTP/1.1 section on
compression
=> https://datatracker.ietf.org/doc/html/rfc7540#section-10.6 HTTP/2
compression section
=> https://datatracker.ietf.org/doc/html/draft-ietf-quic-http-34#page-47
HTTP/3 compression section

It doesn't have to be difficult to mandate or implement an easy solution.

Suppose I am in a life or death situation and I am on a 300 bps link I want
the 2 kilobyte of life-saving information to be there as soon as possible.
Or you have the daily top 12 HN threads mirrored as a Gemtext file (~100
Kilobyte each) and want to read those on a 300 bps link.
Compression speeds up the process. With zstd compression and dictionary
compression https://facebook.github.io/zstd/#small-data it can be there
quicker and on low end hardware.
I think it would be elegant not to rely on having two seperate files for
this on the server, one index.gmi and one index.gmi.zstd - the Gemini
specification could allow index.gmi to be a binary and let the browser
check this after retrieval. If it's a zstd binary automatically extract and
display it.
Ofcourse keeping this poor man's content sniffing method as simple and
secure as possible.
=> https://en.wikipedia.org/wiki/Content_sniffing Wikipedia on Content
sniffing

As mentioned by others in this thread a index.gmi.zstd could also be
extracted through the existing application/zstd mime-type automatically and
display the resulting gmi inline - but as the protocol doesn't enforce this
each client can implement it differently.
To me it seems like a minimal change to the Gemini protocol to allow gmi
text to be served in zstd format and to state that the client must
decompress and display it inline instead of merely offering to download it.

Saluton,
Francis Siefken (NL)

Ben <benulo@systemli.org> wrote:
> I think rather than bothering to compress individual files, we should be 
thinking about compressed archives/mirrors.

Oppen (author of Ariane) built this:

https://codeberg.org/oppenlab/gempub: A new eBook format based on Gemini 
Protocol's Gemtext. Gempub can also serve as a Gemini capsule archive format.

There is already support for GPUB documents in Lagrange. I don't think 
that support will (or should) become ubiquitous, but that was never the 
intention. The MIME type will be registered at some point, too.

Raph

On Sat, Jun 19, 2021 at 02:53:30PM +0200, Stephane Bortzmeyer wrote:
> On Sat, Jun 19, 2021 at 01:13:20PM +0200,
> PJ vM <pjvm742@disroot.org> wrote
>> I expect authors would indicate this in the link text. So:
>> 
>> => article.gmi my article
>> => article.gmi.gz my article (compressed version)
> 
> As an user, my personal opinion is that it is not great UX.

I'd argue that this isn't really a technical issue as much as it is a 
phrasing issue. Perhaps a better example would be:

> # Title
> 
> Author Note: this article is quite long; here's a compressed version:
> 
> => compressed.gmi.gz Compressed article
> 
> (article continues below)

Gemini clients supporting pagination can support a pref to download only 
the first $LINES lines of a gemtext file before the user scrolls down.  
This way, users can choose to open a link to e.g. a compressed copy before 
the page finishes loading.

I don't think that improving the UX should come at the cost of adding any 
new features to the spec. This proposed solution may not be "great UX", 
but it doesn't require adding any new features to clients besides the 
optional ability to pipe through gunzip, and it's easy enough to navigate 
for any user capable of navigating Gemini in the language of the capsule 
in question. Settling on a "good-enough" UX in pursuit of simplicity is 
one of the main appeals of Gemini; fine UX tuning is better suited for the 
Web or desktop applications.

This is doubly true since plaintext pages over 30kb (the threshold at 
which I might consider compression worthwhile) are quite uncommon.

-- /Seirdy

I am the least expert here...

But as author/user if the encryption is transparent for me, in the terms 
that I always upload a GMI file anyway, I don't see any real objection 
against but just more work for the implementers.

However the GMI is the lighter file we can think of, I really have hard 
time thinking any human made GMI file bigger than few kilobytes...

Thinking that a compressed capsule will save my life while modern HTTP, 
despite CDN, minification, etc... Will be always way more, and 
incomparable, bigger that any average GMI page, just makes believing that 
we are over-thinking about that topic!

Thanks... 😉

On June 19, 2021 12:22:49 PM EDT, Francis Siefken <fsiefken@gmail.com> wrote:
>Hi Gnuserland, you wrote today:
>
>
>> As Author I have to decide if to use or not use compression,
>HTML/HTTP
>> doesn't use compression so far I know.
>>
>> The HTTP protocol supports compression; for example
>"Content-Encoding:
>zstd"
>=> https://datatracker.ietf.org/doc/html/rfc8478#section-3
>If both the client and server support gzip or zstd compression a
>compressed
>stream is established. Yes this causes a developer overhead, that's why
>I
>was happy to read compression is in TLS 1.2 and dismayed that it was
>removed in TLS 1.3. While it's not removed in HTTP it warns against
>it's
>use in certain situations, which do or do not apply in Gemini context:
>
>=> https://datatracker.ietf.org/doc/html/rfc2616 HTTP/1.1 section on
>compression
>=> https://datatracker.ietf.org/doc/html/rfc7540#section-10.6 HTTP/2
>compression section
>=>
>https://datatracker.ietf.org/doc/html/draft-ietf-quic-http-34#page-47
>HTTP/3 compression section
>
>It doesn't have to be difficult to mandate or implement an easy
>solution.
>
>Suppose I am in a life or death situation and I am on a 300 bps link I
>want
>the 2 kilobyte of life-saving information to be there as soon as
>possible.
>Or you have the daily top 12 HN threads mirrored as a Gemtext file
>(~100
>Kilobyte each) and want to read those on a 300 bps link.
>Compression speeds up the process. With zstd compression and dictionary
>compression https://facebook.github.io/zstd/#small-data it can be there
>quicker and on low end hardware.
>I think it would be elegant not to rely on having two seperate files
>for
>this on the server, one index.gmi and one index.gmi.zstd - the Gemini
>specification could allow index.gmi to be a binary and let the browser
>check this after retrieval. If it's a zstd binary automatically extract
>and
>display it.
>Ofcourse keeping this poor man's content sniffing method as simple and
>secure as possible.
>=> https://en.wikipedia.org/wiki/Content_sniffing Wikipedia on Content
>sniffing
>
>As mentioned by others in this thread a index.gmi.zstd could also be
>extracted through the existing application/zstd mime-type automatically
>and
>display the resulting gmi inline - but as the protocol doesn't enforce
>this
>each client can implement it differently.
>To me it seems like a minimal change to the Gemini protocol to allow
>gmi
>text to be served in zstd format and to state that the client must
>decompress and display it inline instead of merely offering to download
>it.
>
>Saluton,
>Francis Siefken (NL)

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

I think a lot of aspects were discussed already but let me add my two
cents, coming from a slightly different perspective (webapp developer):

It was proposed to have two links, one to the gmi, one to the compressed
version. This is basically what the accept header in html is for, just
deferred to the user. And you look for some kind of signalling to the
server that you want a specific representation of a resource. You propose
file-extensions which by itself is another convention you introduce here
(or additional protocol). URLs don't know about file extensions. Webapps
don't know about file extensions.

Abstractly speaken: if we don't have a good way to signal to the server
what kind of representation of a resource I want, we should drop this. We
should not make up conventions artificially. Maybe they will emerge
naturally anyway.

One thing I miss in the discussion: I see comments about writing files and
serving files. I got by now that this is pretty common in Gemini space but
there are servers, serving generated content (blogs or games or whatever)
and so no assumption about how big files are should be made...

2021 06 19 15:23 mbays <mbays@sdf.org> wrote:

>
> If I understand correctly, TLS 1.3 actually *removed* compression.
>

Yes, I saw it, I erred - such a disappointment. So in the absence of
a protocol specification that leaves mime-type handling by the client using
two seperate gmi files, but as Peter Much remarked somewhere else in this
thread that doesn't cover compression of dynamic cgi data streams (which is
less of a use case for gemini perhaps - unless).


> P.S. here's how one could handle gzipped gemtext in diohsc, causing it
> to be presented as if it were uncompressed text/gemini:
> set geminator text/gemini+gzip gzip -dc -
>

I wasn't aware of diohsc, a really nice line mode browser which reminds me
of the tridactyl add-on. It has www and gopher proxy support, display
of images throug ascii. It's amazing! Transparent inline decompression with
gmi.gz works (so hopefully other clients can follow). I am still struggling
with getting 7z and zstd decompression to work - i'll figure it out. It
would be nice if I can get diohsc to run on termux in the oculus quest and
android phone so I can browse my gmi files with espeak (or a better android
tts alternative).
Thanks so much for writing it.

Kind regards from the netherlands,
Francis Siefken

On 6/19/21 9:12 PM, raph@raphm.com wrote:
> Gempub can also serve as a Gemini capsule archive format.

I'd be interested in any explorations into its utilization to that end. I 
did not realize gempub was even compressed.

-- gemini://kwiecien.us/

On Sun, Jun 20, 2021 at 02:52:04PM +0430,
 Ben <benulo@systemli.org> wrote 
 a message of 8 lines which said:

> I'd be interested in any explorations into its utilization to that
> end. I did not realize gempub was even compressed.

It inherits that from using Zip, like EPUB.

I use it for instance to store a copy of the Gemini capsules I'm
interested in.

Hi Christian,

Thanks for your swansong concerning this thread.

Apologies that my response deviates from the topic re compression but your 
points and citations (btw, do you have the links) do raise some interesting areas.

Given my experience of HTTP content, I have always meant to valorise my 
workflow across sites. I would like it, so that dependent on the site (or 
subsite), when I am clicking on the content is subsequently loaded in a 
different browser or with a different toolsets.

I have seem approaches for dealing with this at an eLisp end but I havent 
got around to implementing it myself. In some respects, my memory 
association with a specific site and a specific browser allows me to fudge 
it. While I do relish bindings and hotkeys, I have appreciated Emacs 
Hyperbole's ability to do something appropriate based around contents of 
the content in my text buffer (though similarly I havent yet tweaked it 
for all my needs and workflows) - based on simple clicking.

In general, I do feel that a 'workbench' approach has a utility, whereby 
different tools for different functions are available. One should have the 
ability to orientate towards a tool with minimal cost. A utility should be 
dependable and expectiations of deviation reasonable.

Going back to HTTP, Javascripts like uMatrix (and things like NoScript) 
have been informative re the contemptuous/irresponsible behaviour of 
content providers. The combinations for minimal client supplication in 
order to receive content are fascinating, as over time they break down 
(not only from shifts in infrastructure but additionally deterring JS 
blocker users). Im happy to have these impediments in the way, if only to 
be reminded of the harm. My less tech savvy partner tollerates this to an 
extent but edge cases really throw her, so all switches go on (!).

I recall a recent thread concerning clients and the use of favicons, with 
a lot of heat regarding whether an attempt at adding visual flair was 
detrimental. Similarly to the issue of compression, there seems to be an 
awkward dance which may ossilate between different producers and users of 
tools and content.

Is it possible to avoid an approach which is heavily dependent on 
(ambiguous?) norms within a site (ie, non protocol hardened expectations), 
which can be picked up/interpreted by a client? Instead, the client could 
call upon a repo(s) (pre downloaded, like a CA bundle) - which gives an 
indication of a URI's policies.

If such repo(s) existed, it might allow switching between different 
clients to satisfy different use cases.

For example:
* Desire compression and visiting site foo.fr? Use this client for this 
usecase, with settings Y, R and I built in client side
* Desire a favicon and visitng site bar.gr? Use this client for this 
usecase, with settings G, F, and T activated.


Kind regards,



Jonathan




June 19, 2021 12:49 PM, "Christian Seibold" <krixano@mailbox.org> wrote:

> This is the final message that I will be sending on this thread.
> 
> The Gemini protocol restricts what is added to the core protocol. 
Compression is not used there for
> a reason as outlined by the FAQ, which I have read numerous times.
> 
> What the spec does NOT do is outlaw which filetypes can be sent over the 
protocol. Zip files can be
> downloaded over gemini. Same with any other filetype.
> 
> The spec says how to interpret both the protocol itself, and gemtext. It 
has nothing to say about
> handling filetypes, aside from gemtext. Why? Because gemini is a file 
transfer protocol at its
> core. You transfer files over it. The spec also purposefully allows for 
streaming of files, and
> this has been talked about many times in both a mailing list thread and 
on Solderpunk's gemlog.
> 
> What the spec does NOT do is try to censor what files can be downloaded 
over the protocol. Which
> means compressed files are allowed to be downloaded over the protocol, 
just like any other file.
> This is NOT against the protocol, as shown in the spec:
> 
>> Response handling by clients should be informed by the provided MIME 
type information. Gemini
>> defines one MIME type of its own (text/gemini) whose handling is 
discussed below in section 5. In
>> all other cases, clients should do "something sensible" based on the 
MIME type. Minimalistic
>> clients might adopt a strategy of printing all other text/* responses 
to the screen without
>> formatting and saving all non-text responses to the disk. Clients for 
unix systems may consult
>> /etc/mailcap to find installed programs for handling non-text types.
> 
> I will now give various quotes by Solderpunk that show that Solderpunk 
was a fan of a variety of
> different types of clients that do different things - from his article 
on "A vision for Gemini
> applications"
> (gemini://gemini.circumlunar.space/users/solderpunk/gemlog/a-vision-for-g
emini-applications.gmi):
> 
>> What this provides is a nice little, dare I say it, "containerised" 
Gemini application experience.
>> You are easily and reliably identified to one service and one service 
only, and no external content
>> can control what you do with that identity, and your identity can't 
accidentally leak out to
>> anywhere external.
>> 
>> Meanwhile, your "everyday" Gemini client, which will let you go 
anywhere you like and follow
>> whichever links you like, does not know the paths to any of your client 
certificates. Maybe it
>> doesn't even support client certificates at all! If you roam the 
dangerous wild internet frontiers
>> with this client, and you accidentally follow a malicious link to one 
of your apps running on
>> localhost, as long as that app requires an approved client certificate 
to do anything of
>> consequence, no damage can be done.
> 
> Later on:
> 
>> Some people may still be thinking that this looks like an ugly 
complication, even if you can wrap
>> these containers up in nice convenient shell scripts (and I'm sure some 
kind of GUI management
>> solution could be whipped up for people who want one): "What, now I 
need *two* kinds of client to
>> use Gemini? Gimme a break!". But I think it's a small surface 
complication which yields large
>> simplifications deeper down. The containerised identities approach 
using ultra-slim clients creates
>> two clearly separate ecological niches for clients: reading static 
textual content like gemlogs,
>> technical documentation, fiction, news reports, weather forecasts, etc. 
on the one hand and making
>> use of individual, certificate-secured dynamic applications on the 
other. This allows for client
>> authors to target one niche only and therefore write simpler clients.
>> 
>> In short, different tools for different jobs, but with a common 
underlying protocol and markup
>> language.
>> 
>> Or they can partake of both Gemini experiences. I really hope that this 
approach can ease some of
>> the tensions that are building between people with different visions 
for what Geminispace should
>> be, and make the lives of developers easier at the same time.
> 
> In summary, and this is the most important one:
>> Most Gemini clients will fall into one of two distinct categories
>> * "Reading-centric" clients will probably have very little or no 
support for client certificates at
>> all. Instead they will focus on beautiful and customisable typography, 
bookmarking, feed
>> subscription, TOC generation, etc. There will be many clients of this 
kind, taking different
>> approaches. Many will be graphical. People will love some and hate 
others. Plenty of room room for
>> all.
>> * "App-centric" clients will instead have good support for client 
certificates and will be designed
>> to very securely limit the use of those clients to a single app in 
order to totally avoid the risk
>> of CSRF. There will be much less variation amongst these clients 
because they are all just doing
>> the same basic thing. Most of them will be terminal-based, because 
that's just the better interface
>> for interacting with text-based applications, but they needn't all be.
>> Hybrid clients attempting to address both the document reading and 
secure app niches will surely
>> exist, but to avoid the risk of CSRF they will require either slightly 
clunky interfaces or good
>> awareness and understanding from users. These may be the preferred 
clients of "power users".