Three possible uses for IRIs

(Starting a separate thread for this)

I think there are three possible places where IRIs could possibly appear in
Gemini:

1) In client inputs (the address bar or CLI analogue) and outputs
(revealing a link)

2) In the Gemini protocol

3) In text/gemini link lines

I think it's important to disentangle these three cases.  Case 1 just
affects individual clients and can be left up to them, except that there is
some best-practice advice about when *n?t* to display an IRI, specifically
when there are cross-script confusables involved.  For example,
"gemini://gemini.circumlunar.xn--spce-63d/" should not be displayed as
"gemini://gemini.circumlunar.sp?ce", because that would be deceptive, even
in Gemini: you might be pointed to the Evil Version of the Gemini spec and
not realize it.

I think everyone agrees that Case 2 is a mistake: the protocol elements
should continue to be URIs.

Case 3 is the difficult one.  Should authors be allowed to write
text/gemini links with IRI references? It's not that hard for a client to
convert them to URI references.  No normalization is needed except as part
of punycoding.  However, everyone has to agree on whether this should work
or not; we don't want a user trying to follow a link and sending the Wrong
Thing to the server.

Gemini isn't just supposed to be easy to program for, it's supposed to be
easy to author, too.  Unfortunately these objectives are in conflict here.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
Lope de Vega: "It wonders me I can speak at all.  Some caitiff rogue
did rudely yerk me on the knob, wherefrom my wits yet wander."
An Englishman: "Ay, belike a filchman to the nab'll leave you
crank for a spell." --Harry Turtledove, Ruled Britannia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201207/ed25
588c/attachment.htm>

> (Starting a separate thread for this)

Thanks, that's helpful. Hopefully all IRI discussion can move here.

> Case 3 is the difficult one.? Should authors be allowed to write text/gemini
> links with IRI references??It's not that hard for a client to convert them
> to URI references.? No normalization is needed except as part of punycoding.
> However, everyone has to agree on whether this should work or not; we don't
> want a user trying to follow a link and sending the Wrong Thing to the server.

"It's not that hard for a client to convert them to URI references. No
normalization is needed except as part of punycoding."

I don't think that's true. To convert them to a URI reference, the domain needs
to be extracted and punycoded, then the path and query string needs to be
extracted and percent-encoded in the blessed Gemini way that doesn't allow plus
signs. Doing all this requires parsing, and as I explained a couple times in the
other thread, IRI parsing is not feasible across multiple programming languages
at this time, the libraries just don't exist.

And what if the IRI is a relative reference? As I explained in the other thread,
this will definitely require IRI parsing.

Furthermore, it's breaking change to Gemini. I don't think that's a good idea in
any case with the possible exception of TLS security. Gemini must be reliable,
and it's too late for a breaking change.

> Gemini isn't just supposed to be easy to program for, it's supposed to be easy to
> author, too.? Unfortunately these objectives are in conflict here.

Yes, and that's unfortunate. But I think it makes sense for the stability of Gemini
and the ease of programming to come first.


Cheers,
makeworld

On Mon, Dec 7, 2020 at 9:47 PM <colecmac at protonmail.com> wrote:


> I don't think that's true. To convert them to a URI reference, the domain
> needs
> to be extracted and punycoded,


Agreed.  But if you have a Punycode encoder, then the following steps will
convert an IRI reference to a URI reference, without regard to whether it
is an IRI or a relative reference:

1) Look in the IRI reference for a "//" and a following "/"; if they exist,
pass the characters in between through your encoder and substitute the
result into the IRI reference.

2) Start over from the beginning.  If a character is ASCII, leave it
unchanged.  Otherwise, take the character, convert it to UTF-8 bytes (easy)
and each byte to hex digits (trivial), decorate it with leading %
(trivial), and move on.  When you come to the end, stop.


> Furthermore, it's breaking change to Gemini. I don't think that's a good
> idea in
> any case with the possible exception of TLS security. Gemini must be
> reliable,
> and it's too late for a breaking change.
>

Probably true.  ~~sigh~~



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
Normally I can handle panic attacks on my own; but panic is, at the moment,
a way of life.                 --Joseph Zitt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201207/67b2
3603/attachment.htm>

On Mon, Dec 7, 2020, at 18:47, colecmac at protonmail.com wrote:

> 
> Yes, and that's unfortunate. But I think it makes sense for the 
> stability of Gemini
> and the ease of programming to come first.

I'm perplexed that "ease of programming" is considered more important than 
"ease of adoption."

You mention that not every language supports the libraries needed for 
internationalized URLs. 

What does that lose the project vs. accessibility and broader adoption by 
non-English-speaking users for who Gemini would be a boon with limited 
bandwidth and hardware?

I feel like I'm missing something with the emphasis on ease of client 
implementation over adoption.

Emma Humphries
gemini://gemini.djinn.party/

> > Yes, and that's unfortunate. But I think it makes sense for the
> > stability of Gemini
> > and the ease of programming to come first.
>
> I'm perplexed that "ease of programming" is considered more important 
than "ease of adoption."
>
> You mention that not every language supports the libraries needed for 
internationalized URLs.
>
> What does that lose the project vs. accessibility and broader adoption 
by non-English-speaking users for who Gemini would be a boon with limited 
bandwidth and hardware?
>
> I feel like I'm missing something with the emphasis on ease of client 
implementation over adoption.


I was unsure when I wrote that, and I was worried it would be controversial.
But I still think it makes sense. For Gemini to be accessible, have "broader
adoption", and be "a boon" as you mention, clients need to be easy to write
and maintain. Otherwise, what will these non-English speaking users browse
and serve their content with? A few clients and servers, likely not written
in their native language?

Gemini is a non-commercial hobby project for all the developers I am aware of,
and there are advantages to that. But it also means that if the protocol is hard
to implement, the whole community suffers, because there will be fewer clients
and servers.

The fact that writing URLs for non-English languages is difficult sucks. But
due the complexity, and most of all the fact that this would be a breaking
change, I don't see IRIs as an option.


makeworld


P.S. I'll admit I'm biased. I write more code for Gemini than I do content, and
primarily use my native language English.

It was thus said that the Great colecmac at protonmail.com once stated:
> 
> The fact that writing URLs for non-English languages is difficult sucks. But
> due the complexity, and most of all the fact that this would be a breaking
> change, I don't see IRIs as an option.

  I thought I might see what's involved with handling IRIs.

  The actual differences between RFC-3986 (URI) and RFC-3987 (IRI) besides
one being a standard (URI) and one being a proposed standard (IRI) comes
down to the characters that are accepted---the unreserved set 

   unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"

becomes

   iunreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~" / ucschar
   ucschar        = %xA0-D7FF / %xF900-FDCF / %xFDF0-FFEF
                  / %x10000-1FFFD / %x20000-2FFFD / %x30000-3FFFD
                  / %x40000-4FFFD / %x50000-5FFFD / %x60000-6FFFD
                  / %x70000-7FFFD / %x80000-8FFFD / %x90000-9FFFD
                  / %xA0000-AFFFD / %xB0000-BFFFD / %xC0000-CFFFD
                  / %xD0000-DFFFD / %xE1000-EFFFD

and the query portion changes from

   query         = *( pchar / "/" / "?" )
   pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"


to

   iquery         = *( ipchar / iprivate / "/" / "?" )
   ipchar         = iunreserved / pct-encoded / sub-delims / ":" / "@" 
   iprivate       = %xE000-F8FF / %xF0000-FFFFD / %x100000-10FFFD

and that's it as far as the RFCs go (aside from the rule name changes).  As
a quick proof-of-concept, I just accepted all non-control UTF-8 characters
as unreserved (including the private space) as that was the easiest thing to
do, and yes, it works (but does allow potentially bad IRIs through).  

  But the code to *build* a URL from the parsed representation [2] ssumes
US-ASCII.  Again, it would take just a few small changes to allow UTF-8
characters on input and escape them properly for a URL.  That's something
I'll try working on tomorrow.

  That still leaves the question of punycode [3] and Unicode normalization
(ugh).

> P.S. I'll admit I'm biased. I write more code for Gemini than I do content, and
> primarily use my native language English.

  I am biased too, as a monolingual US mutt, but I do want to try this stuff
out.

  -spc

[1]	https://github.com/spc476/LPeg-Parsers/blob/master/url.lua

[2]	https://github.com/spc476/GLV-1.12556/blob/master/Lua/GLV-1/url-util.lua

[3]	RFC-3492, which includes C code to encode and decode punycode text,
	which is valgrind clean (I checked).

On Mon, 7 Dec 2020 23:00:01 -0500
John Cowan <cowan at ccil.org> wrote:
 
> Agreed.  But if you have a Punycode encoder, then the following steps will
> convert an IRI reference to a URI reference, without regard to whether it
> is an IRI or a relative reference:
> 
> 1) Look in the IRI reference for a "//" and a following "/"; if they exist,
> pass the characters in between through your encoder and substitute the
> result into the IRI reference.
> 
> 2) Start over from the beginning.  If a character is ASCII, leave it
> unchanged.  Otherwise, take the character, convert it to UTF-8 bytes (easy)
> and each byte to hex digits (trivial), decorate it with leading %
> (trivial), and move on.  When you come to the end, stop.

There's a "drawl the owl" step somewhere here regarding Unicode
normalization. Does the server like your ?:s fully composed or
decomposed, or should the server itself be responsible for
normalization?

-- 
Philip
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201208/cf4d
c147/attachment.sig>

> On Dec 8, 2020, at 05:06, Emma Humphries <ech at emmah.net> wrote:
> 
> I'm perplexed that "ease of programming" is considered more important 
than "ease of adoption."

Or basic use for that matter.

Gemini's narrative is build upon the fallacy that everything is 
"just-oh-so-trivial": any Dick and Jane can fire up the most bare bone of 
telnet over their trusty dial-up modem and be done. Some sort of 
citizen-programmer-publisher nirvana, without any barriers to entry whatsoever. 

Admirable.

But not quite practical. The internet stack is deep, old, and brittle. Tooling matter. 

Still, all very admirable :)

Long live Gemini.

On Mon, 07 Dec 2020 20:06:27 -0800
"Emma Humphries" <ech at emmah.net> wrote:

> I'm perplexed that "ease of programming" is considered more important 
than "ease of adoption."

Consider "ease of programming" and in particular stability a subset of
"ease of adoption". There are numerous client and server
implementations because it is easy to implement, and easy to maintain
because the protocol is relatively stable even in these early stages.
The different software allows people with different goals to adopt the
protocol, and helps in weeding out shortcomings of clarity in the
specification by analysis of the subtle differences between
implementations.

> You mention that not every language supports the libraries needed for 
internationalized URLs. 
> 
> What does that lose the project vs. accessibility and broader adoption 
by non-English-speaking users for who Gemini would be a boon with limited 
bandwidth and hardware?

It seems more likely that a change to this end would hurt adoption.
Numerous pieces of existing Gemini software would immediately be
invalidated. Not all of them will be updated to accommodate the change.
I could perhaps see a more pressing need for the change if internet
users worldwide weren't already used to transliteration. It's such a
small part as well. UTF-8 is acceptable (and default) in text/gemini
documents, and the text content of a capsule can indeed be written in
any of the scripts supported by Unicode.

-- 
Philip
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201208/e511
9f33/attachment.sig>

On Mon, Dec 07, 2020 at 09:37:28PM -0500,
 John Cowan <cowan at ccil.org> wrote 
 a message of 117 lines which said:

> I think there are three possible places where IRIs could possibly appear in
> Gemini:

A fourth one is server configuration. (When you declare the virtual
host, for instance.)

> some best-practice advice about when *n?t* to display an IRI, specifically
> when there are cross-script confusables involved.  For example,
> "gemini://gemini.circumlunar.xn--spce-63d/" should not be displayed as
> "gemini://gemini.circumlunar.sp?ce", because that would be deceptive, even
> in Gemini: you might be pointed to the Evil Version of the Gemini spec and
> not realize it.

As I said, I regard "homograph attacks" as mostly a tale to discourage
people to use Unicode. They are not a real-world problem.

On Tue, Dec 08, 2020 at 02:47:22AM +0000,
 colecmac at protonmail.com <colecmac at protonmail.com> wrote 
 a message of 50 lines which said:

> Furthermore, it's breaking change to Gemini. I don't think that's a
> good idea in any case with the possible exception of TLS
> security. Gemini must be reliable, and it's too late for a breaking
> change.

Hold on. I'm a newbie in Gemini and I was under the impression that
Gemini is still experimental and the specification still in flux. If
it is not true, if Gemini is frozen and "take it or leave it", that's
a different matter, and we could save some time by rejecting many
discussions.

It was thus said that the Great Stephane Bortzmeyer once stated:
> On Tue, Dec 08, 2020 at 02:47:22AM +0000,
>  colecmac at protonmail.com <colecmac at protonmail.com> wrote 
>  a message of 50 lines which said:
> 
> > Furthermore, it's breaking change to Gemini. I don't think that's a
> > good idea in any case with the possible exception of TLS
> > security. Gemini must be reliable, and it's too late for a breaking
> > change.
> 
> Hold on. I'm a newbie in Gemini and I was under the impression that
> Gemini is still experimental and the specification still in flux. If
> it is not true, if Gemini is frozen and "take it or leave it", that's
> a different matter, and we could save some time by rejecting many
> discussions.

  I know Solderpunk wants to do a series of freezes then thaws as things are
worked on, but I think things progress a bit faster than he can deal with,
or wants to deal with, given his long absences on the list.

  For me personally, I think this should be worked out, and I'm working
towards that with my own server [1].  I've had to make changes to
GLV-1.12556 in the past when the protocol changed, I can change it again.

  -spc

[1]	https://github.com/spc476/GLV-1.12556

Philip Linde writes:

> On Mon, 07 Dec 2020 20:06:27 -0800
> "Emma Humphries" <ech at emmah.net> wrote:
>
>> I'm perplexed that "ease of programming" is considered more important 
than "ease of adoption."
>
> Consider "ease of programming" and in particular stability a subset of
> "ease of adoption". There are numerous client and server
> implementations because it is easy to implement, and easy to maintain
> because the protocol is relatively stable even in these early stages.
> The different software allows people with different goals to adopt the
> protocol, and helps in weeding out shortcomings of clarity in the
> specification by analysis of the subtle differences between
> implementations.
>
>> You mention that not every language supports the libraries needed for 
internationalized URLs. 
>> 
>> What does that lose the project vs. accessibility and broader adoption 
by non-English-speaking users for who Gemini would be a boon with limited 
bandwidth and hardware?
>
> It seems more likely that a change to this end would hurt adoption.
> Numerous pieces of existing Gemini software would immediately be
> invalidated. Not all of them will be updated to accommodate the change.
> I could perhaps see a more pressing need for the change if internet
> users worldwide weren't already used to transliteration. It's such a
> small part as well. UTF-8 is acceptable (and default) in text/gemini
> documents, and the text content of a capsule can indeed be written in
> any of the scripts supported by Unicode.

Hey,

I'm new to this list, and a new Gemini user, but this topic is fairly 
important to me. It's discouraging to see a lot of fear-mongering around 
this topic already.

Some points that have come up a few times already in this thread as well 
as the IDN thread that I think are worth addressing:

1. Homograph attacks

Stephane has already mentioned in a different response that homograph 
attacks are fairly rare. I don't have the knowledge to say whether or not 
that's accurate, but I can speak to how they're mitigated.

In general, browsers will render the domain in the URI bar if all of the 
characters in the each section belong to the same script. As an example, 
https://?pple.com will not render correctly in Firefox in the URI bar, but 
https://?????.com/ will render correctly (both domains do not exist if you 
want to check).

The other half of this comes down to domain registrars not allowing 
registrations of domains with homographs (depends on the TLD, of course).

What this comes down to, is that Gemini clients, if they wish to mitigate 
this type of attack, should apply the same algorithm as web browsers. 
Again, given the preference for client certs for authenticating sessions, 
it doesn't seem like this attack would have dire consequences anyway.

I also think I saw someone mention that they're worried about it from the 
IRI side as well? That attack doesn't seem like much of a realistic case, 
since if they direct you to a different page on the same server, you're 
well, still on the same server. This only becomes problematic in the case 
of shared hosting of untrusted tenants.

2. Normalization

There's been a bit of fear-mongering about normalization which I can 
totally understand, since a first look at Unicode technical reports and 
the 4 normalization forms looks intimidating at first glance.

However, as pointed out in a few RFCs, NFC is more or less the only 
normalization form that you need to worry about in *most* circumstances. 
Typed URIs should be normalized in NFC, both on server-side and 
client-side. When resolving files to the filesystem, the filename should 
be normalized to NFC. (this all assumes that your fs supports Unicode paths).

NFKC becomes more relevant in the case that you want to implement 
something like search, or find in page, or something. You may want a user 
to be able to type in something like 'e' have their find include 
everything whose NFKC form is basically an 'e' (see the full set here: 
https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=%5B%3ANFKC_Casef
old%3De%3A%5D&g=&i=).

3. Language support

Normalization is generally supported across different languages p easily.

Python has it in its stdlib: 
https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize

Golang has support: https://pkg.go.dev/golang.org/x/text/unicode/norm

Rust: https://unicode-rs.github.io/unicode-normalization/unicode_normalization/index.html

C get its support through the venerable libicu library (you're already 
using libs for TLS): 
https://unicode-org.github.io/icu/userguide/transforms/normalization/

I will say that I don't know of any explicit IRI-handling libraries, nor 
do I know what the state of support is in different URI-handling 
libraries, but it will be something I play with as I work on gemini 
projects. I'm happy to share my experiences when I have more of them. :)

-

To address some non-technical points, I don't think that starting a new 
protocol and then deciding to ignore internationalization is necessarily 
the right way to go. In a lot of cases, internationalization sucks because 
of legacy support, and gemini doesn't *have* legacy to preserve 
compatibility. As I understand it, that's why TLS is mandatory, even 
though it arguably locks out some retro systems from being able to use it.

Personally, I'd like to see the spec say something about how this is 
handled before any type of freeze takes place.

--
worr

>   I know Solderpunk wants to do a series of freezes then thaws as things are
> worked on, but I think things progress a bit faster than he can deal with,
> or wants to deal with, given his long absences on the list.

I'd love to see a spec freeze, too. There are already a lot of gemini
servers, clients and other tools out there and breaking changes should
be avoided unless absolutely necessary.

>   For me personally, I think this should be worked out, and I'm working
> towards that with my own server [1].  I've had to make changes to
> GLV-1.12556 in the past when the protocol changed, I can change it again.

How about waiting for a consensus to develop, *at the very least*?

If the protocol were to change to allow IRIs, that's a *major breaking* 
change that to me, as someone actually serving non-English content, is
not only completely unnecessary but harmful.

1. I would still have to percent-encode my links to stay compatible with
existing clients.
2. With clients now potentially sending IRIs and not encoded URIs as
requests I would have to change the request handling in my server code
to allow for this, possibly having to add third-party dependencies.
3. I'm still not convinced this would help anyone - IRIs still have
reserved characters that have to be properly encoded - so completely 
non-technical text/gemini authors will still have to rely on proper
tooling.

bie

On Tue, Dec 8, 2020 at 5:10 AM Petite Abeille <petite.abeille at gmail.com> wrote:
> Gemini's narrative is build upon the fallacy that everything is 
"just-oh-so-trivial": any Dick and Jane can fire up the most bare bone of 
telnet over their trusty dial-up modem and be done. Some sort of 
citizen-programmer-publisher nirvana, without any barriers to entry whatsoever.
Well, not telnet, because of TLS, but openssl-s_client at least.

On Tue, Dec 08, 2020 at 12:46:47PM +0100, William Orr wrote:

> In general, browsers will render the domain in the URI bar if all of
> the characters in the each section belong to the same script. As an
> example, https://?pple.com will not render correctly in Firefox in the
> URI bar, but https://?????.com/ will render correctly (both domains do
> not exist if you want to check).

A lot of extra complexity for very little value.

FWIW, first url you showed looks absolutely the same as legit
"https://apple.com" I typed manually in my vim in TERM=linux.

I came to gemini because for web I, inhabitant of /dev/tty1, is
third-class citizen. Please, don't bring this to Gemini. If "curl
gemini://foo.example/" is not good enough, than your feature is too
complicated.

My native language is Russian (which is not even latin-based), and
goverment website has URL of "https://gosuslugi.ru", and everything
works fine. If you ask me, IRI is a huge mistake.

> On Dec 8, 2020, at 12:49, bie <bie at 202x.moe> wrote:

[2020-12-08T11:31:16.041Z] <bie> fff
[2020-12-08T11:52:29.189Z] <bie> fuck this
[2020-12-08T11:52:32.918Z] <bie> lol
[2020-12-08T11:52:45.090Z] <bie> time to unsubscribe
...

[2020-12-08T12:51:08.517Z] <khuxkm> my favorite was whoever's response to 
you saying that was "oh we're a frozen spec now?"
[2020-12-08T12:51:16.729Z] <khuxkm> like YES we've been a frozen spec since, like, June
...

[2020-12-08T12:51:45.154Z] <makeworld> khuxkm: I pinged Solderpunk on 
Masto and he got back to me very quickly saying he had read the IDN thread 
and was going to come to a decision soon
[2020-12-08T12:51:52.567Z] <makeworld> Yeah lol
[2020-12-08T12:52:08.982Z] <makeworld> I think spc is getting nerd sniped
...

etc, etc, etc...

Certainly you must be aware that the logs from #gemini on tilde.chat are 
fully accessible to everyone who can be bothered, snarky comments & all. For posterity.

https://portal.mozz.us/gemini/makeworld.gq/cgi-bin/gemini-irc

> On Dec 8, 2020, at 14:17, A. E. Spencer-Reed <easrng at gmail.com> wrote:
> 
> Well, not telnet, because of TLS, but openssl-s_client at least.

TLS-based Telnet Security
https://tools.ietf.org/html/draft-ietf-tn3270e-telnet-tls-00

But yes, some sort of TLS layer of one kind or another  :)

On Tue, Dec 8, 2020 at 6:49 AM bie <bie at 202x.moe> wrote:


> If the protocol were to change to allow IRIs, that's a *major breaking*
> change that to me, as someone actually serving non-English content, is
> not only completely unnecessary but harmful.
>

As I said at the beginning of this thread, I don't think anyone is actually
arguing for a change to the protocol.  What does warrant discussion is
allowing IRI references as links in the text/gemini format.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201208/2c55
7dee/attachment.htm>

[2020-12-08T15:01:30.527Z] <makeworld> Hello, Petite Abeille. Apparently 
you're watching the logs, courtesy of my server, so that you can send them 
back on to the mailing list.
[2020-12-08T15:01:50.519Z] <makeworld> I don't see the point, other than 
trying to stir the pot. Please stop.
[2020-12-08T15:02:34.637Z] <bie> ?
[2020-12-08T15:02:39.645Z] <makeworld> The fact that this channel is 
logged is in the topic, it is known. None of the comments you sent were rude, as well.

On Tuesday, December 8, 2020 10:57 AM, John Cowan <cowan at ccil.org> wrote:

> On Tue, Dec 8, 2020 at 6:49 AM bie <bie at 202x.moe> wrote:
> ?
>
> > If the protocol were to change to allow IRIs, that's a *major breaking*
> > change that to me, as someone actually serving non-English content, is
> > not only completely unnecessary but harmful.
>
> As I said at the beginning of this thread, I don't think anyone is actually
> arguing for a change to the protocol.? What does warrant discussion is
> allowing IRI references as links in the text/gemini format.


I think some people really were calling for a breaking change to the protocol.
But I'm glad you're not, and I hope we can move on and stop talking about it.
What you propose here is allowing IRIs in link lines only? Or do you mean allowing
only IRIs for relative references?

I'm unsure whether that would require an IRI parser or not, but I'd feel more
confident with one. However, there is already a client torture test that *sort of*
covers this. It's not designed as an IRI test, but it includes invalid
characters in a link line.

gemini://gemini.conman.org/test/torture/0031

That page contains a link line that looks like this:

=> <0032> "Beware the bad link"

And the Go stdlib will actually correct this link and output a correct
absolute one. So in Amfora, it will go to the correct URL, which is
gemini://gemini.conman.org/test/torture/%3C0032%3E

I've set up my own test that contains a more complex Unicode character: ?.
It tests the path, as well as Unicode in the query strings.
You can access it at: gemini://makeworld.gq/test/iri-link.gmi

Go also corrects the link in that one, and it works. Allowing IRIs in link
lines (maybe only for relative links to ease parsing) would solve all
multi-lingual author problems.

But this is still a somewhat-breaking change, as once authors start using
these, other non-Go clients will likely begin to fail. And the correction
that Go does is not even complete, because it will not work on query strings.
And even if it did, it would not work in the Gemini way that doesn't allow
pluses, etc etc.

We're almost there with this one, but I still think it's a mistake, and it'll
make Gemini more complex. :/


makeworld

> On Dec 8, 2020, at 18:55, colecmac at protonmail.com wrote:
> 
> [2020-12-08T15:01:50.519Z] <makeworld> I don't see the point, other than 
trying to stir the pot. Please stop.
> [2020-12-08T15:02:39.645Z] <makeworld> The fact that this channel is 
logged is in the topic, it is known. None of the comments you sent were rude, as well.

Did I touch a nerve? Apologies. Back to our regular program then.

On Tue, Dec 8, 2020 at 2:09 PM <colecmac at protonmail.com> wrote:


> I think some people really were calling for a breaking change to the
> protocol.
> But I'm glad you're not, and I hope we can move on and stop talking about
> it.
> What you propose here is allowing IRIs in link lines only?


Yes.

> Or do you mean allowing
> only IRIs for relative references?
>

No.

> I'm unsure whether that would require an IRI parser or not,


It will not, because conversion can be done before parsing, other than the
trivial parsing required to find the hostname and punycode it.  Once that
is done, converting an IRI reference to a URI reference is as
straightforward as transcoding from one character set to another, and
totally indifferent to the IRI format.  So my two steps for IRI->URI
conversion become three:

1)  NFC normalization.

2) Punycode conversion of the hostname.

3) Percent-encoding: find non-ASCII characters and convert them to %nn%nn,
or %nn%nn%nn, or %nn%nn%nn%nn sequences, where nn is two hex digits.

It turns out that all of this is spelled out in more detail at <
https://tools.ietf.org/html/rfc3987#section-3.1>.  That section says not to
normalize unless you have the IRI in non-digital or non-UTF* format, but
since the world is not full of editors that normalize, I think Gemini
clients need to do it themselves.   That said, most keyboard drivers (even
for hard cases like Vietnamese, which has way too many vowels to dedicate a
key to each) now deliver normalized text to applications.

It's good to know that some existing URI libraries support IRIs, but that
section should be convincing evidence that you can change an IRI to a URI
without parsing it (always excepting the domain name, which is trivial to
find).

But this is still a somewhat-breaking change, as once authors start using
> these, other non-Go clients will likely begin to fail. And the correction
> that Go does is not even complete, because it will not work on query
> strings.
> And even if it did, it would not work in the Gemini way that doesn't allow
> pluses, etc etc.
>

The above transformation will work, however.  Sometimes DIY is the Right
Thing.

> We're almost there with this one, but I still think it's a mistake, and
> it'll
> make Gemini more complex. :/
>

It will.  But in the end, if Gemini succeeds even modestly there will be
more authors than programmers.

[*] 72 lower-case vowel letters: 6 vowels without diacritics plus 6 vowels
with vowel-quality diacritics, as in French, times 6 tone marks (one of
which is "no mark") as in Chinese.  And the same number in upper case.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
It's like if you meet an really old, really rich guy covered in liver
spots and breathing with an oxygen tank, and you say, "I want to be
rich, too, so I'm going to start walking with a cane and I'm going to
act crotchety and I'm going to get liver disease. --Wil Shipley
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201208/6720
d39c/attachment.htm>

> > We're almost there with this one, but I still think it's a mistake, and it'll
> > make Gemini more complex. :/
>
> It will.  But in the end, if Gemini succeeds even modestly there will be more
> authors than programmers.

This is the point that sticks out to me. Perhaps I was wrong.

The method you outlined does not seem that complex, and it really would benefit
authors. It's still a breaking change, but all existing links would still work.

The most difficult part of what you outlined is the Unicode normalization,
which maybe not all languages have libraries for, and would also require
updating every so often. But it wouldn't be a requirement for clients at all,
just something nice to have.

However, it does raise a few questions:

I assume you mean NFC normalization? Any other option seems nonsensical to me,
but I'm also new to this in general. Would be happy to be corrected.

What if the user named a domain/file/folder in a non-NFC way? Now does the server
need to support NFC as well, and apply it to vhost recognition or local file paths
to correctly match requests? That seems wrong. But so does the user entering
something visually identical to what the the the sysadmin typed, and things not
working.

I'm not keen to muddle up the threads again, but it seems like this proposal
completely covers IDNs as well, which is handy.

Overall, I like it. The biggest thing holding me back is the fact that it will
break clients, over time. But perhaps that's worth it for the ease-of-writing
gain for non-English speakers.

I wouldn't mind updating Amfora to support this. As I explained in my previous email,
it sort of already does this by accident.


Cheers,
makeworld

It was thus said that the Great colecmac at protonmail.com once stated:
> 
> I'm unsure whether that would require an IRI parser or not, but I'd feel more
> confident with one. However, there is already a client torture test that *sort of*
> covers this. It's not designed as an IRI test, but it includes invalid
> characters in a link line.
> 
> gemini://gemini.conman.org/test/torture/0031
> 
> That page contains a link line that looks like this:
> 
> => <0032> "Beware the bad link"
> 
> And the Go stdlib will actually correct this link and output a correct
> absolute one. So in Amfora, it will go to the correct URL, which is
> gemini://gemini.conman.org/test/torture/%3C0032%3E

  What you failed to quote from that test is:

	I'm not entirely sure what the proper response should be ...

  And it was a last minute thing to add the link to %3C0032%3E---I was
thinking it was more of an Easter Egg type of thing than what the actual
result should be.  

> I've set up my own test that contains a more complex Unicode character: ?.
> It tests the path, as well as Unicode in the query strings.
> You can access it at: gemini://makeworld.gq/test/iri-link.gmi

  I tried both the Gemini Client Torture Test 31, and your link with the
Gemini portal at portal.mozz.us.  The results were interesting.  If failed
the Gemini Client Torture Test, but loaded the page with the Unicode
character on your site.  So at least it supports percent encoding of
characters outside the ASCII range.

  -spc (So that's one more data point ... )

It was thus said that the Great Petite Abeille once stated:
> 
> [2020-12-08T12:52:08.982Z] <makeworld> I think spc is getting nerd sniped

  I don't know if I'm being nerd sniped or not, but I do think this has
brought to my attention some encoding bugs I have---namely, I don't encode
data with non-US-ASCII characters.  Fixing bugs is always A Good Thing (TM). 
I'm also looking into just how hard it would be to support IRIs.  Except for
the normalization thing, it looks to be fairly straightforward, but I
haven't worked on it that much yet.  I've already had my preconceived
notions of DNS blown out of the water over this thread (and I've implemented
a DNS library [1] so that's saying something).

  -spc (What it says, I don't know)

[1]	https://github.com/spc476/SPCDNS

> 3) Percent-encoding: find non-ASCII characters and convert them to %nn%nn,
> or %nn%nn%nn, or %nn%nn%nn%nn sequences, where nn is two hex digits.

One extra thing: Gemini will need it's own list of reserved characters.
The URI spec defines[1] this list:

gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"


(It also defines a list called sub-delims, but that only applies to query
strings I believe, and is irrelevant to the way Gemini uses them.)

These characters are reserved because of their use in other parts of
a URI. But Gemini does not use all those parts, such as userinfo. I
believe a reserved character list for Gemini could look like this:

":" / "/" / "#" / "?" / "[" / "]"

I left fragments ("#") in, so that clients can add support for them later,
if/when a header-to-fragment algorithm is defined, like exists for Markdown.
But that character could be removed too, which would prevent it ever being
used in that manner.

1: https://tools.ietf.org/html/rfc3986#section-2.2


makeworld

On Tue, Dec 8, 2020 at 4:10 PM <colecmac at protonmail.com> wrote:


> The most difficult part of what you outlined is the Unicode normalization,
> which maybe not all languages have libraries for, and would also require
> updating every so often. But it wouldn't be a requirement for clients at
> all,
> just something nice to have.
>

If a client has an unnormalized IRI, it needs to normalize it before
sending it to the server.  That said, a 2009 study looked at a sample of
700 million HTML documents, of which only 0.02% were not in NFC already,
which suggests that NFC text is already pretty dominant.

I assume you mean NFC normalization?
>

Yes.  When I speak of normalization, I mean NFC normalization exclusively.

> What if the user named a domain/file/folder in a non-NFC way? Now does the
> server
> need to support NFC as well, and apply it to vhost recognition or local
> file paths
> to correctly match requests? That seems wrong. But so does the user
> entering
> something visually identical to what the sysadmin typed, and things not
> working.
>

I'm okay with that just failing, as file names are not really part of
text/gemini content.  The difference will be obvious to the admin by
checking the requested URIs from the server log against the %-encoded names
of the folders.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
                I am a member of a civilization. --David Brin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201208/6b45
b640/attachment.htm>

On Tue, Dec 8, 2020 at 4:42 PM <colecmac at protonmail.com> wrote:


> (It also defines a list called sub-delims, but that only applies to query
> strings I believe, and is irrelevant to the way Gemini uses them.)
>

Gemini query strings can certainly be formatted like Web query strings if
the client knows that's what the server expects.  Simple free text isn't
the only possibility.  I'm going to talk about that in a posting at some
point.

These characters are reserved because of their use in other parts of
> a URI. But Gemini does not use all those parts, such as userinfo. I
> believe a reserved character list for Gemini could look like this:
>
> ":" / "/" / "#" / "?" / "[" / "]"
>

We still need the square brackets for the rare case when the host-part is
an IPv6 address.  The only character we could leave out with complete
safety is @, and I don't think that's worth special-casing for Gemini.
It's simpler and better to have the same rules for all URIs.

> I left fragments ("#") in, so that clients can add support for them later,
> if/when a header-to-fragment algorithm is defined, like exists for
> Markdown.
>

+1 to leaving # reserved, not only for that reason but for the same reason
as @; it's not worth making a special rule for Gemini to avoid a trivial
amount of %-encoding, especially given that most file names don't have
either one in their names.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
weirdo:    When is R7RS coming out?
Riastradh: As soon as the top is a beautiful golden brown and if you
stick a toothpick in it, the toothpick comes out dry.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201208/4739
3bea/attachment.htm>

> > The most difficult part of what you outlined is the Unicode normalization,
> > which maybe not all languages have libraries for, and would also require
> > updating every so often. But it wouldn't be a requirement for clients at all,
> > just something nice to have.
>
> If a client has an unnormalized IRI, it needs to normalize it before sending
> it to the server.

Could you justify this? It's a good thing to have, but it feels like a big ask,
as Unicode support and especially things like normalization are not straightforward
in all languages. I don't see why it can't just be recommended and not required.

> > I assume you mean NFC normalization?
>
> Yes.? When I speak of normalization, I mean NFC normalization exclusively.

Sounds good!

> > What if the user named a domain/file/folder in a non-NFC way? Now does the server
> > need to support NFC as well, and apply it to vhost recognition or local file paths
> > to correctly match requests? That seems wrong. But so does the user entering
> > something visually identical to what the sysadmin typed, and things not
> > working.
>
> I'm okay with that just failing, as file names are not really part of text/gemini
> content.? The difference will be obvious to the admin by checking the requested
> URIs from the server log against the %-encoded names of the folders.

The issue is that admins are not the only ones who create folders and files.
Non-technical people will as well, and a bug like this will be very confusing.
Everything will look right, but it just won't work. However, I doubt this will
occur very often, and it's an acceptable tradeoff to supporting Unicode.


makeworld

It was thus said that the Great bie once stated:
> >   I know Solderpunk wants to do a series of freezes then thaws as things are
> > worked on, but I think things progress a bit faster than he can deal with,
> > or wants to deal with, given his long absences on the list.
> 
> I'd love to see a spec freeze, too. There are already a lot of gemini
> servers, clients and other tools out there and breaking changes should
> be avoided unless absolutely necessary.
> 
> >   For me personally, I think this should be worked out, and I'm working
> > towards that with my own server [1].  I've had to make changes to
> > GLV-1.12556 in the past when the protocol changed, I can change it again.
> 
> How about waiting for a consensus to develop, *at the very least*?

  If I waited for consensus, Gemini would not be where it is today [1].
Also, it brought out a what I consider a bug in my code (generating links
from filenames) that it doesn't properly URL encode data [2].

> If the protocol were to change to allow IRIs, that's a *major breaking* 
> change that to me, as someone actually serving non-English content, is
> not only completely unnecessary but harmful.

  I don't expect that an IRI will be allowed for a request, but that an IRI
could be in a Gemini text file and it's up to the client to do the
conversion.  And it's that bit that I'm currently exploring.  

> 3. I'm still not convinced this would help anyone - IRIs still have
> reserved characters that have to be properly encoded - so completely 
> non-technical text/gemini authors will still have to rely on proper
> tooling.

  And we won't know until somebody tries.

  -spc

[1]	There's a reason why GLV-1.12556 and gemini.conman.org were the
	first Gemini server software and server in existance, becauxe I just
	went ahead and implemented it while solderpunk was still talking
	about it.  And I think the presense of GLV-1.12556 and
	gemini.conman.org sparked others to get busy.  And GLV-1.12556 was
	*NOT* following the specification at the time, as I disagreed with
	parts of the specification.

[2]	I don't have any non-ASCII file names, so it never crossed my mind
	to handle such things.  That is a blind spot as far as I'm
	concerned.

colecmac at protonmail.com writes:

> The issue is that admins are not the only ones who create folders and files.
> Non-technical people will as well, and a bug like this will be very confusing.
> Everything will look right, but it just won't work. However, I doubt this will
> occur very often, and it's an acceptable tradeoff to supporting Unicode.

It's arguably worse than that; consider the case where your filesystem
doesn't store filenames in UTF-8 ? notably Windows stores them in UCS2.
If you're treating filenames as Unicode strings and not byte arrays, and
your language provides good abstractions for that, you're okay, but the
upshot is that both the client and the server really do need to be
Unicode aware.

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |