Some reading on IRIs and IDNs

Hi, all. The discussion on IRIs and IDNs is a little intense, and I
thought I would take a step back and do some reading on it. I'm not
monolingual, but I am ISO-8859-1-lingual, if that makes sense, so some
of the issues are new to me.

So, there's an overview of all the issues involved here:
https://www.w3.org/International/articles/idn-and-iri/. This article
(from 2008) goes over the things you need to do to implement support for
IRIs, without going too much into the technical details. It makes things
look pretty straightforward and cut-and-dried, but...

In terms of actual standardization, things are kind of a mess. See this
page: https://www.w3.org/International/wiki/IRIStatus. This page brings
up the real issues with the standard.

It seems like the effort to standardize IRIs in the same framework as
URLs, URNs, and URIs fell apart in 2014. The effort was picked up by the
HTML5 WHATWG, which has their own "living standard" called URL:
http://url.spec.whatwg.org/. The URL standard focuses somewhat on
parsing/processing/serializing international URLs, which is useful to
us, but it is also *extremely* WWW-centric. It doesn't really take into
account non-HTTP(S) URLs, especially ones that are not very web-like,
like mailto or schemes where the authority field is not a domain name. 
Much of the spec focuses on things like how a web browser should
represent URLs in the address bar and in text.

This *probably* contributes to the lack of IRI-parsing libraries for
various languages: there's no standard for them to implement!

Given all that... maybe we should just consider our use cases and see
what the minimum we have to do is?

As I see it, the main requirement is that authors want to be able to use
non-ASCII characters in both the domain part and the path part of the
links in their documents, and have that work with no problems. IMO this
is a *reasonable expectation* for a retrofuturistic protocol like
Gemini.

Now, what does that require of client authors and server authors?

What is the *absolute minimum* we can require of client and server
authors and have things work?

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

It was thus said that the Great Jason McBrayer once stated:
>   
> Now, what does that require of client authors and server authors?
> 
> What is the *absolute minimum* we can require of client and server
> authors and have things work?

  As I've stated, I've created an IRI parser per RFC-3987 [1] and it was a
very minimal change to my original URL parser per RFC-3986 [2].  Basically,
it allows any UTF-8 character past codepoint 128 to used, as is in the IRI. 
Languages that have URL parsers may or may not support UTF-8 data.  So IRI
parsing may or may not be an issue (aside from Unicode normalization) on a
per-language basis.

  I've also started down the punycode rabit hole.  As Stephane has stated,
DNS *can* support UTF-8, but such support isn't wide, nor is it a standard. 
Punycode was developed to encode UTF-8 with ASCII in a most Byzantine way. 
It does have an RFC (RFC-3492) and said RFC does contain code for encoding
and decoding punycode (but it's in C, and the API is ... not what I would
define but it can be worked with).  IDN support, from my experience over the
past two days, is *harder* than IRI, although the concern was mostly the
other way.  I haven't actually *gotten* to the part of converting a domain
name to punycode but in general, to convert a domain name:

	for each label [3]:
		if label has non-ASCII characters
			convert to punycode, prepend "xn--" to result

so a domain name like "??.english.s?d?r.???" is converted thusly:

	?? 	-> 99zt52a -> xn--99zt52a
	english	-> (no conversion required)
	s?d?r 	-> sdr-rlad -> xn--sdr-rlad

	???

		-> wgbh1c -> xn--wgbh1c

to become "xn--99zt52a.english.xn--sdr-rlad.xn--wgbh1c" (and that last
segment is giving my editor fits because it's right-to-left).  The example
is extreme but it's just there to serve as an example of how to go about it.

  So given my experiences so far, I would say the easiest way to deal with
all this is to make it a client issue.  Hold off on IDN support for now (see
below for some more questions about it), but UTF-8 in the path and query
should be allowed in text/gemini, but encoded before making a request.  A
client, given a link like:

=> gemini://gemini.bortzmeyer.org:8965/caf??foo=bar Order from the Caf?

should be able to parse it with the UTF-8 characters, but convert it to:

	gemini://gemini.bortzmeyer.org:8965/caf%C3%A9?foo=bar

before making the request.  At the very least, tools could be developed to
encode links in text/gemini before publishing them if no one wants the spec
to change at all.  

I feel that would be the easiest, less breaking, thing to do now.  Making
IDN (punycode) mandatory might require a bit more discussion as I'm not sure
of the language support.  I'm not even sure what name should be in a
certificate for an IDN---the full UTF-8 version, or the punycode version, or
both?  What's currently done in HTTP land about this? (answering this will
at least point in a direction, even if we don't want to go that direction).

  -spc

[1]	https://github.com/spc476/LPeg-Parsers/blob/master/iri.lua

[2]	https://github.com/spc476/LPeg-Parsers/blob/master/url.lua

[3]	The domain name "gemini.conman.org" has three labels, "gemini",
	"conman" and "org".  The term "label" is DNS lingo.

On Tue, Dec 08, 2020 at 09:49:50PM -0500,
 Jason McBrayer <jmcbray at carcosa.net> wrote 
 a message of 48 lines which said:

[Thanks for the detailed analysis of the issue]

> this is a *reasonable expectation* for a retrofuturistic protocol
> like Gemini.

I love "retrofuturistic" and I hope it will be used in the official
specification.

On Wed, Dec 09, 2020 at 12:26:51AM -0500,
 Sean Conner <sean at conman.org> wrote 
 a message of 73 lines which said:

> DNS *can* support UTF-8, but such support isn't wide, nor is it a
> standard.

Wrong. DNS 2181, which clarifies that "any binary string whatever can
be used as the label of any resource record" is part of the Standards
Track. The reasons why few people use UTF-8 in domain names are:

* Ignorance and lack of care for internationalization,

* Non-DNS reasons such as the fact that existing libraries and applications
  may react badly to non-ASCII domain names,

* As often, the problem of normalization. DNS has normalization of
  names (the case-insensitivity rule) but it does not extend to
  Unicode.

> It does have an RFC (RFC-3492) and said RFC does contain code for
> encoding and decoding punycode (but it's in C, and the API is
> ... not what I would define but it can be worked with).

There is an implemention of Punycode in every standard library,
whatever your language.

> so a domain name like "??.english.s?d?r.???" is converted thusly:

In Python (but it is as simple in any other language):

>>> print(codecs.encode("??.English.s?d?r.???", encoding="idna"))
b'xn--99zt52a.English.xn--sdr-rlad.xn--wgbh1c'

(Note that the encodings.idna library of Python standard library is
limited to IDN v1.)

So, almost nothing to do for the programmer. I don't agree with your
assessment that IDN is simpler than IRI.

> I'm not even sure what name should be in a certificate for an
> IDN---the full UTF-8 version, or the punycode version, or both?
> What's currently done in HTTP land about this? (answering this will
> at least point in a direction, even if we don't want to go that
> direction).

gemini://gemini.bortzmeyer.org/rfc-mirror/rfc8399.txt

But it depends on the CA. It seems Let's Encrypt does not want to
handle UTF-8 and requires Punycode.

> [3]	The domain name "gemini.conman.org" has three labels, "gemini",
> 	"conman" and "org".  The term "label" is DNS lingo.

Let's be picky, there are four, there is also the root :-)

Stephane Bortzmeyer <stephane at sources.org> writes:

> On Wed, Dec 09, 2020 at 12:26:51AM -0500,
>  Sean Conner <sean at conman.org> wrote 
>  a message of 73 lines which said:
>
>> DNS *can* support UTF-8, but such support isn't wide, nor is it a
>> standard.
>
> Wrong. DNS 2181, which clarifies that "any binary string whatever can
> be used as the label of any resource record" is part of the Standards
> Track. The reasons why few people use UTF-8 in domain names are:

How widespread is support in client resolver libraries and in servers,
though? It's one thing to say, "yes, the standard is to support
non-ASCII names", and another entirely to say "just sending non-ASCII
names to your DNS server will work".

> There is an implemention of Punycode in every standard library,
> whatever your language.

Not strictly true. There isn't one in Common Lisp's standard library,
for example. There is one in Quicklisp, the widely used package
repository though, so that's okay for me.

I kind of feel like we should just bite the bullet and admit that a
fully-compliant client needs to punycode domains when looking them up,
and encode URLs when sending them to the server, and that
fully-compliant servers need to decode URLs when resolving them.

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

On Wed, Dec 09, 2020 at 10:08:25AM -0500, Jason McBrayer wrote:

[punycode library]

> There is one in Quicklisp,

Glad to hear that! :)

Bye!
C.

> On Dec 9, 2020, at 09:38, Stephane Bortzmeyer <stephane at sources.org> wrote:
> 
> There is an implemention of Punycode in every standard library,
> whatever your language.

There is also GNU Libidn (idn):

# idn --quiet r?ksm?rg?s.se bl?b?rgr?d.no
xn--rksmrgs-5wao1o.se
xn--blbrgrd-fxak7p.no

https://www.gnu.org/software/libidn/manual/html_node/Invoking-idn.html

It was thus said that the Great Stephane Bortzmeyer once stated:
> On Wed, Dec 09, 2020 at 12:26:51AM -0500,
>  Sean Conner <sean at conman.org> wrote 
>  a message of 73 lines which said:
> 
> > It does have an RFC (RFC-3492) and said RFC does contain code for
> > encoding and decoding punycode (but it's in C, and the API is
> > ... not what I would define but it can be worked with).
> 
> There is an implemention of Punycode in every standard library,
> whatever your language.
> 
> > so a domain name like "??.english.s?d?r.???" is converted thusly:
> 
> In Python (but it is as simple in any other language):
> 
> >>> print(codecs.encode("??.English.s?d?r.???", encoding="idna"))
> b'xn--99zt52a.English.xn--sdr-rlad.xn--wgbh1c'
> 
> (Note that the encodings.idna library of Python standard library is
> limited to IDN v1.)
> 
> So, almost nothing to do for the programmer. I don't agree with your
> assessment that IDN is simpler than IRI.

  I'm sorry, but the two languages I work in do *not* have an implementation
of punycode in their standard library.  I *was* able to find code for C
(from the RFC, which at least I know will work per the RFC) and could not
find one for Lua.  There's a reason why I'm having to muck with this.  The
API I have for C is *not* set up to handle domain names (breaking out the
labels, prepending or removing the "xn--", etc.).

  It's wonderful that the language you use comes with punycode support in
its standard library.  Not all languages have that.  I'm looking at the list
of clients [1] and there's one client written in a language I haven't heard
of before (Vala).  Other languages used are Nim, scheme and Tcl.  I would be
surprised if Vala or Nim have a punycode implementation.

  -spc (But hey, write your own client that does eveything you want and show
	us all how easy it is)

[1]	gemini://gemini.circumlunar.space/software/

> I'm sorry, but the two languages I work in donot have an implementation
> of punycode in their standard library.

Isn't this somewhat irrelevant in this case? It's unfortunate they don't
have an implementation, but as it stands right now, most Gemini clients
will not handle Unicode domain names at all. Punycoding domains will
solve that issue, and languages that don't have it in their stdlib
can either use a third-party library, or if that's not possible then
those languages will just ignore punycoding entirely. Obviously that's
not great for those languages, but it doesn't make sense to me to not
have punycoding at all because of that.

At the end of the day, Gemini clients must be allowed to support Unicode
domains. Perhaps the term "SHOULD" as defined by RFC2119 should be used
in the spec in this case.


makeworld

> On Dec 9, 2020, at 23:17, Sean Conner <sean at conman.org> wrote:
> 
> could not find one for Lua

Would this suit?

https://github.com/haste/lua-idn/blob/master/idn.lua

It was thus said that the Great Petite Abeille once stated:
> 
> > On Dec 9, 2020, at 23:17, Sean Conner <sean at conman.org> wrote:
> > 
> > could not find one for Lua
> 
> Would this suit?
> 
> https://github.com/haste/lua-idn/blob/master/idn.lua

  Huh ... turns out I should have searched for 'IDN' instead of 'punycode'. 
The code is for Lua 5.2---it will take a bit of work to get it working for
Lua 5.3 but that still leaves normalization issues.  I'm working with the
GNU libidn right now because it can do normailzation, otherwise, I can get
two different IDNs for the same (visually) domain:

	r?sum?	resume-jxde
	r?sum?	rsum-bpad

  -spc

> On Dec 10, 2020, at 00:23, Sean Conner <sean at conman.org> wrote:
> 
> 	r?sum?	resume-jxde
> 	r?sum?	rsum-bpad

Talking of which:

Lua Parser for Punycode/IDN Homograph Attack
https://community.rsa.com/community/products/netwitness/blog/2017/04/24/lua
-parser-for-punycodeidn-homograph-attack

The lua wiki has various libs for normalization: http://lua-users.org/wiki/LuaUnicode

I also mentioned libicu earlier, which is a very commonly used lib for 
unicode handling, and the wiki even mentions a lua library that provides 
bindings to libicu (although the bindings may be out of date).

https://github.com/unicode-org/icu

Hope that helps

> On Dec 10, 2020, at 01:01, William Orr <will at worrbase.com> wrote:
> 
> The lua wiki has various libs for normalization: http://lua-users.org/wiki/LuaUnicode

Indeed, ustring sports some normalization routines:

https://github.com/wikimedia/mediawiki-extensions-Scribunto/tree/master/inc
ludes/engines/LuaCommon/lualib/ustring

While on the topic, do people validate all UTF-8 coming their way? Ala 
iconv -f UTF-8 -t UTF-8?

It was thus said that the Great colecmac at protonmail.com once stated:
> > I'm sorry, but the two languages I work in donot have an implementation
> > of punycode in their standard library.
> 
> Isn't this somewhat irrelevant in this case? 

  It was more a comment about this quote:

> There is an implemention of Punycode in every standard library,
> whatever your language.

  There is *not* an implementation of punycode in every standard library,
whatever your language.  In a lot of currently in-use languages?  Probably,
but not *all*.  

> It's unfortunate they don't
> have an implementation, but as it stands right now, most Gemini clients
> will not handle Unicode domain names at all. 

  And I'm not seeing anyone else trying to update clients to do this, even
if in an exploratory nature.  What?  Are they just waiting for a decree?

> Punycoding domains will
> solve that issue, and languages that don't have it in their stdlib
> can either use a third-party library, or if that's not possible then
> those languages will just ignore punycoding entirely. 

  Poking around the GNU libidn documentation, I found Appendix B [1]
worrisome because the IDN rabbit hole just got deeper with U+2024 (ONE DOT
LEADER) and U+2485 (DIGIT FIVE FULL STOP).  Should I worry about it?  I
don't know.  This internationalization stuff is complex and makes me want to
throw up hands in the air, scream a bit, and go back to the simplicity of
ASCII.

  In the end, I'll probably just do Unicode normalization, then punycode and
call it a day.

> Obviously that's
> not great for those languages, but it doesn't make sense to me to not
> have punycoding at all because of that.
> 
> At the end of the day, Gemini clients must be allowed to support Unicode
> domains. Perhaps the term "SHOULD" as defined by RFC2119 should be used
> in the spec in this case.

  So when are you going to update gemget and Amfora to support punycode? 
I've heard it's easy to do.  Or are you waiting for a spec change first?

  -spc (Seriously, I feel like I'm the only one *doing* anything here)

[1]	https://www.gnu.org/software/libidn/manual/html_node/On-Label-Separators.html

> > It's unfortunate they don't
> > have an implementation, but as it stands right now, most Gemini clients
> > will not handle Unicode domain names at all.
>
> And I'm not seeing anyone else trying to update clients to do this, even
> if in an exploratory nature. What? Are they just waiting for a decree?
>
> [snip]
>
> So when are you going to update gemget and Amfora to support punycode?
> I've heard it's easy to do. Or are you waiting for a spec change first?
>
> -spc (Seriously, I feel like I'm the only one doing anything here)

Yes, I am waiting for Solderpunk. On the subject of IDNs, it seems obvious
to me that punycode will accepted as the thing to do for DNS, and that
Unicode should be sent to the server, but some questions around certs
and normalization still remain. I've outlined them here[1].

I suppose you're correct about being the only one doing anything, but I
don't feel like it makes sense to do anything yet. The solution is simple
code-wise (for IDNs), and so I don't feel the need to experiment, and I'd
rather implement this once, in-line with the spec, rather than multiple
times if Solderpunk says something different.

I guess this is just two different approaches to handling issues with a spec.
Gemini is intended to be very strict and not extensible, and is driven by
its spec rather than what people end up doing in the wild, like on the Web.

I don't think you're doing something wrong or bad by publicly experimenting,
but I'd rather not make things more uncertain by implementing something
in non-toy/demo clients before it's official.

1: https://github.com/makeworld-the-better-one/go-gemini/issues/10

Cheers,
makeworld

It was thus said that the Great colecmac at protonmail.com once stated:
> > > It's unfortunate they don't
> > > have an implementation, but as it stands right now, most Gemini clients
> > > will not handle Unicode domain names at all.
> >
> > And I'm not seeing anyone else trying to update clients to do this, even
> > if in an exploratory nature. What? Are they just waiting for a decree?
> >
> > [snip]
> >
> > So when are you going to update gemget and Amfora to support punycode?
> > I've heard it's easy to do. Or are you waiting for a spec change first?
> >
> > -spc (Seriously, I feel like I'm the only one doing anything here)
> 
> Yes, I am waiting for Solderpunk. On the subject of IDNs, it seems obvious
> to me that punycode will accepted as the thing to do for DNS,

  And why would Solderpunk choose this if no one has bothered to even look
into the possible issues wit respect to coding?  

> and that
> Unicode should be sent to the server, 

and that would be a breaking change on the protocol.  Just having the client
accept IRIs and send URIs wouldn't change the protocol.  But aside for me,
NO ONE bothered to even test this out!

> but some questions around certs
> and normalization still remain. I've outlined them here[1].
> 
> I suppose you're correct about being the only one doing anything, but I
> don't feel like it makes sense to do anything yet. The solution is simple
> code-wise (for IDNs), and so I don't feel the need to experiment, 

  And if it's so simple, why not do it? But I get it, you'd rather wait
until a yeah/nay decision is made.  I mean, who wouldn't like

	single digit resonse codes
	no client certificates
	a link line of [text|url]
	no virtual hosting
	a request format ala gopher (including TABs!)
	no rediection
	no indication of pages are actually gone vs not found
	no MIME parameters

  That's pretty much what the Gemini spec *would have been* had some people
*cough cough* not bothered with doing any implementations.  But i get it. 
Bikeshedding with talk is easy, bikeshedding with an implementation is
tedious.

  -spc

I've been following along with my own software in the background.

First of all, my domain registrar won't even let me put unicode characters
in an A record without automatically converting them to punycode for me.

caf?.mozz.us -> xn--caf-dma.mozz.us

Next, my naive python test client just kind of works as-is [0][1]. It will
convert unicode DNS names to punycode under the hood before doing the lookup.
Any unicode in the URL (IRI?) is left alone because.. why would a
client ever muck
around with the URL that the user gives them? That sounds like a bad idea to
me.

My server (running jetforce) also works as-is. All I had to do was add an entry
for "caf?.mozz.us" as a recognized hostname, and there you go.

 ```
jetforce-client gemini://caf?.mozz.us
Welcome to AV-98!
Enjoy your patrol through Geminispace...
?? WELCOME TO MOZZ.US ??
 ```

Requesting unicode path names also works with no changes on my part

 ```
jetforce-client gemini://caf?.mozz.us/files/?????.txt
20 text/plain
This is a test file with unicode characters in the name.?
 ```

As do quoted path names (the server will unquote the URL before it
attempts to load the file)

 ```
jetforce-client
gemini://caf?.mozz.us/files/%F0%9D%92%BB%F0%9D%92%B6%F0%9D%93%83%F0%9D%92%B
8%F0%9D%93%8E.txt
20 text/plain
This is a test file with unicode characters in the name.
 ```

Does this mean my server is already compliant? What else should I try?

- Michael

[0] https://github.com/michael-lazar/jetforce/blob/master/jetforce_client.py
[1] It's nice to finally get a win for python after fighting with TLS
for so long

It was thus said that the Great Michael Lazar once stated:
> I've been following along with my own software in the background.

  Thank you.  Without an implementation it is difficult to see where the
landmines are.  So, with that said ...

> First of all, my domain registrar won't even let me put unicode characters
> in an A record without automatically converting them to punycode for me.
> 
> caf?.mozz.us -> xn--caf-dma.mozz.us

  Okay.

> Next, my naive python test client just kind of works as-is [0][1]. It will
> convert unicode DNS names to punycode under the hood before doing the lookup.
> Any unicode in the URL (IRI?) is left alone because.. why would a
> client ever muck
> around with the URL that the user gives them? That sounds like a bad idea to
> me.

  That's debatable.  The percent encoding doesn't change the meaning, just
the "envelope" so-to-speak.  

> My server (running jetforce) also works as-is. All I had to do was add an entry
> for "caf?.mozz.us" as a recognized hostname, and there you go.

  Okay, about that.  I modified my own stupid-simple client to support IRIs
and to convert the hostname via punycode (finally!).  The code changes in
the client weren't that large (once I got the punycode module written, it
was one line to switch from URI parsing to IRI parsing, one line to add the
punycode module, and one line modified to punycode the host when making a
connection) but I'm encountering an issue.  If I use:

	gemini://caf?.mozz.us/files/?????.txt

(and send that as the request) It works, and I get the file.  But when I go
to:

	gemini://xn--caf-dma.mozz.us/files/%F0%9D%92%BB%F0%9D%92%B6%F0%9D%93%83%F0
%9D%92%B8%F0%9D%93%8E.txt

(and send that as the request) I get an error 53 (no proxy allowed).  When I
go to:

	gemini://caf?.mozz.us/files/%F0%9D%92%BB%F0%9D%92%B6%F0%9D%93%83%F0%9D%92%
B8%F0%9D%93%8E.txt

(and send that as the request) it works as well.  I would expect the second
example to work along with the first and third examples. They all reference
the same resource in the same server.

  Another issue that I've thought of, the length of each request---the first
is 53 bytes, the second is 99 bytes and the third is 93 bytes.  This *could*
be an issue with respect to the the overall limit of 1024 bytes for a
request.

  As far as servers go, GLV-1.12556 still uses the URL parser, and would
choke on an IRI being given as a request (since it expects non-ASCII
characters to be encoded per RFC-3986).  That would be an easy fix for me
(just switch to the IRI parser) but allowing IRIs would be an actual change
to the protocol.  I'm just saying.

> Does this mean my server is already compliant? What else should I try?

  Perhaps allow "xn--caf-dma.mozz.us" as a hostname?

  -spc

> [0] https://github.com/michael-lazar/jetforce/blob/master/jetforce_client.py
> [1] It's nice to finally get a win for python after fighting with TLS
> for so long

> On Dec 10, 2020, at 02:16, Sean Conner <sean at conman.org> wrote:
> 
>  -spc (Seriously, I feel like I'm the only one *doing* anything here)

:))

To be is to do. ?Socrates
To do is to be. ?Plato
Do-be-do-be-do. ?Sinatra

https://quoteinvestigator.com/2015/01/26/doing/

> On Dec 10, 2020, at 05:16, Michael Lazar <lazar.michael22 at gmail.com> wrote:
> 
> Next, my naive python test client just kind of works as-is [0][1]. It will
> convert unicode DNS names to punycode under the hood before doing the lookup.

Perhaps of interest:

How do I know when to do a UTF8 or punycode DNS query?
https://stackoverflow.com/questions/16837513/how-do-i-know-when-to-do-a-utf
8-or-punycode-dns-query

It was thus said that the Great Petite Abeille once stated:
> 
> 
> > On Dec 10, 2020, at 02:16, Sean Conner <sean at conman.org> wrote:
> > 
> >  -spc (Seriously, I feel like I'm the only one *doing* anything here)
> 
> :))
> 
> To be is to do. ?Socrates
> To do is to be. ?Plato
> Do-be-do-be-do. ?Sinatra

  You forgot:

  Do-do-do-do. -Serling

  -spc (There is a fifth dimension, beyond that which is known to man ...)

> On Dec 10, 2020, at 09:22, Sean Conner <sean at conman.org> wrote:
> 
>  Do-do-do-do. -Serling

?It may be said with a degree of assurance that not everything that meets 
the eye is as it appears.? 
? Rod Serling, The Twilight Zone

On Wed, Dec 09, 2020 at 08:16:34PM -0500,
 Sean Conner <sean at conman.org> wrote 
 a message of 51 lines which said:

> This internationalization stuff is complex and makes me want to
> throw up hands in the air, scream a bit, and go back to the
> simplicity of ASCII.

ASCII is not simple (think of case-insensitivity) and then only for
people whose latin is the first script they learned.

On Wed, Dec 09, 2020 at 11:16:30PM -0500,
 Michael Lazar <lazar.michael22 at gmail.com> wrote 
 a message of 55 lines which said:

> First of all, my domain registrar won't even let me put unicode
> characters in an A record without automatically converting them to
> punycode for me.

Small detail: it is an issue with the DNS *hoster*. Which may be the
same as the registrar or not (if you host your own authoritative name
servers).

> My server (running jetforce) also works as-is.

But not Gemserv (I have to figure out why).

On Wed, Dec 09, 2020 at 10:25:40PM -0500,
 Sean Conner <sean at conman.org> wrote 
 a message of 53 lines which said:

>   And if it's so simple, why not do it? But I get it, you'd rather wait
> until a yeah/nay decision is made.

It is reasonable to discuss it first because we need a *standard* way
of doing it. Clients and servers must agree or there will be no
interoperability.

Also, I suspect the problem is partially a social one:  programs are
written by programmers. Most programmers are familiar with english and
with the latin script. Therefore, the issue does not seem pressing for
most of them.

Hi

> > This internationalization stuff is complex and makes me want to
> > throw up hands in the air, scream a bit, and go back to the
> > simplicity of ASCII.
> 
> ASCII is not simple (think of case-insensitivity) and then only for
> people whose latin is the first script they learned.

I am struggling to take that statement seriously,
and not just because it breaks set theory :-)

Case conversion in ascii is xor 0x20 - that doesn't
even require a branch/comparison and can compile down
to a single assembly instruction.

This versus *many* tens or even hundreds of thousands of
lines of puny/unicode/etc logic.

But lets assume upper/lowercase characters in ascii
are confusing. That would be an argument to restrict
a simple system such as gemini urls to a subset of ascii
which excludes uppercase characters. Which I could support,
and which is effectively what dns ends up doing - as
do the majority of http urls. "Lowest common denominator
for maximum interoperability" is a good maxim.

If ascii case conversion is confusing, then this isn't
an excuse to grow this confusion by many orders of
magnitude. That makes the problem a lot worse.

  "Oops, I've burnt my toast - I know, lets solve that
   by burning down the house"

regards

marc

Petite Abeille <petite.abeille at gmail.com> writes:

>> On Dec 10, 2020, at 05:16, Michael Lazar <lazar.michael22 at gmail.com> wrote:
>> 
>> Next, my naive python test client just kind of works as-is [0][1]. It will
>> convert unicode DNS names to punycode under the hood before doing the lookup.
>
> Perhaps of interest:
>
> How do I know when to do a UTF8 or punycode DNS query?
> https://stackoverflow.com/questions/16837513/how-do-i-know-when-to-do-a-u
tf8-or-punycode-dns-query

You can unconditionally just run the punycode encoder over domain names,
though ? an all-ASCII domain name will be unchanged. The stackoverflow
question deals with non-internet domain names in Active Directory, which
we don't have to support.

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

> On Dec 10, 2020, at 15:20, Jason McBrayer <jmcbray at carcosa.net> wrote:
> 
> non-internet domain names in Active Directory, which
> we don't have to support.

Hmmm... so... no .local queries ala Cheshire?

https://tools.ietf.org/html/rfc6762#appendix-F

Perhaps worthwhile quoting in full:

Appendix F.  Use of UTF-8

   After many years of debate, as a result of the perceived need to
   accommodate certain DNS implementations that apparently couldn't
   handle any character that's not a letter, digit, or hyphen (and
   apparently never would be updated to remedy this limitation), the
   Unicast DNS community settled on an extremely baroque encoding called
   "Punycode".  Punycode is a remarkably ingenious encoding
   solution, but it is complicated, hard to understand, and hard to
   implement, using sophisticated techniques including insertion unsort
   coding, generalized variable-length integers, and bias adaptation.
   The resulting encoding is remarkably compact given the constraints,
   but it's still not as good as simple straightforward UTF-8, and it's
   hard even to predict whether a given input string will encode to a
   Punycode string that fits within DNS's 63-byte limit, except by
   simply trying the encoding and seeing whether it fits.  Indeed, the
   encoded size depends not only on the input characters, but on the
   order they appear, so the same set of characters may or may not
   encode to a legal Punycode string that fits within DNS's 63-byte
   limit, depending on the order the characters appear.  This is
   extremely hard to present in a user interface that explains to users
   why one name is allowed, but another name containing the exact same
   characters is not.  Neither Punycode nor any other of the "ASCII-
   Compatible Encodings" proposed for Unicast DNS may be used
   in Multicast DNS messages.  Any text being represented internally in
   some other representation must be converted to canonical precomposed
   UTF-8 before being placed in any Multicast DNS message.

On Thu, Dec 10, 2020 at 10:15:42AM +0100, Stephane Bortzmeyer wrote:

Hi folks!

> On Wed, Dec 09, 2020 at 10:25:40PM -0500,
>  Sean Conner <sean at conman.org> wrote
>  a message of 53 lines which said:
>
> >   And if it's so simple, why not do it? But I get it, you'd rather wait
> > until a yeah/nay decision is made.
>
> It is reasonable to discuss it first because we need a *standard* way
> of doing it. Clients and servers must agree or there will be no
> interoperability.

I have read most of the messages in this thread, i would just say that
one  of  the  problems  with  WWW is  that  browser  are  getting  not
manageable by a single user or a hobbyist programmer.

This  issue  lead to  centralization  (see  chromium and  the  company
behind) as most of us can easily see.

Adding  more complexity  and more  responsibility to  software authors
will shrink diversity in the gemini software landscape.

Of  course i  talk here  as  a client  author  here but  in the  niche
language i  chose (common lisp) i  was forced to write  my URL parsing
procedure, adding i18n domains will require  a lot of work because the
third party library  (the only one) Jason McBrayer  wrote some message
above does  not implement  punycode->unicode conversion (if  i checked
the right library, thanks to the author anyway, better than nothing!).

Probably many of you just thinking "who cares about CL?", and probably
this is the mindset that lead to the mess it is now the web.

Internationalized  hostname  has  advantages  but  how  this  adding
complexity impact software author? Is this complexity needed?

I have no answer, just would want to express some of my concerns.

Bye!
C.

PS: i am not a native English speaker (as you can see :-) )

> On Dec 10, 2020, at 16:54, cage <cage-dev at twistfold.it> wrote:
> 
> Internationalized  hostname  has  advantages  but  how  this  adding
> complexity impact software author? Is this complexity needed?

Ah, yes, le charme discret du r?gionalisme.

It all boils down to the unmeasurable joy of Unicode ?

> I have no answer, just would want to express some of my concerns.

Archibald 'Harry' Tuttle had about 3 lines in Terry Gilliam's' Brazil:

? Well, that's a pipe of a different color.
? Listen, this whole system of yours could be on fire and I couldn't even 
turn on the kitchen tap without filling out a twenty-seven B stroke six... 
bloody paperwork.
? Listen, kid, we're all in it together.

This sums it up in terms of retrofitting Unicode into ASCII:

https://www.youtube.com/watch?v=VRfoIyx8KfU

Perhaps Unicode should be abandoned altogether, and we all move back to to 
the original simplicity of ASCII. 

 From retro to retrograde. Sigh. 

It's all about the plumbing.

On Thu, Dec 10, 2020 at 05:27:05PM +0100, Petite Abeille wrote:
>
>
> > On Dec 10, 2020, at 16:54, cage <cage-dev at twistfold.it> wrote:
> >
> > Internationalized  hostname  has  advantages  but  how  this  adding
> > complexity impact software author? Is this complexity needed?
>
> Ah, yes, le charme discret du r?gionalisme.
>
> It all boils down to the unmeasurable joy of Unicode ?
>
> > I have no answer, just would want to express some of my concerns.
>
> Archibald 'Harry' Tuttle had about 3 lines in Terry Gilliam's' Brazil:

One of my favourite movie! I  love the "retr? style" terminal shown in
the office! :)

Bye!
C.

On Thu, Dec 10, 2020 at 10:55 AM cage <cage-dev at twistfold.it> wrote:


> I have read most of the messages in this thread, i would just say that
> one  of  the  problems  with  WWW is  that  browser  are  getting  not
> manageable by a single user or a hobbyist programmer.
>

This is not strictly a Web issue: it is a DNS issue and affects all
protocols, including FTP, email, Gopher, etc.

> Probably many of you just thinking "who cares about CL?"
>

As a Schemer, I definitely do care about it.


> Internationalized  hostname  has  advantages  but  how  this  adding
> complexity impact software author? Is this complexity needed?
>

It's a balancing act between the needs of software authors and the needs of
content authors.  If Gemini succeeds, the latter will be much more common.
I think that internationalized link lines (which will have to become part
of the definition of text/gemini) are very important to authors.  Whether
clients accept IRIs in the address bar (or equivalent) is up to the client
author.  And I very strongly feel that changing the Gemini *protocol* to
pass IRIs serves nobody and shouldn't even be considered.


> PS: i am not a native English speaker (as you can see :-) )
>

Good.  As the saying is, if you want to know if there is antisemitism in a
particular place, ask the Jews who live there.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
I now introduce Professor Smullyan, who will prove to you that either
he doesn't exist or you don't exist, but you won't know which.
                               --Melvin Fitting
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201210/38ec
235c/attachment.htm>

On Thu, Dec 10, 2020 at 11:27 AM Petite Abeille <petite.abeille at gmail.com>
wrote:


> It all boils down to the unmeasurable joy of Unicode ?
>

As someone who is intimately familiar with i18n in the pre-Unicode era, I
can say that things were 100 times worse then.  Unicode is flawed because
it had to compromise with existing encodings, which is why we need
normalization.  But without that compromise (which permits 1-1
convertibility from almost all encodings to and from Unicode), it would
never have been so widely adopted.


> Perhaps Unicode should be abandoned altogether, and we all move back to to
> the original simplicity of ASCII.


Perhaps we should abandon all modern languages and just use Latin on the
Internet.

John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
Gules six bars argent on a canton azure 50 mullets argent
six five six five six five six five and six
   --blazoning the U.S. flag <http://web.meson.org/blazonserver>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201210/6915
9af2/attachment-0001.htm>

It was thus said that the Great Stephane Bortzmeyer once stated:
> On Wed, Dec 09, 2020 at 10:25:40PM -0500,
>  Sean Conner <sean at conman.org> wrote 
>  a message of 53 lines which said:
> 
> >   And if it's so simple, why not do it? But I get it, you'd rather wait
> > until a yeah/nay decision is made.
> 
> It is reasonable to discuss it first because we need a *standard* way
> of doing it. Clients and servers must agree or there will be no
> interoperability.

  Okay,  Here's a IRI:

	gemini://caf?.mozz.us/files/?????.txt

  Please specify what a client and server MUST do to properly handle this.

  -spc

Sean Conner <sean at conman.org> writes:

>   Okay,  Here's a IRI:
>
> 	gemini://caf?.mozz.us/files/?????.txt
>
>   Please specify what a client and server MUST do to properly handle this.

Well, if I'm following all of these conversations correctly to date, I
believe the procedure looks like this:

1. Punycode the hostname.

2. Percent-encode reserved characters and non-US-ASCII characters in the
   path, query, and fragment components.

3. Make a DNS query with the punycoded hostname.

4. Send the punycode + percent-encoded URI as the request to the Gemini
   server.

5. The server parses the URI into scheme, host, port, path, query, and
   fragment components and then percent-decodes the path, query, and
   fragment strings.

6. The parsed and decoded URI information can then either be used to
   perform a file retrieval, generate a directory listing, or run a CGI
   script, ultimately sending back a valid Gemini response to the
   client. Redirect responses should make sure to percent-encode the
   path, query, and fragment components of the redirected URI.

My Gemini server (Space Age) handles steps 5 and 6 as described here (as
I suspect most Gemini servers do). Clients should already be performing
step 2 as per the Gemini spec.

I suspect the missing piece of the puzzle here is *just* having client
authors implement steps 1, 3, and 4 (for some definition of "just"). I
don't think these client changes would require any changes to the
current Gemini spec.

There is also the open question of whether servers should convert
punycoded hostnames back into unicode hostnames for the purposes of
virtual hosting (either via SNI or post-handshake). Since at least one
poster has indicated that the widespread unevenness in DNS support for
unicode has lead to the need to store A records in their punycoded form,
this suggests to me that virtual hosting may be performed most
universally by just directly matching the received punycoded domain
names.

Of course, YMMV.

Happy hacking,
  Gary

-- 
GPG Key ID: 7BC158ED
Use `gpg --search-keys lambdatronic' to find me
Protect yourself from surveillance: https://emailselfdefense.fsf.org
=======================================================================
()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

Why is HTML email a security nightmare? See https://useplaintext.email/

Please avoid sending me MS-Office attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

On Thu, Dec 10, 2020 at 8:12 PM Gary Johnson <lambdatronic at disroot.org>
wrote:

1. Punycode the hostname.
>

If there is one.  You can look for "//" on the left and the next "/" on the
right, so you don't need full parsing.

> 2. Percent-encode reserved characters and non-US-ASCII characters in the
>    path, query, and fragment components.
>

You don't want to escape the ASCII reserved characters, because they should
already be escaped.  Changing the path /foo/bar.gmi to %25foo%25bar.gmi
would be Evil and Wrong.  If you really want that path, you have to encode
it yourself.

In addition, you can safely %-encode the whole IRI reference without
parsing it, since Punycode names are always safe.
2.5. If the IRI is a relative reference, resolve it against the URI of the
text/gemini file that contains it.

3. Make a DNS query with the punycoded hostname.
>
> 4. Send the punycode + percent-encoded URI as the request to the Gemini
>    server.
>

Note that fragments must not be sent, so if there is one, chop it off.


> 5. The server parses the URI into scheme, host, port, path, query, and
>    fragment components and then percent-decodes the path, query, and
>    fragment strings.
>

Consequently, the server will not get a fragment string.  There would be no
need for fragment strings if they were understood on the server side;
they'd just be part of the path.

Whether it %-decodes or not is up to the server.  If it's serving a
conventional file system, then it needs to document whether it does such
decoding.  If it isn't, it can do whatever it wants to with the paths.


>  6. The parsed and decoded URI information can then either be used to

   perform a file retrieval, generate a directory listing, or run a CGI
>    script, ultimately sending back a valid Gemini response to the
>    client. Redirect responses should make sure to percent-encode the
>    path, query, and fragment components of the redirected URI.
>

Except not the fragment.

> Since at least one
> poster has indicated that the widespread unevenness in DNS support for
> unicode has lead to the need to store A records in their punycoded form,
>

Indeed, I don't think that any registrar using the standard DNS root will
even register non-punycoded names.  MS Active Directory DNS servers are
another story.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
This great college [Trinity], of this ancient university [Cambridge],
has seen some strange sights. It has seen Wordsworth drunk and Porson
sober. And here am I, a better poet than Porson, and a better scholar
than Wordsworth, somewhere betwixt and between.  --A.E. Housman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201210/33b5
9f84/attachment.htm>

> 4.  Send the punycode + percent-encoded URI as the request to the Gemini
>     server.

This probably makes sense since IRIs aren't being used. I originally
advocated for sending Unicode to the server, for the domain only, but that's
just a weird mix isn't it.

Good server software should be taking the admin's hostname input (from config)
and punycoding it though, so that the admin can enter the Unicode domain name
and not have to worry. Obviously this is outside of the spec, but I think
it's a good thing to implement.


makeworld

Before percent-encoding/punycoding, the URI needs to be NFC normalized.

As a matter of course, I'd say that servers should normalize the path 
before doing fs lookups/proxying it as well.

On Thu, Dec 10, 2020 at 04:54:52PM +0100,
 cage <cage-dev at twistfold.it> wrote 
 a message of 44 lines which said:

> Adding more complexity and more responsibility to software authors
> will shrink diversity in the gemini software landscape.

The idea is that it will not be done by the programmer (Unicode is
complicated) of the Gemini client but mostly by the libraries she or
he uses. It is the same with TLS: TLS is very complicated but most
people do not program it by themselves (and rightly so).

> Internationalized  hostname  has  advantages  but  how  this  adding
> complexity impact software author? Is this complexity needed?

Since I write Gemini clients, I have sympathy for this point of
view. However, let me quote RFC 8890

=> gemini://gemini.bortzmeyer.org/rfc-mirror/rfc8890.txt

4.5.  Deprioritizing Internal Needs

   There are several needs that are very visible to us as specification
   authors but should explicitly not be prioritized over the needs of
   end users.

   These include convenience for document editors, IETF process matters,
   and "architectural purity" for its own sake.


=> https://www.w3.org/TR/html-design-principles/#priority-of-constituencies
 See aso this statement by W3C

On Thu, Dec 10, 2020 at 04:09:19PM -0500,
 John Cowan <cowan at ccil.org> wrote 
 a message of 76 lines which said:

> Unicode is flawed because it had to compromise with existing
> encodings,

And also because human scripts (and languages) are a mess and Unicode
choosed, a long time ago, to deal with it instead of whining "why
can't they just speak english?"

> Perhaps we should abandon all modern languages and just use Latin on the
> Internet.

=> https://en.wikipedia.org/wiki/Lojban No, Lojban

On Thu, Dec 10, 2020 at 08:12:04PM -0500,
 Gary Johnson <lambdatronic at disroot.org> wrote 
 a message of 69 lines which said:

> 1. Punycode the hostname.

Not always, for the reasons explained in RFC 6055. To summarize: the
application does not always know which name resolution system will be used.

gemini://gemini.bortzmeyer.org/rfc-mirror/rfc6055.txt

> 3. Make a DNS query with the punycoded hostname.

Most applications don't do DNS queries, both because DNS is
complicated and because there are other name resolutions system. They
call a system routine (getaddrinfo() in C) to do the resolution.

> There is also the open question of whether servers should convert
> punycoded hostnames back into unicode hostnames for the purposes of
> virtual hosting (either via SNI or post-handshake). Since at least one
> poster has indicated that the widespread unevenness in DNS support for
> unicode has lead to the need to store A records in their punycoded form,
> this suggests to me that virtual hosting may be performed most
> universally by just directly matching the received punycoded domain
> names.

This is what Apache and Nginx do in the Web world (which does not mean
they are right).

On Thu, Dec 10, 2020 at 09:45:55PM -0500,
 John Cowan <cowan at ccil.org> wrote 
 a message of 177 lines which said:

> Indeed, I don't think that any registrar using the standard DNS root
> will even register non-punycoded names.

Counter-example about a similar case: the registry of .ws accepts
names with emojis, which are forbidden by the standard (because they
are symbols, not letters). So, anything can happen.

=> https://www.worldsite.ws/idn/emoji.dhtml?sponsor=index.dhtml And they boast about it

(Also, not all name registration goes through a registrar: when I edit
bortzmeyer.org, I can add what I want without any intermediary.)

On Fri, Dec 11, 2020 at 09:57:02AM +0100, Stephane Bortzmeyer wrote:

Hi!

> On Thu, Dec 10, 2020 at 04:54:52PM +0100,
>  cage <cage-dev at twistfold.it> wrote
>  a message of 44 lines which said:
>
> > Adding more complexity and more responsibility to software authors
> > will shrink diversity in the gemini software landscape.
>
> The idea is that it will not be done by the programmer (Unicode is
> complicated) of the Gemini client but mostly by the libraries she or
> he uses.

If such library does exists, otherwise  more and more works is needed,
and this will exclude the programmers  that have no time or skills (to
me very likely the former) to do the all the work.

> It is the same with TLS: TLS is very complicated but most
> people do not program it by themselves (and rightly so).

This  is matter  of  balance,  the advantages  of  TLS  are worth  the
complexity added, IDN? I am not sure.

> > Internationalized  hostname  has  advantages  but  how  this  adding
> > complexity impact software author? Is this complexity needed?
>
> Since I write Gemini clients, I have sympathy for this point of
> view. However, let me quote RFC 8890

OK the point is  valid to me, for TLS, but not for  IDN. Anyway i have
the impression i am in a minority  here, and i think i should start to
do a minimal wrapping of libidn at this point. :-)

Bye!
C.

On Thu, Dec 10, 2020 at 03:02:44PM -0500, John Cowan wrote:

Hi!

> On Thu, Dec 10, 2020 at 10:55 AM cage <cage-dev at twistfold.it> wrote:
>
>
> > I have read most of the messages in this thread, i would just say that
> > one  of  the  problems  with  WWW is  that  browser  are  getting  not
> > manageable by a single user or a hobbyist programmer.
> >
>
> This is not strictly a Web issue: it is a DNS issue and affects all
> protocols, including FTP, email, Gopher, etc.

Correct, i meant  that this is a client issue  deriving from debatable
choices.

> > Probably many of you just thinking "who cares about CL?"
> >
>
> As a Schemer, I definitely do care about it.

[OT] Nice! Do  yo have a preferred  dialect? I like Guile a  lot but i
fear i  end missing CLOS  (GOOPS is  not the same,  unfortunately) and
condition system.

>
> > Internationalized  hostname  has  advantages  but  how  this  adding
> > complexity impact software author? Is this complexity needed?
> >
>
> It's a balancing act between the needs of software authors and the needs of
> content authors.  If Gemini succeeds, the latter will be much more common.
> I think that internationalized link lines (which will have to become part
> of the definition of text/gemini) are very important to authors.  Whether
> clients accept IRIs in the address bar (or equivalent) is up to the client
> author.  And I very strongly feel that changing the Gemini *protocol* to
> pass IRIs serves nobody and shouldn't even be considered.

I agree!  My only concerns is  i have the impression  that client that
will  not supports  IRI will  be second  class citizen  in the  gemini
space, and they will die slowly.

So i think that IRI will be a de facto standard. :/

Bye!
C.

colecmac at protonmail.com writes:

> Good server software should be taking the admin's hostname input (from config)
> and punycoding it though, so that the admin can enter the Unicode domain name
> and not have to worry. Obviously this is outside of the spec, but I think
> it's a good thing to implement.

To implement, *and* to document ? if not in the spec, then in a 'best
practices for implementers' document.

-- 
Jason McBrayer      | ?Strange is the night where black stars rise,
jmcbray at carcosa.net | and strange moons circle through the skies,
                    | but stranger still is lost Carcosa.?
                    | ? Robert W. Chambers,The King in Yellow

On Fri, Dec 11, 2020 at 4:13 AM Stephane Bortzmeyer
<stephane at sources.org> wrote:
> Counter-example about a similar case: the registry of .ws accepts
> names with emojis, which are forbidden by the standard (because they
> are symbols, not letters). So, anything can happen.
However, those names are still punycoded. Most registries will not
accept punycoded emojis, but .ws does.

John Cowan <cowan at ccil.org> writes:

>> 2. Percent-encode reserved characters and non-US-ASCII characters in the
>>    path, query, and fragment components.

> You don't want to escape the ASCII reserved characters, because they should
> already be escaped.  Changing the path /foo/bar.gmi to %25foo%25bar.gmi
> would be Evil and Wrong.  If you really want that path, you have to encode
> it yourself.

Yes, that is quite right. I suppose we are using a different
interpretation of the phrase "reserved characters" here. For clarity, I
meant characters such as those in the string " ?#", which are either
forbidden (when unencoded) within the path, query, and fragment
components or are used to delimit them.

> 2.5. If the IRI is a relative reference, resolve it against the URI of the
> text/gemini file that contains it.

Yep.

>> 4. Send the punycode + percent-encoded URI as the request to the Gemini
>>    server.
>
> Note that fragments must not be sent, so if there is one, chop it off.

I'm not sure that is the case here. To quote the Gemini spec:

========================================================================
1.2 Gemini URI scheme

Resources hosted via Gemini are identified using URIs with the scheme
"gemini". This scheme is syntactically compatible with the generic URI
syntax defined in RFC 3986, but does not support all components of the
generic syntax. In particular, the authority component is allowed and
required, but its userinfo subcomponent is NOT allowed. The host
subcomponent is required. The port subcomponent is optional, with a
default value of 1965. The path, query and fragment components are
allowed and have no special meanings beyond those defined by the generic
syntax. Spaces in gemini URIs should be encoded as %20, not +.
========================================================================

Please note the text about fragment components being allowed. I'm not
currently aware of any good uses for them in Gemini, but the spec
supports them, so I've included that support in my server.

>> 5. The server parses the URI into scheme, host, port, path, query, and
>>    fragment components and then percent-decodes the path, query, and
>>    fragment strings.
>
> Consequently, the server will not get a fragment string.  There would be no
> need for fragment strings if they were understood on the server side;
> they'd just be part of the path.

See above.

>>  6. The parsed and decoded URI information can then either be used to
>>     perform a file retrieval, generate a directory listing, or run a
>>     CGI script, ultimately sending back a valid Gemini response to
>>     the client. Redirect responses should make sure to percent-encode
>>     the path, query, and fragment components of the redirected URI.
>>
>
> Except not the fragment.

Again, see above.

Yada yada...spec compliance...yada yada.

Happy hacking,
  Gary

-- 
GPG Key ID: 7BC158ED
Use `gpg --search-keys lambdatronic' to find me
Protect yourself from surveillance: https://emailselfdefense.fsf.org
=======================================================================
()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

Why is HTML email a security nightmare? See https://useplaintext.email/

Please avoid sending me MS-Office attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

On Friday, December 11, 2020 4:58 AM, cage <cage-dev at twistfold.it> wrote:

> I agree! My only concerns is i have the impression that client that
> will not supports IRI will be second class citizen in the gemini
> space, and they will die slowly.
>
> So i think that IRI will be a de facto standard. :/


I do not think this will happen, and if it starts to happen I will
fight against it. I don't think the idea of "de facto standards" fits within
the Gemini ethos at all. It's not supposed to be extensible and clients
aren't supposed go off and do random things while others have to decide
what to do and play catch-up. That is how the Web grew and became more complex,
and it's why we have only a few browsers today.

The ecosystem benefits when we all just stick to the standard, with the perhaps
obvious exception of demos and toys.

Stay united, Gemini!

Cheers,
makeworld

It was thus said that the Great Stephane Bortzmeyer once stated:
> On Thu, Dec 10, 2020 at 08:12:04PM -0500,
>  Gary Johnson <lambdatronic at disroot.org> wrote 
>  a message of 69 lines which said:
> 
> > 1. Punycode the hostname.
> 
> Not always, for the reasons explained in RFC 6055. To summarize: the
> application does not always know which name resolution system will be used.

  Yes, this is why I hata the incessent talking.  If you bothered to try it
on a few systems, you may have encountered *not* encoding with punycode
*also* fails.  I know, because I tried on a few systems I have access to,
and they all failed to look up "caf?.mozz.us" (yes, via getaddrinfo() even). 
They all worked when I looked up "xn--caf-dma.mozz.us".

  So now that I've injected some ugly reality in to a nice theoretical
discussion, what's next?

  -spc

It was thus said that the Great colecmac at protonmail.com once stated:
> On Friday, December 11, 2020 4:58 AM, cage <cage-dev at twistfold.it> wrote:
> 
> > I agree! My only concerns is i have the impression that client that
> > will not supports IRI will be second class citizen in the gemini
> > space, and they will die slowly.
> >
> > So i think that IRI will be a de facto standard. :/
> 
> I do not think this will happen, and if it starts to happen I will
> fight against it. 

  There's a reason why UTF-8 was selected as the default character set for
text/gemini, and one of those is to allow other people than English speakers
a means of expressing themsevles [1].  I don't think it's entirely
unreasonable to expect such a person to use Unicode for both domain name and
filenames [2].  Yes, tooling could be made to handle "canonicalizing" links
[3] but why not look into allowing IRIs?  Without an attempt at it, it would
be difficult to know what would work, what doesn't and where the difficulty,
if any, lie.  *That's* why I'm so insistent on coding up
"proof-of-concepts".  Just decreeing "this is how it shall be done" rarely
works out well [4].  And decreeing "this shall NOT be done" could put off
non-technical, non-English speaking people. [5]

> I don't think the idea of "de facto standards" fits within the Gemini
> ethos at all. It's not supposed to be extensible and clients aren't
> supposed go off and do random things while others have to decide what to
> do and play catch-up. That is how the Web grew and became more complex,
> and it's why we have only a few browsers today.

  And this is working out if the specification should be ammended to allow
IRIs, and if not, at at least have a jutification.

> The ecosystem benefits when we all just stick to the standard, with the
> perhaps obvious exception of demos and toys.

  One more point of reference.  The Gopher RFC (RFC-1436) states the use of
ISO-8859-1 for a character set.  It is wrong then, for gopher servers to
serve up UTF-8 documents even though it's not standard?  Yes, gopher is not
Gemini, but UTF-8 does seem to be a modern "de facto standard" in
gopherspace.

  -spc

[1]	For example, gemini://blekksprut.net/

[2]	Otherwise, punycode wouldn't exist.

[3]	Conversion from IRI to URI, with Unicode normalization,  prior to
	publication.

[4]	Such as X.200---lovingly developed and standardized but no one used
	it.  Or Xanadu.  Over 60 years of design work and still not working.

[5]	And I'm saying this being a thoroughly American mut that speaks only
	English.

It was thus said that the Great cage once stated:
> On Fri, Dec 11, 2020 at 09:57:02AM +0100, Stephane Bortzmeyer wrote:
> 
> > Since I write Gemini clients, I have sympathy for this point of
> > view. However, let me quote RFC 8890
> 
> OK the point is  valid to me, for TLS, but not for  IDN. Anyway i have
> the impression i am in a minority  here, and i think i should start to
> do a minimal wrapping of libidn at this point. :-)

  Here's the code I wrote to wrap libidn:

	https://github.com/spc476/lua-conmanorg/blob/master/src/idn.c

It's geared for Lua, but the code itself is in C, but it should be pretty
easy to see what is going on.

  -spc

On Fri, Dec 11, 2020 at 06:49:17PM -0500, Sean Conner wrote:

[...]

>
>   Here's the code I wrote to wrap libidn:
>
> 	https://github.com/spc476/lua-conmanorg/blob/master/src/idn.c
>
> It's geared for Lua, but the code itself is in C, but it should be pretty
> easy to see what is going on.

Thank  you Sean!   I am,  in fact  starting to  wrap libidn  (actually
libidn2) so far seems that i got a working unicode->ascii function. Of
course i will be happy to share  the results and maybe, if some people
are going to be  interested (and if i succeed!  :))  i could extract a
library from this code (it is integrated in the client at moment).

Bye!
C.

On Fri, Dec 11, 2020 at 08:49:45PM +0000, colecmac at protonmail.com wrote:

[...]

> >
> > So i think that IRI will be a de facto standard. :/
>
>
> I do not think this will happen, and if it starts to happen I will
> fight against it. I don't think the idea of "de facto standards" fits within
> the Gemini ethos at all. It's not supposed to be extensible and clients
> aren't supposed go off and do random things while others have to decide
> what to do and play catch-up. That is how the Web grew and became more complex,
> and it's why we have only a few browsers today.

I am with you  with this but i think the only way  to let not developers
do their own way is to clarify  issues in the specs as much as possible.

> The ecosystem benefits when we all just stick to the standard, with the perhaps
> obvious exception of demos and toys.

Totally agree of course! I am trying to do so! :)

> Stay united, Gemini!

:) ?

Bye!
C.

> On Dec 12, 2020, at 00:43, Sean Conner <sean at conman.org> wrote:
> 
> There's a reason why UTF-8 was selected as the default character set for
> text/gemini, and one of those is to allow other people than English speakers
> a means of expressing themsevles [1].  I don't think it's entirely
> unreasonable to expect such a person to use Unicode for both domain name and
> filenames [2]

Yes. agree. People should be able to express themselves in the most 
idiomatic -and frictionless- way they see fit. 

It's a moral imperative -and duty- for Gemini to make it so.

The year is 2020, no more easy and lazy excuses. This is not a technical 
choice, but a moral one. 

Timely article in The Economist:

Accent discrimination betrays a small mind
https://www.economist.com/books-and-arts/2020/12/12/accent-discrimination-b
etrays-a-small-mind

History will judge you: be one the right side.

Do the right thing Solderpunk.

On Fri, 11 Dec 2020 18:14:47 -0500
Sean Conner <sean at conman.org> wrote:

> I know, because I tried on a few systems I have access to,
> and they all failed to look up "caf?.mozz.us" (yes, via getaddrinfo() even). 
> They all worked when I looked up "xn--caf-dma.mozz.us".

I don't think Stephane means different systems in the sense of
different computers, but different systems as in different name
resolution systems. On my computer, for example, there are at least
three sources for names: DNS, mDNS and /etc/hosts. getaddrinfo() can
resolve using any of these systems via the name service switch, but IDN
only concerns DNS names.

caf?.mozz.us in your tests probably always resolved via DNS, but if you
had a caf?.local mDNS name or "caf?" in your hosts file you might not
be able to use IDN. I don't know about the hosts file, but mDNS for
example uses UTF-8 encoded names directly.

-- 
Philip
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201214/b3d7
ae31/attachment-0001.sig>

On Mon, Dec 14, 2020 at 11:46:49AM +0100,
 Philip Linde <linde.philip at gmail.com> wrote 
 a message of 45 lines which said:

> but different systems as in different name resolution systems.

Yes.

> caf?.mozz.us in your tests probably always resolved via DNS, but if
> you had a caf?.local mDNS name or "caf?" in your hosts file you
> might not be able to use IDN. I don't know about the hosts file, but
> mDNS for example uses UTF-8 encoded names directly.

I just tested with a Debian box and it seems getaddrinfo (both from a
Python program and from a C one, ping), requires the name in
/etc/hosts to be present in Punycode form (A-label).

The good news is that the Python program does not have to do
punycoding itself, it is handled automatically by the standard
library.

On Sat, Dec 12, 2020 at 12:10:48PM +0100, cage wrote:
> On Fri, Dec 11, 2020 at 06:49:17PM -0500, Sean Conner wrote:
>
> [...]
>
> >
> >   Here's the code I wrote to wrap libidn:
> >
> > 	https://github.com/spc476/lua-conmanorg/blob/master/src/idn.c
> >
> > It's geared for Lua, but the code itself is in C, but it should be pretty
> > easy to see what is going on.
>
> Thank  you Sean!   I am,  in fact  starting to  wrap libidn  (actually
> libidn2) so far seems that i got a working unicode->ascii function. Of
> course i will be happy to share  the results and maybe, if some people
> are going to be  interested (and if i succeed!  :))  i could extract a
> library from this code (it is integrated in the client at moment).

FWIW i managed to  switch from URI to IRI in  my client, fortunately i
was able to reuse  most of the URL parser. For Punycode  i wrapped a C
library.

Not sure if i did everything right but have met no regression so far.

If other  lisper (CL) are interested  we could arrange a  library from
this code.

Bye!
C.

> On Dec 14, 2020, at 16:09, cage <cage-dev at twistfold.it> wrote:
> 
> FWIW i managed to  switch from URI to IRI in  my client,

Bravo! :)

"Don't worry about what anybody else is going to do. The best way to 
predict the future is to invent it."
? Alan Kay

On Mon, Dec 14, 2020 at 04:37:19PM +0100, Petite Abeille wrote:
>
>
> > On Dec 14, 2020, at 16:09, cage <cage-dev at twistfold.it> wrote:
> >
> > FWIW i managed to  switch from URI to IRI in  my client,
>
> Bravo! :)

Thank you! :)

Honestly  most of  the  hard work  has  been done  by  libidn and  two
excellent lisp libraries; sometimes people complains that CL is full of
half-baked  libraries but  there are  some high  quality too,  FFI and
parser generator are  two that excels, in my opinion.  Also the author
of the latter actually helped me spotting a bug in the code. :)

Bye!
C.

It was thus said that the Great Philip Linde once stated:
> On Fri, 11 Dec 2020 18:14:47 -0500
> Sean Conner <sean at conman.org> wrote:
> 
> > I know, because I tried on a few systems I have access to,
> > and they all failed to look up "caf?.mozz.us" (yes, via getaddrinfo() even). 
> > They all worked when I looked up "xn--caf-dma.mozz.us".
> 
> I don't think Stephane means different systems in the sense of
> different computers, but different systems as in different name
> resolution systems. 

  Yes, I understand that, but on the systems I used, all FAILED to resolve
"caf?.mozz.us".  Does that mean the client just gives up and says "domain
not found?" because local configuration doesn't work with UTF-8 domain
names?  That, to me, sounds like what Stephane is advocating for when they
say "no conversion to punycode".  It's wonderful that Stephane's language
du jour will apparently handle it for the user, but are the rest of us out
of luck? [1]  THIS is what I'm asking about.

  -spc

[1]	And a response of "here's a nickel, get yourself a real computer
	language" is NOT a valid resonse.

Hello,

I didn't see any email mentioning this so I thought I'd share the
link here. Lagrange[1] has gone ahead with IDN support, and the details
are found in this post[2] by skyjake.

The relevant points are as follows.

> * The full URL is NFC normalized before sending it to a server.
> * Domain names with non-ASCII characters are encoded to Punycode before
>   doing a DNS lookup. The Punycode version of the domain name is sent to
>   the server in the request URL, and also used for verifying the server
>   certificate.

This is what I plan on doing in Amfora as well. I will defer to Solderpunk's
judgement, which is coming[3], but until then that's my plan. The only
difference is that I was planning on allowing both punycoded domains and IDNs
in certs, to simplify things for sysadmins. But if Lagrange isn't allowing
it, then maybe I shouldn't... this is quickly approaching "de facto standard"
territory.

For now I will err on the permissive side in that case, allowing both, but this
is something I'd like hear from Solderpunk on.

gemget will do the same, as it uses the go-gemini[4] library as well.


1: https://gmi.skyjake.fi/lagrange/
2: gemini://skyjake.fi/gemlog/2020-12_idns-in-lagrange.gmi
3: gemini://gemini.circumlunar.space/~solderpunk/pikkulog/2020-12.gmi
4: https://github.com/makeworld-the-better-one/go-gemini/issues/10


Cheers,
makeworld