ANN: go-hg — Mercury Protocol client & server library for Go programming language

Hello everyone,
(And Happy Halloween for those who celebrate Halloween.)

This is my first time posting to this mailing list.

I implemented the Mercury Protocol as a Go library:

https://github.com/reiver/go-hg

This implementation includes both the client & server side of the Mercury
Protocol.

For those familiar with the Go programming language — I tried to make
package hg similar to Go's built-in "net/http" library. So that it would
seem familiar; and thus hopefully easier to work with.

⁂

For me I prefer to separate out the TLS part from the server part. So, this
implementation isn't an attempt to get rid of encryption, but instead a way
of dividing up the technology to make it easier to work with.

I was originally doing this in a Gemini Protocol Go implementation, and
calling the TLS-free version "naked Gemini". But when I started reading the
mailing list archive, and some of the gemlogs — still working through them
— and noticed Mercury was the same thing, I created Go package hg.

Coincidentally, I was using TCP port 1961 for my "naked Gemini". I noticed
someone else on the mailing list happened to pick that too. (Although for
different reasons that I did.)

⁂

Feedback is welcomed. If anyone has any questions, comments, or complaints,
let me know.

--
Charles Iliya Krempeaux, B.Sc.
cikrempeaux@gmail.com

It was thus said that the Great Charles Iliya Krempeaux once stated:
> Hello everyone,

  Hello.

> For me I prefer to separate out the TLS part from the server part. So, this
> implementation isn't an attempt to get rid of encryption, but instead a way
> of dividing up the technology to make it easier to work with.
> 
> I was originally doing this in a Gemini Protocol Go implementation, and
> calling the TLS-free version "naked Gemini". But when I started reading the
> mailing list archive, and some of the gemlogs — still working through them
> — and noticed Mercury was the same thing, I created Go package hg.

  The Mercury protocol:

	https://portal.mozz.us/gemini/gemini.circumlunar.space/users/solderpunk/ge
mlog/the-mercury-protocol.gmi

was (at the time solderpunk proposed it) a thought experiement and is
actually bit less than the current Gemini-TLS---it's more akin to the
original protocol he propsed with single digit status codes [1] and a very
simple native text format.  The critical part of the Mercury "spec":

	3. ... then a lot of distinctions made by the remaining codes (e.g. 
	temporary vs permanent redirects or failures) become far less
	important, so we can get rid of more codes and end up below 10,
	allowing them to be single digits.

	4. The 'charset' parameter from the text/gemini MIME type is removed
	and UTF-8 encoding is obligatory.  The 'lang' parameter currently
	under discussion for Gemini is not added.

	5. The text/gemini syntax is stripped back to just two line types:
	links, and plain text.  Plain text lines are still wrapped by the
	client, as they currently are in Gemini.

  As for the requirement for TLS for Gemini, solderpunk explained his ideas
in this post:

	gopher://zaibatsu.circumlunar.space/0/~solderpunk/phlog/why-gopher-needs-crypto.txt

  In fact, a lot of the early history of Gemini is documented here:

	gopher://zaibatsu.circumlunar.space:70/1/~solderpunk/gemini

  -spc

[1]	gopher://zaibatsu.circumlunar.space/0/~solderpunk/gemini/status-codes.txt

A number of things to reply to. I wasn't sure whether to reply to
everything in one giant e-mail, or to reply in separate e-mails (that could
turn into their own separate threads). I think I'll create a small number
of separate replies to make it easier for others to follow.

Regarding:
gopher://
zaibatsu.circumlunar.space/0/~solderpunk/phlog/why-gopher-needs-crypto.txt

I think the main topics of this is:

№1: being able to detect (or prevent) content modification, and

№2: being able to protect one's privacy and make spying very difficult (if
not impossible).

(Please correct me if I missed anything.)

Let's get technical about this —

I haven't read the Gopher spec in a long time, so don't recall whether
there is something technical that would prevent it, but —

One could try to use content-addressing to try to detect content
modification.

For example, there could be a convention created (and Gopher clients
modified) such that the path in the gopher URL would contain a digest (from
a cryptographic hash function) of the content. For example:

gopher://
example.com/content/base64/sha3-512/ld7McvClCuTZ1TeOGyJSWHz8cZd+QyksjxuEZIJ
IUJ8bwYvG8LDQuGBqZD7/YdYRroTm+9SiaDFlcGvW/UizNA==

Notice that there are three main parts to this:

• base64
• sha3-512
•
ld7McvClCuTZ1TeOGyJSWHz8cZd+QyksjxuEZIJIUJ8bwYvG8LDQuGBqZD7/YdYRroTm+9SiaDFlcGvW/UizNA==

The gibberish is base64 encoding of the digest of a sha3-512 hash function.

(One could use base64url if they didn't want the gibberish to have the "/"
symbol.)

Someone would need to modify Gopher clients to recognize that type of
gopher URL, and then, once the data is downloaded, verify that its digest
matches the digest in the URL.

⁂

And encryption (such as TLS, mentioned in the document) could help prevent
the spying to help protect privacy

(Although there are other options than just TLS.)

⁂

But, if you are using your ISP's DNS servers, they can still see what
Internet domain names you are resolving. Which breaks some of your privacy.

(Personally, I would prefer to move away from the Internet domain name
system, for various reasons.)

⁂

As a side note — if you are interested in privacy, you should look into
what information mobile phone providers, and credit card companies (and
their partners) are collecting, and "sharing".



--
Charles Iliya Krempeaux, B.Sc.
cikrempeaux@gmail.com



On Sun, Oct 31, 2021 at 10:27 PM Sean Conner <sean@conman.org> wrote:

> It was thus said that the Great Charles Iliya Krempeaux once stated:
> > Hello everyone,
>
>   Hello.
>
> > For me I prefer to separate out the TLS part from the server part. So,
> this
> > implementation isn't an attempt to get rid of encryption, but instead a
> way
> > of dividing up the technology to make it easier to work with.
> >
> > I was originally doing this in a Gemini Protocol Go implementation, and
> > calling the TLS-free version "naked Gemini". But when I started reading
> the
> > mailing list archive, and some of the gemlogs — still working through
> them
> > — and noticed Mercury was the same thing, I created Go package hg.
>
>   The Mercury protocol:
>
>
> https://portal.mozz.us/gemini/gemini.circumlunar.space/users/solderpunk/g
emlog/the-mercury-protocol.gmi
>
> was (at the time solderpunk proposed it) a thought experiement and is
> actually bit less than the current Gemini-TLS---it's more akin to the
> original protocol he propsed with single digit status codes [1] and a very
> simple native text format.  The critical part of the Mercury "spec":
>
>         3. ... then a lot of distinctions made by the remaining codes
> (e.g.
>         temporary vs permanent redirects or failures) become far less
>         important, so we can get rid of more codes and end up below 10,
>         allowing them to be single digits.
>
>         4. The 'charset' parameter from the text/gemini MIME type is
> removed
>         and UTF-8 encoding is obligatory.  The 'lang' parameter currently
>         under discussion for Gemini is not added.
>
>         5. The text/gemini syntax is stripped back to just two line types:
>         links, and plain text.  Plain text lines are still wrapped by the
>         client, as they currently are in Gemini.
>
>   As for the requirement for TLS for Gemini, solderpunk explained his ideas
> in this post:
>
>         gopher://
> zaibatsu.circumlunar.space/0/~solderpunk/phlog/why-gopher-needs-crypto.txt
>
>   In fact, a lot of the early history of Gemini is documented here:
>
>         gopher://zaibatsu.circumlunar.space:70/1/~solderpunk/gemini
>
>   -spc
>
> [1]     gopher://
> zaibatsu.circumlunar.space/0/~solderpunk/gemini/status-codes.txt
>

Regarding:
>
> The Mercury protocol:
>
>
> https://portal.mozz.us/gemini/gemini.circumlunar.space/users/solderpunk/g
emlog/the-mercury-protocol.gmi
>
> was (at the time solderpunk proposed it) a thought experiement and is
> actually bit less than the current Gemini-TLS---it's more akin to the
> original protocol he propsed with single digit status codes [1] and a very
> simple native text format.  The critical part of the Mercury "spec":
>
>        [...]



        5. The text/gemini syntax is stripped back to just two line types:
>         links, and plain text.  Plain text lines are still wrapped by the
>         client, as they currently are in Gemini.
>

This one feels technically challenging to enforce, since they have the same
MIME type (text/gemini).

MIME types aren't the only way to deal with types, but if one is using MIME
types, shouldn't these two different file data formats have different MIME
types?

(Or maybe I misunderstood. And it was implicitly meant for there to be a
different MIME type for this file data format.)

--
Charles Iliya Krempeaux, B.Sc.
cikrempeaux@gmail.com



On Sun, Oct 31, 2021 at 10:27 PM Sean Conner <sean@conman.org> wrote:

> It was thus said that the Great Charles Iliya Krempeaux once stated:
> > Hello everyone,
>
>   Hello.
>
> > For me I prefer to separate out the TLS part from the server part. So,
> this
> > implementation isn't an attempt to get rid of encryption, but instead a
> way
> > of dividing up the technology to make it easier to work with.
> >
> > I was originally doing this in a Gemini Protocol Go implementation, and
> > calling the TLS-free version "naked Gemini". But when I started reading
> the
> > mailing list archive, and some of the gemlogs — still working through
> them
> > — and noticed Mercury was the same thing, I created Go package hg.
>
>   The Mercury protocol:
>
>
> https://portal.mozz.us/gemini/gemini.circumlunar.space/users/solderpunk/g
emlog/the-mercury-protocol.gmi
>
> was (at the time solderpunk proposed it) a thought experiement and is
> actually bit less than the current Gemini-TLS---it's more akin to the
> original protocol he propsed with single digit status codes [1] and a very
> simple native text format.  The critical part of the Mercury "spec":
>
>         3. ... then a lot of distinctions made by the remaining codes
> (e.g.
>         temporary vs permanent redirects or failures) become far less
>         important, so we can get rid of more codes and end up below 10,
>         allowing them to be single digits.
>
>         4. The 'charset' parameter from the text/gemini MIME type is
> removed
>         and UTF-8 encoding is obligatory.  The 'lang' parameter currently
>         under discussion for Gemini is not added.
>
>         5. The text/gemini syntax is stripped back to just two line types:
>         links, and plain text.  Plain text lines are still wrapped by the
>         client, as they currently are in Gemini.
>
>   As for the requirement for TLS for Gemini, solderpunk explained his ideas
> in this post:
>
>         gopher://
> zaibatsu.circumlunar.space/0/~solderpunk/phlog/why-gopher-needs-crypto.txt
>
>   In fact, a lot of the early history of Gemini is documented here:
>
>         gopher://zaibatsu.circumlunar.space:70/1/~solderpunk/gemini
>
>   -spc
>
> [1]     gopher://
> zaibatsu.circumlunar.space/0/~solderpunk/gemini/status-codes.txt
>

Regarding:
 gopher://zaibatsu.circumlunar.space/0/~solderpunk/gemini/status-codes.txt

A good example of this latter problem, IMHO, is "410 Gone" (which is
> actually in the Conman Gemini server!).  If this is made official in the
> Gemini spec, it sends the message that Real Servers which have a Proper
> Full Implementation should remember every one of their URLs which *has*
> been valid in the past so it can respond to requests for them with 410
> instead of 404.  Similarly a Real Client should remember every 410 it gets
> so that it doesn't request them again.  In the real world, almost nobody
> does this with HTTP, so it's basically dead weight in the spec.
>

And yet:
https://gemini.circumlunar.space/docs/specification.html

52 GONE



The resource requested is no longer available and will not be available
> again. Search engines and similar tools should remove this resource from
> their indices. Content aggregators should stop requesting the resource and
> convey to their human users that the subscribed resource is gone. (cf HTTP
> 410)
>

I'm assuming there is some story about how this got in there.

(I wonder if it was problems with bots.)

--
Charles Iliya Krempeaux, B.Sc.
cikrempeaux@gmail.com



On Sun, Oct 31, 2021 at 10:27 PM Sean Conner <sean@conman.org> wrote:

> It was thus said that the Great Charles Iliya Krempeaux once stated:
> > Hello everyone,
>
>   Hello.
>
> > For me I prefer to separate out the TLS part from the server part. So,
> this
> > implementation isn't an attempt to get rid of encryption, but instead a
> way
> > of dividing up the technology to make it easier to work with.
> >
> > I was originally doing this in a Gemini Protocol Go implementation, and
> > calling the TLS-free version "naked Gemini". But when I started reading
> the
> > mailing list archive, and some of the gemlogs — still working through
> them
> > — and noticed Mercury was the same thing, I created Go package hg.
>
>   The Mercury protocol:
>
>
> https://portal.mozz.us/gemini/gemini.circumlunar.space/users/solderpunk/g
emlog/the-mercury-protocol.gmi
>
> was (at the time solderpunk proposed it) a thought experiement and is
> actually bit less than the current Gemini-TLS---it's more akin to the
> original protocol he propsed with single digit status codes [1] and a very
> simple native text format.  The critical part of the Mercury "spec":
>
>         3. ... then a lot of distinctions made by the remaining codes
> (e.g.
>         temporary vs permanent redirects or failures) become far less
>         important, so we can get rid of more codes and end up below 10,
>         allowing them to be single digits.
>
>         4. The 'charset' parameter from the text/gemini MIME type is
> removed
>         and UTF-8 encoding is obligatory.  The 'lang' parameter currently
>         under discussion for Gemini is not added.
>
>         5. The text/gemini syntax is stripped back to just two line types:
>         links, and plain text.  Plain text lines are still wrapped by the
>         client, as they currently are in Gemini.
>
>   As for the requirement for TLS for Gemini, solderpunk explained his ideas
> in this post:
>
>         gopher://
> zaibatsu.circumlunar.space/0/~solderpunk/phlog/why-gopher-needs-crypto.txt
>
>   In fact, a lot of the early history of Gemini is documented here:
>
>         gopher://zaibatsu.circumlunar.space:70/1/~solderpunk/gemini
>
>   -spc
>
> [1]     gopher://
> zaibatsu.circumlunar.space/0/~solderpunk/gemini/status-codes.txt
>

It was thus said that the Great Charles Iliya Krempeaux once stated:
> A number of things to reply to. I wasn't sure whether to reply to
> everything in one giant e-mail, or to reply in separate e-mails (that could
> turn into their own separate threads). I think I'll create a small number
> of separate replies to make it easier for others to follow.
> 
> Regarding:
> gopher://
> zaibatsu.circumlunar.space/0/~solderpunk/phlog/why-gopher-needs-crypto.txt
> 
> I think the main topics of this is:
> 
> №1: being able to detect (or prevent) content modification, and
> 
> №2: being able to protect one's privacy and make spying very difficult (if
> not impossible).
> 
> (Please correct me if I missed anything.)
> 
> Let's get technical about this —
> 
> I haven't read the Gopher spec in a long time, so don't recall whether
> there is something technical that would prevent it, but —

  Yes there is: <http://boston.conman.org/2019/03/31.1> Basically, it's hard
to retrofit TLS into gopher without breaking existing clients.   You could
possibly force it, <http://boston.conman.org/2021/09/28.1>, but there are
security concerns about forced downgrades.  

> One could try to use content-addressing to try to detect content
> modification.
> 
> For example, there could be a convention created (and Gopher clients
> modified) such that the path in the gopher URL would contain a digest (from
> a cryptographic hash function) of the content. For example:
> 
> gopher://
> example.com/content/base64/sha3-512/ld7McvClCuTZ1TeOGyJSWHz8cZd+QyksjxuEZ
IJIUJ8bwYvG8LDQuGBqZD7/YdYRroTm+9SiaDFlcGvW/UizNA==
> 
> Notice that there are three main parts to this:
> 
> • base64
> • sha3-512
> •
> ld7McvClCuTZ1TeOGyJSWHz8cZd+QyksjxuEZIJIUJ8bwYvG8LDQuGBqZD7/YdYRroTm+9Sia
DFlcGvW/UizNA==
> 
> The gibberish is base64 encoding of the digest of a sha3-512 hash function.
> 
> (One could use base64url if they didn't want the gibberish to have the "/"
> symbol.)
> 
> Someone would need to modify Gopher clients to recognize that type of
> gopher URL, and then, once the data is downloaded, verify that its digest
> matches the digest in the URL.

  There's an awful large number of gopher clients that would need updating,
and probably won't.  Also, this topic might be better discussed on the
gohper mailing list: <https://lists.debian.org/gopher-project/>.

> And encryption (such as TLS, mentioned in the document) could help prevent
> the spying to help protect privacy
> 
> (Although there are other options than just TLS.)

  And as I've stated before, present both a server and client as a
proof-of-concept, then it can be discussed.  Until then, it's a no go (at
least, that would be my stance but I stepped down from Gemini development).

  -spc

It was thus said that the Great Charles Iliya Krempeaux once stated:
> 
> (Or maybe I misunderstood. And it was implicitly meant for there to be a
> different MIME type for this file data format.)

  People seem to gloss over the THOUGHT EXPERIMENT part of the Mercury
specification. 

  -spc

It was thus said that the Great Charles Iliya Krempeaux once stated:
> Regarding:
>  gopher://zaibatsu.circumlunar.space/0/~solderpunk/gemini/status-codes.txt
> 
> > A good example of this latter problem, IMHO, is "410 Gone" (which is
> > actually in the Conman Gemini server!).  If this is made official in the
> > Gemini spec, it sends the message that Real Servers which have a Proper
> > Full Implementation should remember every one of their URLs which *has*
> > been valid in the past so it can respond to requests for them with 410
> > instead of 404.  Similarly a Real Client should remember every 410 it gets
> > so that it doesn't request them again.  In the real world, almost nobody
> > does this with HTTP, so it's basically dead weight in the spec.
> >
> 
> And yet:
> https://gemini.circumlunar.space/docs/specification.html
> 
> 52 GONE
> 
> I'm assuming there is some story about how this got in there.

  Yes.  I convinced solderpunk to add it.  Along with both temporary and
permanent redirections.  

  -spc

On 10/31/21 21:30, Charles Iliya Krempeaux wrote:

> For me I prefer to separate out the TLS part from the server part. So, 
this implementation isn't an attempt to get rid of encryption, but instead 
a way of dividing up the technology to make it easier to work with.
> 
> I was originally doing this in a Gemini Protocol Go implementation, and 
calling the TLS-free version "naked Gemini". But when I started reading 
the mailing list archive, and some of the gemlogs — still working through 
them — and noticed Mercury was the same thing, I created Go package hg.

i've interpreted mercury similarly, though it doesn't exactly match the 
thought experiment post that solderpunk made about it. i like this 
approach as well to be honest. my gemini server project has a similar 
approach and serves "mercury" or gemini sans tls on port 1958. i assume 
this isn't canonical by any means, but it amuses me.

good luck with your project!

On Mon, 2021-11-01 at 18:59 -0400, Sean Conner wrote:
> It was thus said that the Great Charles Iliya Krempeaux once stated:
> > Regarding:
> >  gopher://zaibatsu.circumlunar.space/0/~solderpunk/gemini/status-codes.txt
> > 
> > > A good example of this latter problem, IMHO, is "410 Gone" (which is
> > > actually in the Conman Gemini server!).  If this is made official in the
> > > Gemini spec, it sends the message that Real Servers which have a Proper
> > > Full Implementation should remember every one of their URLs which *has*
> > > been valid in the past so it can respond to requests for them with 410
> > > instead of 404.  Similarly a Real Client should remember every 410 it gets
> > > so that it doesn't request them again.  In the real world, almost nobody
> > > does this with HTTP, so it's basically dead weight in the spec.
> > > 
> > 
> > And yet:
> > https://gemini.circumlunar.space/docs/specification.html
> > 
> > 52 GONE
> > 
> > I'm assuming there is some story about how this got in there.
> 
>   Yes.  I convinced solderpunk to add it.  Along with both temporary and
> permanent redirections.

I honestly think this is a good code. Basic clients of course don't have
to remember their past invalid links (because they don't even have to
recognize the difference between codes 52 and 50), but for the server,
remembering past links is as simple as appending to a text file when a
new file is added.

Cheers,
-- 
DJ Chase
They, Them, Theirs

Interesting — did you post your work online somewhere?

--
Charles Iliya Krempeaux, B.Sc.
cikrempeaux@gmail.com



On Mon, Nov 1, 2021 at 4:19 PM <gemini@xj-ix.luxe> wrote:

>
>
> On 10/31/21 21:30, Charles Iliya Krempeaux wrote:
>
> > For me I prefer to separate out the TLS part from the server part. So,
> > this implementation isn't an attempt to get rid of encryption, but
> > instead a way of dividing up the technology to make it easier to work
> with.
> >
> > I was originally doing this in a Gemini Protocol Go implementation, and
> > calling the TLS-free version "naked Gemini". But when I started reading
> > the mailing list archive, and some of the gemlogs — still working
> > through them — and noticed Mercury was the same thing, I created Go
> > package hg.
>
> i've interpreted mercury similarly, though it doesn't exactly match the
> thought experiment post that solderpunk made about it. i like this
> approach as well to be honest. my gemini server project has a similar
> approach and serves "mercury" or gemini sans tls on port 1958. i assume
> this isn't canonical by any means, but it amuses me.
>
> good luck with your project!
>

On 11/3/21 00:03, Charles Iliya Krempeaux wrote:
> Interesting — did you post your work online somewhere?
> 

nothing especially coherent, but it is available. various prototypes 
written in rcshell and sed. likely i'll rewrite them in erlang or scheme in the future.

=> https://source.heropunch.luxe/gemini/tamactiluya/ source code
=> gemini://xj-ix.luxe/wiki/ docs and other remarks

Hi Charles!

On Mon, 1 Nov 2021 00:01:19 -0700
Charles Iliya Krempeaux <cikrempeaux@gmail.com> wrote:

> I haven't read the Gopher spec in a long time, so don't recall whether
> there is something technical that would prevent it, but —
> 
> One could try to use content-addressing to try to detect content
> modification.

This is tangential, but I've done some experiments with Gopher and IPFS
before that you may find interesting:

https://github.com/boomlinde/ipfs-gopher

IPFS is content addressed and peer-to-peer via a distributed hash
table. ipfs-gopher will run a local gopher server that parses a special
menu format that can be pinned to the IPFS network, exposing a
Gopher-compatible way to browse them and other documents on IPFS.

-- 
Philip

Very cool.

It is even in Golang 🙂

--
Charles Iliya Krempeaux, B.Sc.
cikrempeaux@gmail.com



On Fri, Nov 5, 2021 at 7:26 PM Philip Linde <linde.philip@gmail.com> wrote:

> Hi Charles!
>
> On Mon, 1 Nov 2021 00:01:19 -0700
> Charles Iliya Krempeaux <cikrempeaux@gmail.com> wrote:
>
> > I haven't read the Gopher spec in a long time, so don't recall whether
> > there is something technical that would prevent it, but —
> >
> > One could try to use content-addressing to try to detect content
> > modification.
>
> This is tangential, but I've done some experiments with Gopher and IPFS
> before that you may find interesting:
>
> https://github.com/boomlinde/ipfs-gopher
>
> IPFS is content addressed and peer-to-peer via a distributed hash
> table. ipfs-gopher will run a local gopher server that parses a special
> menu format that can be pinned to the IPFS network, exposing a
> Gopher-compatible way to browse them and other documents on IPFS.
>
> --
> Philip
>