Synchronizing bookmarks - Request for comments

Hello,

I?ve put together an idea of simple specification to allow clients to 
synchronize bookmarks and feeds across Gemini, as this is something that 
I?m currently missing in my daily Gemini use.

You can read it at 
gemini://gemlog.lanterne.chilliet.eu/2020-12-15%20link-sync-rev-1.en.gmi

I?m particularly interested in comments from client authors:
* Would you consider implementing support for something like this?
* Does your client have some kind of plugin system which would allow a 
third-party to add support?

C?me

Hello,

I was initially excited by this email because I thought it was proposal
for how to share bookmarks across clients on the same machine, similar to
Drew's idea for TOFU. But it seems it's about synchronizing bookmarks and
subscriptions across the network, using the Gemini protocol.

First off I appreciate the write up, it's nice to have these longer-form
ideas in Gemini. Secondly, I think this misusing the Gemini the protocol
and is outside its scope. I think this is similar to how Gemini doesn't have
a clear way for authors to send files to a server. If you want to push data
to a server, I think Gemini is not the thing to use.

While I personally don't have the need to synchronize my Gemini bookmarks
and subscriptions, several Amfora users have told me that they synchronize
their Amfora data and settings, and they do so through external means as one
might expect. Git repos mainly, but I wouldn't be surprised to hear about people
using sftp, rsync, etc. I don't think this bespoke protocol would be advantageous
to these people.

Obvious these are technical solutions, but I'm biased because Amfora is a terminal
based client. But I think that synchronizing files across the network is already
a technical operation, and a more user-friendly option like syncthing would
still work fine for people.


To answer the questions directly asked:

> Would you consider implementing support for something like this?

Not really, for the reasons I explained above. I'd be happy to hear why you think
I'm wrong though.

> Does your client have some kind of plugin system which would allow a third-party
> to add support?

Amfora does not. It's an interesting idea and I'm open to it, but I'm not entirely
sure how it would work, or what sort of things the plugins would hook in to.


Now, I don't want to hijack your thread... but if anyone is interested in a common
format for Gemini bookmarks as I mentioned earlier, an Amfora user proposed[1]
XBEL[2] to me, and I plan on implementing it in the next-next release, replacing the
TOML format being currently used, which was a bad choice for that purpose. Is
this something other client authors would like to do as well?


1: https://github.com/makeworld-the-better-one/amfora/issues/68#issuecomment-674424896
2: http://xbel.sourceforge.net/


Cheers,
makeworld

On Tue Dec 15, 2020 at 4:32 PM EST,  wrote:
> Now, I don't want to hijack your thread... but if anyone is interested in a common
> format for Gemini bookmarks as I mentioned earlier, an Amfora user proposed[1]
> XBEL[2] to me, and I plan on implementing it in the next-next release, replacing the
> TOML format being currently used, which was a bad choice for that purpose. Is
> this something other client authors would like to do as well?

What about using a plain Gemini text file for storing bookmarks? That
would make it very easy for people to manage their bookmarks externally.
You could implement folders as level 1 or 2 headings in the file.

For example:

# Folder 1
=> gemini://gemini.circumlunar.space Project Gemini
=> gemini://example.com Example Bookmark

# Folder 2
=> gemini://example.org Example Bookmark

Le mardi 15 d?cembre 2020, 22:32:15 CET colecmac at protonmail.com a ?crit :
> To answer the questions directly asked:
> 
> > Would you consider implementing support for something like this?
> 
> Not really, for the reasons I explained above. I'd be happy to hear why you think
> I'm wrong though.

 From what I gather from your answer, your take on the synchronizing 
bookmarks across clients/computers is to use a common format (like xbel), 
and to use an external system to synchronize this file across computers.

I can appreciate the KISS aspect of this.

My usecase is more like synchronizing lagrange and a client on my mobile 
phone (currently ariane, but I may change if another one come up), and I?m 
not convinced it will be one day possible/easy to do it like you 
suggested, given how files are handled on Android.

An other answer to my problem would be using something like 
https://github.com/kensanata/moku-pona and use it though any client, but I 
like having my feeds and bookmarks well integrated into my brower 
lagrange, so I?d really prefer a solution which fits in the client and not 
only a Gemini capsule with some interaction.

My solution is also inspired by web browser addons like wallabag and 
pocket which adds a read-it-later feature well integrated in the browser.

C?me

On 15. Dec 2020, at 23:08, C?me Chilliet <come at chilliet.eu> wrote:

> I?ve put together an idea of simple specification to allow clients to 
synchronize bookmarks and feeds across Gemini, as this is something that 
I?m currently missing in my daily Gemini use.
> 
> You can read it at 
gemini://gemlog.lanterne.chilliet.eu/2020-12-15%20link-sync-rev-1.en.gmi

I think this is a valid use case, and it's something I have been thinking 
about myself just running Lagrange on a number of different computers. 
When you bring in a mix of different clients and desktop/mobile operating 
systems, sync is even more important.

The lowest common denominator should be export to/import from a .gmi file. 
Such a file (that Adnan Maolood suggested, for example) is guaranteed to 
be at least viewable by all clients. It supports folders, has no line 
length limitations, and can support tags as well (e.g., an optional 
line/bullet/quote following a link).  The user can then manually manage 
the files via any means they see fit.

However, the UX of doing that on an ongoing basis to sync your 
bookmarks/feeds isn't great. An automated mechanism is certainly more desirable.

When it comes to basing sync on the Gemini protocol, it makes a certain 
amount of sense (clients know how to do Gemini requests) but it's not a perfect fit:

* one add/del per request ? what if you have a 100 bookmarks to sync?
* request URL length limit
* does not address tags, folders, manual ordering

> * Would you consider implementing support for something like this?

Something like this, yes.

The current proposal needs revising, though. I suggest creating a 
mechanism that allows using Gemini subscriptions as a core part of the 
sync. A client would view and/or subscribe to a bookmarks page served by 
the bookmark sync server. Client-side, the subscription would track all 
links instead of dated links, and ensure that all links on the bookmarks 
page have copies in the local set of bookmarks. (And somehow get rid of 
deleted bookmarks, too.)

This way,  a public bookmarks feed could be served as a static file by any Gemini server.

It's a good idea that a client cert is used to ensure privacy of the 
served bookmarks, unless they are meant to be public.

The upload part is where it's gets tricky, and I don't think the Gemini 
protocol should be used for that. As a client author what I'd like to see 
is that I just export the bookmarks .gmi file as usual, and then send it 
to the server, perhaps via a TLS socket at a port specifically reserved 
for uploads ? just open a socket, set a client cert, and write the .gmi 
file to the socket. The client cert would be required for user 
identification. It would be up to the server to merge the bookmarks and 
get rid of deleted ones on server-side.

This does make things a bit more difficult for the server implementation, 
but keeping things easy for clients at least encourages wider support.

> * Does your client have some kind of plugin system which would allow a 
third-party to add support?

Not currently. However, Lagrange's MIME hooks could be extended to allow 
manual activation, so you could run an external program on your 
"about:bookmarks" page to do the sync. It would need to edit the internal 
bookmarks.txt file, and Lagrange would then need to detect that the file 
has changed after the hook program finishes. I wouldn't prefer going down 
this route, though. It's better to have a mechanism that benefits all clients equally.

So let's keep working on this! Automatic bookmark (and client cert?) sync 
would be great features for Gemini clients.

--jaakko

"Adnan Maolood" <me at adnano.co> writes:

> On Tue Dec 15, 2020 at 4:32 PM EST,  wrote:
>> Now, I don't want to hijack your thread... but if anyone is interested in a common
>> format for Gemini bookmarks as I mentioned earlier, an Amfora user proposed[1]
>> XBEL[2] to me, and I plan on implementing it in the next-next release, replacing the
>> TOML format being currently used, which was a bad choice for that purpose. Is
>> this something other client authors would like to do as well?
>
> What about using a plain Gemini text file for storing bookmarks? That
> would make it very easy for people to manage their bookmarks externally.
> You could implement folders as level 1 or 2 headings in the file.

This makes sense. It would also let you delegate the syncing to
something more appropriate, like NextCloud, syncthing, unison, or simply
rsync. 

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

Le mercredi 16 d?cembre 2020, 07:57:49 CET skyjake a ?crit :
> I think this is a valid use case, and it's something I have been 
thinking about myself just running Lagrange on a number of different 
computers. When you bring in a mix of different clients and desktop/mobile 
operating systems, sync is even more important.
> 
> The lowest common denominator should be export to/import from a .gmi 
file. Such a file (that Adnan Maolood suggested, for example) is 
guaranteed to be at least viewable by all clients. It supports folders, 
has no line length limitations, and can support tags as well (e.g., an 
optional line/bullet/quote following a link).  The user can then manually 
manage the files via any means they see fit.

I did not put folders and tag in the specification to start with something 
as simple as possible, but it could of course be added, to the gmi format at least.

> However, the UX of doing that on an ongoing basis to sync your 
bookmarks/feeds isn't great. An automated mechanism is certainly more desirable.
> 
> When it comes to basing sync on the Gemini protocol, it makes a certain 
amount of sense (clients know how to do Gemini requests) but it's not a perfect fit:
> 
> * one add/del per request ? what if you have a 100 bookmarks to sync?

This seems like an uncommon case, most likely you will sync 100 bookmarks 
from server to client, not the other way. It does not bother me if the 
initialization of the system is a bit under-optimized if after this it?s 
okay for everyday use.

> * request URL length limit

Yeah that may be a problem, especially if we start adding tags and such. 
It?s not clear how big a limit that would be.

> * does not address tags, folders, manual ordering

I did not think of ordering, indeed it may be needed.

> > * Would you consider implementing support for something like this?
> 
> Something like this, yes.
> 
> The current proposal needs revising, though. I suggest creating a 
mechanism that allows using Gemini subscriptions as a core part of the 
sync. A client would view and/or subscribe to a bookmarks page served by 
the bookmark sync server. Client-side, the subscription would track all 
links instead of dated links, and ensure that all links on the bookmarks 
page have copies in the local set of bookmarks. (And somehow get rid of 
deleted bookmarks, too.)
> 
> This way,  a public bookmarks feed could be served as a static file by 
any Gemini server.

This is intended by the spec and is why the only MUST is the text/gemini link list.

> It's a good idea that a client cert is used to ensure privacy of the 
served bookmarks, unless they are meant to be public.
> 
> The upload part is where it's gets tricky, and I don't think the Gemini 
protocol should be used for that. As a client author what I'd like to see 
is that I just export the bookmarks .gmi file as usual, and then send it 
to the server, perhaps via a TLS socket at a port specifically reserved 
for uploads ? just open a socket, set a client cert, and write the .gmi 
file to the socket. The client cert would be required for user 
identification. It would be up to the server to merge the bookmarks and 
get rid of deleted ones on server-side.
> 
> This does make things a bit more difficult for the server 
implementation, but keeping things easy for clients at least encourages wider support.

I thought it was easier for clients if they can reuse their Gemini 
handling code instead of adding support for another protocol. Same thing 
for server actually.

> > * Does your client have some kind of plugin system which would allow a 
third-party to add support?
> 
> Not currently. However, Lagrange's MIME hooks could be extended to allow 
manual activation, so you could run an external program on your 
"about:bookmarks" page to do the sync. It would need to edit the internal 
bookmarks.txt file, and Lagrange would then need to detect that the file 
has changed after the hook program finishes. I wouldn't prefer going down 
this route, though. It's better to have a mechanism that benefits all clients equally.
> 
> So let's keep working on this! Automatic bookmark (and client cert?) 
sync would be great features for Gemini clients.

If I?m not mistaking, lagrange has no support for folders, ordering and 
tagging bookmarks/feeds?
This is also why I did not put such things in the spec, to start with the 
minimal common denominator among clients features.

C?me

On Tuesday, December 15, 2020 5:42 PM, Adnan Maolood <me at adnano.co> wrote:

> [snip]
>
> What about using a plain Gemini text file for storing bookmarks? That
> would make it very easy for people to manage their bookmarks externally.
> You could implement folders as level 1 or 2 headings in the file.
>
> For example:
>
> # Folder 1
>
> => gemini://gemini.circumlunar.space Project Gemini
> => gemini://example.com Example Bookmark
>
> # Folder 2
>
> => gemini://example.org Example Bookmark

I agree that people being able to manage their own bookmarks easily in
a text editor is attractive. However I think I'd rather stick with a more
strict format rather than one that's endlessly extensible like this one.

Maybe one browser will use text lines as comments, and another will use
them as descriptions, and Amfora will ignore all text lines, but then one
day someone will want Amfora to support descriptions, but it can't tell
what's what, etc.

XML and the XBEL spec prevents these issues at the expense of user writing.
I'm not sure how many users really are going to be writing their own
bookmarks though, and of those, how many are going to have trouble with XML.

Often re-using gemtext is useful because of rendering, but in this case
the gemtext would have to be interpreted and then rendered in a different
format, which seems counter-intuitive. I think gemtext works best as a
markup format as it was designed, rather than for data storage.


Cheers,
makeworld

On 16. Dec 2020, at 20:26, C?me Chilliet <come at chilliet.eu> wrote:

> If I?m not mistaking, lagrange has no support for folders, ordering and 
tagging bookmarks/feeds?

Lagrange lets you enter tags for bookmarks/feeds as a free-format string 
of text. In fact, a feed is a tagged bookmark. The quick search feature 
looks through bookmark tags.

Folders and ordering are on my to-do list. I'm getting a bit annoyed using 
the simple alphabetically ordered flat list of bookmarks. :)

--jaakko

On 17. Dec 2020, at 0:41, colecmac at protonmail.com wrote:

> I agree that people being able to manage their own bookmarks easily in
> a text editor is attractive. However I think I'd rather stick with a more
> strict format rather than one that's endlessly extensible like this one.

I think the primary advantage of this bookmarks .gmi format would be 
interchange between clients (and instances of the same client) without 
humans editing the file. I don't think a client would want to store their 
bookmarks in this format, as there may be additional (possible internal) 
metadata to deal with, too.

When it comes to manual editing, I could see reordering and tagging your 
existing bookmarks as real-world use cases, and a simple line-based format 
makes these quite straightforward. But in any case, I'd probably prefer 
doing this via the client GUI myself.

> Maybe one browser will use text lines as comments, and another will use
> them as descriptions, and Amfora will ignore all text lines, but then one
> day someone will want Amfora to support descriptions, but it can't tell
> what's what, etc.

A simple companion spec for client bookmark exchange similar to the Gemini 
subscriptions one should solve this problem adequately by defining what is 
meaningful and what should be ignored. Another format like XBEL could 
still be used if one's client supports additional information per bookmark.

> XML and the XBEL spec prevents these issues at the expense of user writing.
> I'm not sure how many users really are going to be writing their own
> bookmarks though, and of those, how many are going to have trouble with XML.

XBEL is certainly an existing solution to exchanging bookmarks, and but it 
would be nice to have something that is compatible with the Gemini ethos. 
Personally I would even prefer JSON-based formats to XML (easier to 
parser, easier for humans to read), but gemtext has the unique advantage 
of being viewable by all clients even if they have no concept of importing 
bookmarks, or bookmarks in general.

--jaakko

If you want to synchronize bookmarks over the Internet for all clients, 
this feature is already built into Gemini.

Basically all you need to do is create a gemdoc with links to the things 
you would like to bookmark/sync, then place that file on a Gemini server 
somewhere. You can then direct any client to open this file, and then 
you've got your bookmarks.

Also you can update the file any time and it will automatically 
propagate across all clients.

Ben

-- 
gemini://kwiecien.us/

On Thu, 17 Dec 2020 10:37:37 +0330, Ben wrote:

> If you want to synchronize bookmarks over the Internet for all clients,
> this feature is already built into Gemini.
> 
> Basically all you need to do is create a gemdoc with links to the things
> you would like to bookmark/sync, then place that file on a Gemini server
> somewhere. You can then direct any client to open this file, and then
> you've got your bookmarks.
> 
> Also you can update the file any time and it will automatically
> propagate across all clients.
> 
> Ben

Brilliant! (No need for JSON, TOML, XML (XBEL, OPML) ? I guess)

> I think the primary advantage of this bookmarks .gmi format would be interchange
> between clients (and instances of the same client) without humans editing the file.
> I don't think a client would want to store their bookmarks in this format, as there
> may be additional (possible internal) metadata to deal with, too.

Well, that works for me. Amfora already has support for file:// URIs in master, so
in the next release you'll be able to just open any gemtext file and use it as your
free-form bookmarks.


makeworld

On 18. Dec 2020, at 0:31, colecmac at protonmail.com wrote:

>> I think the primary advantage of this bookmarks .gmi format would be interchange
>> between clients (and instances of the same client) without humans editing the file.
>> I don't think a client would want to store their bookmarks in this format, as there
>> may be additional (possible internal) metadata to deal with, too.
> 
> Well, that works for me. Amfora already has support for file:// URIs in master, so
> in the next release you'll be able to just open any gemtext file and use it as your
> free-form bookmarks.

Great! I've also added support for using bookmark .gmi files/pages in Lagrange (v1.0).


On 15. Dec 2020, at 23:08, C?me Chilliet <come at chilliet.eu> wrote:

> I?ve put together an idea of simple specification to allow clients to 
synchronize bookmarks and feeds across Gemini, as this is something that 
I?m currently missing in my daily Gemini use.
> * Would you consider implementing support for something like this?

I'm changing my previous answer now with Lagrange v1.0 having support for 
remote bookmarks. It solves 80% of the problem, especially if you throw in 
Dropbox/etc. to sync your bookmark files across devices, or get them via 
your existing Gemini server.

So what remains is automated upload of added/deleted bookmarks, and for 
that it seems setting up a custom server might be overkill. The UX 
benefits of automation are still there, but maybe there is something even 
simpler that could do the trick.

I'll give it some thought.

--jaakko