Subject: Re: Protocol design: the Gemini project

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 1 Dec 2020 11:40:42 -0500

On Mon, Nov 30, 2020 at 1:37 PM Joseph Touch <[email-masked]> wrote:

> My primary criticism:
>
> - port 1965 is already assigned
> - new system ports are advised against, per RFC7605
>
> I already let them know.
>
> As to whether the rest is useful, I sincerely hope they’ve studied
> previous protocols they think are too complex. The protocol design space is
> littered with examples of “I think I can do better by doing simpler”, only
> to end up reinventing the wheel (the most infamous example being HTTP vs
> using FTP).
>

Having (re)implemented FTP for LIBWWW, I can assure you that it was done
for very good reason.

FTP is not really an independent protocol. It is actually designed as a
feature add on for Telnet. As a result, FTP is vastly less efficient than
HTTP because you have two socket creations and teardowns per connection.

FTP requires multiple control plane interactions where HTTP requires only
one.

FTP has no provision for transfer of metadata.

FTP doesn't support proxying or caching, both of which were essential in
the early Web when bandwidth was severely constrained.

Of course we could have stuck to vinyl. But FTP is more like grandad's 78s.
There is absolutely nothing to recommend FTP over HTTP. Rsync is vastly
superior for file transfer.

Sometimes the best answer is to simply chuck out the old and start from a
clean slate. Software reuse is good but only to a point and only if the
software in question is fit for reuse. FTP is not and it is high time we
moved beyond it.

The same is becoming true of SMTP. Yes, we can try to continue to prop up a
protocol that has been spammed to death. But the simplest way to get to the
heart of the matter is to switch to a new protocol that has authentication
and end to end encryption built in from the start.

In fact the only good reason to not start from scratch is the difficulty of
deployment. HTTP succeeded so there was really no reason not to switch.

There is absolutely no part of the FTP design I am sorry we left out. None.

---

From: Keith Moore <[email-masked]>
Date: Tue, 1 Dec 2020 12:40:14 -0500

On 12/1/20 11:40 AM, Phillip Hallam-Baker wrote:

> Of course we could have stuck to vinyl. But FTP is more like grandad's 
> 78s. There is absolutely nothing to recommend FTP over HTTP. Rsync is 
> vastly superior for file transfer.

False, on multiple levels.

The biggest flaw of HTTP for file transfer (if one wants to transfer 
more than one file) is that HTTP doesn't have a built-in way to list 
files, distinguish files from directories from other kinds of nodes, and 
walk a file system.
Rsync, certainly at the time that HTTP was designed (it may have 
improved since then) had a LOT of overhead because it tried to analyze 
each file for changes within the file, minimizing bandwidth used (which 
to be fair, was quite scarce) at a cost of CPU time and latency.  Circa 
1993 I looked at using rsync to replicate a web site to multiple 
locations (early CDN I suppose) and found it completely inadequate.

Of course FTP was designed for file transfer, including between 
dissimilar systems (which were very common in ARPAnet days), and HTTP 
wasn't designed for that purpose.  There was nothing wrong with 
designing a new protocol for the web especially since the web had 
different needs, different assumptions, and operated under different 
conditions.

But the web has NEVER been a good way to do file transfer. Wasn't in 
1991, and isn't today.   And the protocol designed for the web isn't 
either without adding some additional features.

Keith

---

From: Joseph Touch <[email-masked]>
Date: Tue, 1 Dec 2020 10:27:06 -0800

Speaking of false...

> On Dec 1, 2020, at 8:40 AM, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> ...
> Having (re)implemented FTP for LIBWWW, I can assure you that it was done for very good reason.
> 
> FTP is not really an independent protocol. It is actually designed as a feature add on for Telnet. As a result, FTP is vastly less efficient than HTTP because you have two socket creations and teardowns per connection. 

FTP needs only N+1 connections per set of N transfers from a given site.

You should never need to close and reopen the control connection unless you leave the site.

> FTP requires multiple control plane interactions where HTTP requires only one.

You can specify an entire transfer with one command.

> FTP has no provision for transfer of metadata.

FTP has both records and data structures.

> FTP doesn't support proxying or caching, both of which were essential in the early Web when bandwidth was severely constrained.

FTP has third party transfers, which are a much more powerful mechanism than simply proxying. It can cache just like HTTP can (that’s an implementation issue) - and can prefetch too:
Touch, J; Farber, D: An Experiment in Latency Reduction <>. In: Infocom, pp. 175-181, IEEE, 1994.

Joe

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 1 Dec 2020 19:25:46 -0500

On Tue, Dec 1, 2020 at 12:40 PM Keith Moore <[email-masked]>
wrote:

> The biggest flaw of HTTP for file transfer (if one wants to transfer more
> than one file) is that HTTP doesn't have a built-in way to list files,
> distinguish files from directories from other kinds of nodes, and walk a
> file system.
> Rsync, certainly at the time that HTTP was designed (it may have improved
> since then) had a LOT of overhead because it tried to analyze each file for
> changes within the file, minimizing bandwidth used (which to be fair, was
> quite scarce) at a cost of CPU time and latency.  Circa 1993 I looked at
> using rsync to replicate a web site to multiple locations (early CDN I
> suppose) and found it completely inadequate.
>
> Of course FTP was designed for file transfer, including between dissimilar
> systems (which were very common in ARPAnet days), and HTTP wasn't designed
> for that purpose.  There was nothing wrong with designing a new protocol
> for the web especially since the web had different needs, different
> assumptions, and operated under different conditions.
>
> 1991, and isn't today.   And the protocol designed for the web isn't either
> without adding some additional features.
>
The original attack made was that HTTP was unnecessary because FTP existed.
So the fact that FTP can do other things is utterly irrelevant to what I
was responding to. We wrote a new protocol because FTP is a really shitty
hypertext transfer protocol. That wasn't an arrogant or ignorant approach,
libwww always had an FTP client in it.

> Speaking of false...

Of course I have the benefit of knowing what I am talking about having
actually tried to make FTP work. And as of 1993 it simply didn't as you
would know had you tried it.

On Dec 1, 2020, at 8:40 AM, Phillip Hallam-Baker <[email-masked]>
> wrote:

Not in 1993 it didn't. I took a look at the practicality of caching the
connections for multiple transfers from the same site and it plain didn't
work because of all sorts of technical issues such as socket exhaustion and
the limited capabilities of the FTP servers of the era. At the time HTTP
was written, HTML didn't support any form of transclusion so it was one
transfer per page.

The problem with caching the connection is that the FTP servers of the era
were typically limited to 8 simultaneous users. So if browsers kept the
control connection open, they would prevent other users connecting. Sure
you could rewrite the FTP server but why do that when you can write a new
server with less effort?

Oh and that is before we start to look at CGI etc.

The gap between the theory as you understand it and implementation at the
time was huge.

I repeat, we understood what we were doing and we had good reasons for
doing what we did.

Reuse is good but attempting to reuse something that is not suitable for
purpose is bad. HTTP is a very effective and reasonably efficient hypertext
transfer protocol. It is also a pretty decent presentation layer, hence Web
Services. HTTP/3 over QUIC is a very effective and very efficient hypertext
transfer protocol. But it isn't as good at being a presentation layer as
HTTP/1.1.

And that is OK

We don't need to use the same tool for every task. I did argue early on in
HTTP/2 for some features that would make it better as a presentation layer
but stopped when I realized what I wanted was HTTP/1.1 with most of the
features stripped out. And that is better achieved as a separate protocol.

---

From: Joe Touch <[email-masked]>
Date: Tue, 1 Dec 2020 17:40:34 -0800

> On Dec 1, 2020, at 4:26 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> The gap between the theory as you understand it and implementation at the time was huge.

The gap between your implementation and our measured experiments on commercial versions appears to be bigger. 

We saw connection limits at ftp.isi.edu but that was for socket exhaustion per service per client, in the thousands per 2 minute time wait.  Not 8 - and we never override the defaults. 

Joe

---

From: Larry Masinter <[email-masked]>
Date: Tue, 1 Dec 2020 17:42:31 -0800

If we're going to talk history, HTTP was never "designed for" file
transfer. HTTP/0.9 was designed for a Gopher replacement; and Gopher was
designed for information access. HTTP/0.9 followed the Gopher model: open a
TCP session, send a request, read the results until the connection was
closed. Gopher replies could be menus or any of a handful of other types
(not really extensible). Dan Connolly gets credited with promoting the idea
of using MIME to get an extensible information set. Early on, HTTP acquired
content negotiation, with the addition of an "Accept:" header where the
client would inform the server of its preferences for MIME types and
parameters. I thought the idea came from work at PARC on "System 33", in
which the server would do format conversion to meet the needs of the client
(including OCR of image files or generating images from text). In general,
HTTP has retained the distinction of the "resource" vs. its
"representation".  A "file" is a resource.
HTTP doesn't send resources, it sends representations of resources, which
might vary depending on the Accept header or others (user-agent being
common).

I think with a "file transfer" you might expect to retrieve enough data in
response to a request that the receiver might be able to replicate the
original resource and all of its representations.

Accept: text/rfc;pagenumbers=yes,text/html,application/pdf
--
https://LarryMasinter.net <- chair of the room where it happened