Subject: Re: Last Call: <draft-cardenas-dff-09.txt> (Depth-First Forwarding in Unreliable Networks (DFF)) to Experimental RFC

From: Thomas Heide Clausen <[email-masked]>
Date: Sat, 9 Feb 2013 14:48:48 +0100

Hello folks,

I'd humbly like to encourage publication of this document.

I have been aware of DFF for a while, and we have implemented & tested the mechanisms specified therein fairly exhaustively. 

	1)	Testing DFF conjointly with a variety of (IETF and non-IETF) routing, introducing
		DFF yielded - in our tests - moderate to impressive performance improvements.

	2)	We did not see any negative performance impacts, in our tests, from the use of DFF.
	
	3)	During our implementation & tests, I've been  providing occasional feed-back & reviews
		to the authors (clarifications, mainly), which has been well reflected by the authors 
		(thanks!)

	4) 	Having reviewed this -09 version of the specification, I find it to be very clearly and 
		concisely written, and sufficiently detailed to permit straight-forward implementation.
		(Kudos!).

	5)	A couple of nits remain; I'll ad those to the end of this email. They are exclusively
		of editorial nature, so please do not hold up the document on behest of those.

Concluding the above, I encourage that this document be published as RFC. It seems to be a very good idea, which is well documented. 

I understand the motivation for this document being published on the Experimental track is, that we need more operational data. I also understand from Ralph's emails to various WGs that he is AD sponsoring this document, since it doesn't seem to fit anywhere, at the moment, in the IETF.

If I may be so bold, I would like to suggest that a path be identified for if and when progressing towards std.track - it was coincidental that I came across the document (and I'm glad that I did).

Now, my few *editorial* nits from reviewing -09:

	Notation and Terminology:

		There is some inconsistency in the use of colon's, dash'es or spaces after the term and
		before its definition, for example:

			Foo:  The definition of Foo
	
			Bar -  The definition of Bar

			Foobar This specification uses Foobar

	Information Base Overview:

		Suggest the first sentence be something of the form:

		"The information base required on each router contains a single 
			set, the Processed Set"

		Or something, since otherwise the term "Information Base" seems orphaned.

	Also, the document later (section 6) calls it "Information Sets". Suggest that section 6
	be "Information Base" instead (or otherwise rendering the terminology consistent 
	there)

	Section 8, would it be worth calling out if these parameters can be varied, and need
	not be the same on all routers in a given network? I.e., if heterogeneous parameters 
	across a network does not impede on interoperability?

	Section 16: 
		"implies that any headers specific to DFF do not traverse the boundaries
		of the routing domain"
		
		Should that not be "...specific to DFF MUST NOT traverse the ..." to be prescriptive?

That's all!

Best,

Thomas

Sent from my iPad

On 7 févr. 2013, at 14:21, The IESG <[email-masked]> wrote:

> 
> The IESG has received a request from an individual submitter to consider
> the following document:
> - 'Depth-First Forwarding in Unreliable Networks (DFF)'
>  <draft-cardenas-dff-09.txt> as Experimental RFC
> 
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action. Please send substantive comments to the
> [email-masked] mailing lists by 2013-02-24. Exceptionally, comments may be
> sent to [email-masked] instead. In either case, please retain the
> beginning of the Subject line to allow automated sorting.
> 
> Abstract
> 
> 
>   This document specifies the "Depth-First Forwarding" (DFF) protocol
>   for IPv6 networks, a data forwarding mechanism that can increase
>   reliability of data delivery in networks with dynamic topology and/or
>   lossy links.  The protocol operates entirely on the forwarding plane,
>   but may interact with the routing plane.  DFF forwards data packets
>   using a mechanism similar to a "depth-first search" for the
>   destination of a packet.  The routing plane may be informed of
>   failures to deliver a packet or loops.  This document specifies the
>   DFF mechanism both for IPv6 networks (as specified in RFC2460) and in
>   addition also for LoWPAN "mesh-under" networks (as specified in
>   RFC4944).
> 
> 
> 
> 
> The file can be obtained via
> http://datatracker.ietf.org/doc/draft-cardenas-dff/
> 
> IESG discussion can be tracked via
> http://datatracker.ietf.org/doc/draft-cardenas-dff/ballot/
> 
> 
> The following IPR Declarations may be related to this I-D:
> 
>   http://datatracker.ietf.org/ipr/1645/
>   http://datatracker.ietf.org/ipr/1646/
> 
> 
>

---

From: Abdussalam Baryun <[email-masked]>
Date: Sun, 10 Feb 2013 12:08:04 +0100

Reply to your request dated 07/02/2013
Also following AD advice.
Draft Reviewed By: Abdussalam Baryun (AB)   Dated: 10/02/2013

Reviewer Comment #AB1: Related to Aim and Terminology.
++++++++++++++++++++++++++++++++++++

Overall I support the work, but subject to amendments.

The Abstract is interesting but seem general

>    for IPv6 networks,

AB> do you mean all IPv6 networks, if some please mention *some*

>    a data forwarding mechanism that can increase
>    lossy links.

AB> IPv6 considers dynamic topology, does the paragraph mean that some
networks are static topology. Suggest to clarify the dynamic
frequency. Also not clear what is missing with IPv6 protocol in such
nets that needs DFF.

>  The protocol operates entirely on the forwarding plane,
>    but may interact with the routing plane.

AB> amend> entirely on the forwarding plane,
           to; entirely within the forwarding plane,

>    The routing plane may be informed of
>    failures to deliver a packet or loops.

AB> Amend> routing plane may be informed
          to;   routing plane protocols may be informed

Terminology section>

AB> you need to define that all routers have the DFF protocol, even
though it is known by heart that any forwarder needs to run DFF but
you MUST mention that at least to the reader,

AB> you missed to define the Router, which is mentioned many times but
I don't know which one. I think it is a must because you refer to two
planes forwarding and routing planes. Please define the Forwarder as
well. I don't find any Forwarder action in the draft, why not while
you specify there is a forwarding plane!

AB> suggest that you explaine forwarding and routing in separate and
show where they join relating to your protocol.

This will not be the last, will continue,

Regards
AB
---------------------------------------------------------------------------------------
This message and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
This message is in compliance with the IETF regulations.
---------------------------------------------------------------------------------------

---

From: Ulrich Herberg <[email-masked]>
Date: Tue, 12 Feb 2013 13:59:01 -0800

AB,

thank you for the review. See my answer below:

> Reviewer Comment #AB1: Related to Aim and Terminology.

Do you have any specific suggestions what you would like to add/change
in the abstract?

> AB> do you mean all IPv6 networks, if some please mention *some*

I don't think this is relevant to add in the abstract. As stated in
section 3 of the draft, it can be used for all IPv6 networks, but for
some the protocol is not very beneficial (and examples are given for
such situations).

> networks are static topology.

I am not quite sure what you mean by IPv6 "considers topology". In a
static topology (i.e. with no or very few changes of links) and with
low packet loss rate, the DFF protocol would have little (or no)
benefit, because most likely the packets would be sent from the source
to the destination according to the information provided by the
routing protocol, and no packets would be lost.

> Suggest to clarify the dynamic
> frequency.

I am not sure what you would like to see clarified. Can you be more specific?

> Also not clear what is missing with IPv6 protocol in such
> nets that needs DFF.

If you mean data forwarding, I think the draft is pretty clear about
which cases it is beneficial in. The motivation for DFF is explained
in section 1.1.

>            to; entirely within the forwarding plane,

I am not a native English speaker, but "to operate on" seems a valid
usage of the word. But maybe a native speaker could suggest which is
the right usage.

>           to;   routing plane protocols may be informed

I think the current sentence is sufficient; I don't see the benefit of
adding the word "protocols".

> you MUST mention that at least to the reader,

I think this is actually a good point. I was wondering whether to
enforce all routers in a routing domain to run DFF. That does not seem
like a reasonable approach. I found only one case that could be a
problem when using both DFF and non-DFF routers in the network that
results in a loop, but I think I found an easy fix. Here is the
problem illustrated using an example:

...--- A ----- B ---- C --- ...

Assume routers A and C use DFF, but not B. When B receives a packet
from A, it would ignore the DFF header and skip over it, and then
forward the packet to C (assuming C is the next hop in its RIB). Now,
it is possible that C returns  the packet (RET==1) to B, e.g. because
it does not have any other neighbor. B receives the returned packet
and would resend the packet to C (because it does not care about the
DFF header), and thus RET would still be 1. C, having tried already
all neighbors would return the packet to B (and so on, loop between B
and C until hop-limit == 0).

I see the following options:
1) Change the options header type from something starting with 001
    (i.e. "skip over this option and continue processing the header.") to
    011 ("discard the packet.") in IPv6 to discard the packet if the
    header is unknown. This would avoid the problem, but I don't think
    that's possible for 6lowpan mesh-under.
2) Mandate that all routers in the routing domain "MUST use DFF" for
    forwarding packets.
3) Add a rule that if a packet is returned (RET==1) from P_prev_hop
    (i.e. the original previous hop when first received a packet), it is
    discarded, e.g. if router C receives a packet from B with RET==1 and B
    is the P_prev_hop.

Option 1) would be possible, but not for 6lowpans, only for IPv6
route-over. Option 2) is probably not very good. Option 3) would fix
the loop issue, and thus allows non-DFF and DFF routers in the same
network. I will provide some updated text in the next revision of the
draft.

> you specify there is a forwarding plane!

I am undecided on defining the term "router". It is a well-known term
in IETF; but then, maybe one could define it like in RFC6130 ("Router
- A router running this protocol"). Not sure this conveys a lot of
information, though. So unless others also suggest adding it, I will
not change the terminology section.
Since I don't use the term "forwarder" (which is usually part of the
functions of a router), I don't think I should define that in the
terminology.

> show where they join relating to your protocol.

I think the concept of forwarding and routing plane is fairly well
understood in the IETF, so I don't think I should re-explain that in
the draft.

Best regards
Ulrich

---

From: Abdussalam Baryun <[email-masked]>
Date: Mon, 25 Feb 2013 06:32:51 +0100

Reply to your request dated 07/02/2013
Draft Reviewed By: Abdussalam Baryun (AB)   Dated: 24/02/2013

Reviewer Comment #AB2: Related to Applicability and Processing.
++++++++++++++++++++++++++++++++++++

The DFF mechanism specified in the document can either be used as
"route-over" IPv6 forwarding mechanism (Mode of Operation:
"route-over"), or as "mesh-under"
LoWPAN adaptation layer (Mode of Operation: "mesh-under").

AB> the document should specify that these two modes must not occur in
the same time.

Section 3. Applicability
AB> is general, not sure what is the use-case for the protocol,

AB> does this protocol only for neighbor forwarding and exclude IP
tunnling, then specify.

Section 5.
AB> mentioned RFC6130 this is for MANET so does this protocol apply
for MANETs, not mentioned in section 3. Do you mean that LowPAN is
similar to MANET, not sure how this protocol works in all environment?

AB> how does this protocol forward without considering the IP
forwarding? Should IP forward stop its function (not clear)?

AB> DFF has access on <list of bidirectional neighbors> what kind of
access you mean?

Section 9.1
AB> how can DFF become a routing domain? it is not a routing protocol?
Does this DFF work with ALL routing protocols in the IETF, please
specify either ALL or which?

Regards
AB

---

From: Abdussalam Baryun <[email-masked]>
Date: Mon, 25 Feb 2013 06:59:33 +0100

Reviewer Comment #AB3: Related to Processing and interaction with others.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++

In section 11
AB>The DFF MUST contain the next hop of RIB, but in section 5 it mentions MAY.
AB>suggest> please amend both should be same requirement for protocol.

AB> the way it determines the next hop does not show that this
protocol is sensitive to other parameters in the RIB, it just takes
the addresses without considering the routing protocol strategy in its
RIB. IMHO, this may make the DFF make mistakes in taking the right
next hop,

Section 12
AB> is not understood, it seems a general not specific, I suggest more
explaining how this interaction with routing protocol is occured?

Section 13
AB> who creates the sequence number is it the DFF or the routing
protocol. It seems the DFF, so please specify how it will
maintain/save such sequence number in this section.
AB> suggest this section to be : DFF Sequence Number.

Section 14
AB> again do you mean that both modes can work together. I don't think
that you mean that, so you should specify that each routing domain
MUST have one mode, and specify how to maintain that mode in such
protocol.

Overall> about processing>
AB> this protocol needs to be discussed more how it will interact with
the routing protocols in MANET, 6LowPAN, ROLL. The documents ignore
alot of work done in the IETF which is not fiting the general
applicability it is offering.

+++++++++++++The END+++++++++++

Regards
AB

---

From: Thomas Heide Clausen <[email-masked]>
Date: Tue, 26 Feb 2013 00:01:07 +0100

Note, I am not an author of this particular document, but I've got some experimental and operational experience with it - having implemented DFF and built deployments including it.

On 25 févr. 2013, at 06:59, Abdussalam Baryun <[email-masked]> wrote:

> next hop

What you say above doesn't make any sense. DFF considers the content of the RIB, among other information, and recommends following the "next hop" indicated in the RIB first - i.e., DFF explicitly takes the "right next hop" according to "the routing protocol strategy" as expressed in the RIB.

Of course, if you happen to know more about what the routing protocol does (e.g., have access to internal state of the routing protocol), you could do something more clever - DFF doesn't prohibit that either, so I think that the authors struck a nice middle ground here.

> explaining how this interaction with routing protocol is occured?

It seems pretty clear to me what's expected here: signal to the routing protocol (and then, whatever the routing protocol does with that signal depends on what routing protocol it is).

It's, IMO, hard to be any clearer here.

> AB> suggest this section to be : DFF Sequence Number.

[befuddled] 

Section 13 is already very clear on this.

> MUST have one mode, and specify how to maintain that mode in such

Eh, here too I am befuddled. I do not see anywhere that the authors indicate that the two "modes can work together" - actually, the document doesn't talk about "modes" at all, and  the protocol doesn't have "modes".

Rather, this section describes how one would use DFF, depending on what other choices are made in the network deployment. In other words: DFF doesn't specify a "mode", but rather says "if your network is of this kind, then do like that".

So, I think that there's strictly nothing to change here, no need to maintain a "mode" etc.

> applicability it is offering.

No, it doesn't.

DFF is completely orthogonal to routing protocols

DFF can (logically) layer below any routing protocol, without modifications to that routing protocol. 

DFF can use a RIB and can provide signals to a routing protocol (or not - if the routing protocol doesn't want them)

Thomas

---

From: Abdussalam Baryun <[email-masked]>
Date: Tue, 26 Feb 2013 11:29:48 +0000

Could you give your summary experience of implementing DFF and which
network was it deployed/tested in and the summary result? I think this
information will help other users and also the IESG to make future
decisions :-)

AB

On Mon, Feb 25, 2013 at 11:01 PM, Thomas Heide Clausen <
[email-masked]> wrote:

> Note, I am not an author of this particular document, but I've got some
> experimental and operational experience with it - having implemented DFF
> and built deployments including it.
>
> What you say above doesn't make any sense. DFF considers the content of
> the RIB, among other information, and recommends following the "next hop"
> indicated in the RIB first - i.e., DFF explicitly takes the "right next
> hop" according to "the routing protocol strategy" as expressed in the RIB.
>
> Of course, if you happen to know more about what the routing protocol does
> (e.g., have access to internal state of the routing protocol), you could do
> something more clever - DFF doesn't prohibit that either, so I think that
> the authors struck a nice middle ground here.
>
> > Section 12
> It seems pretty clear to me what's expected here: signal to the routing
> protocol (and then, whatever the routing protocol does with that signal
> depends on what routing protocol it is).
>
> Eh, here too I am befuddled. I do not see anywhere that the authors
> indicate that the two "modes can work together" - actually, the document
> doesn't talk about "modes" at all, and  the protocol doesn't have "modes".
>
> Rather, this section describes how one would use DFF, depending on what
> other choices are made in the network deployment. In other words: DFF
> doesn't specify a "mode", but rather says "if your network is of this kind,
> then do like that".
>
> DFF can use a RIB and can provide signals to a routing protocol (or not -
> if the routing protocol doesn't want them)
>
> Thomas
>
> >>>> [email-masked] mailing lists by 2013-02-24. Exceptionally, comments
> may
> >>>> be

---

From: Thomas Heide Clausen <[email-masked]>
Date: Tue, 26 Feb 2013 12:51:35 +0100

On 26 févr. 2013, at 12:29, Abdussalam Baryun <[email-masked]> wrote:

> Could you give your summary experience of implementing DFF and which network was it deployed/tested in and the summary result? I think this information will help other users and also the IESG to make future decisions :-)

Well, it was about a day worth of implementation effort. Testing in the lab with various ROLL and MANET routing protocols, and OSPF, was trivial: grab off-the-shelf implementation of the routing protocol, run as usual, marvel at the results ;)

It has been deployed outside of the lab. I cannot go too much into the details of those deployments, other than say that the experience matched that which we saw in the lab quite nicely.

We've tested over different L2's, including proprietary low-power wireless, WiFi, PLC, Ethernet...saw no ill effect and for all but Ethernet observed good performance improvements (we included Ethernet in our testing, where there were no effects at all, simply to have a baseline).

What's your experience with this protocol, AB? You seem to be very interested in it, do you have any experimental or operational data to share?

Thomas

>  AB
> 
> On Mon, Feb 25, 2013 at 11:01 PM, Thomas Heide Clausen <[email-masked]> wrote:

---

From: Abdussalam Baryun <[email-masked]>
Date: Tue, 26 Feb 2013 12:12:40 +0000

On Tue, Feb 26, 2013 at 11:51 AM, Thomas Heide Clausen <
[email-masked]> wrote:

> Well, it was about a day worth of implementation effort. Testing in the
> lab with various ROLL and MANET routing protocols, and OSPF, was trivial:
> grab off-the-shelf implementation of the routing protocol, run as usual,
> marvel at the results ;)
>
> It has been deployed outside of the lab. I cannot go too much into the
> details of those deployments, other than say that the experience matched
> that which we saw in the lab quite nicely.
>
> We've tested over different L2's, including proprietary low-power
> wireless, WiFi, PLC, Ethernet...saw no ill effect and for all but Ethernet
> observed good performance improvements (we included Ethernet in our
> testing, where there were no effects at all, simply to have a baseline).
>
> What's your experience with this protocol, AB? You seem to be very
> interested in it, do you have any experimental or operational data to share?
>
>
My experience with the protocol is reading experience only as it is an
individual work, and may go into implementation if I know future works
related. However, I thank you for your input because it is important,

AB