Subject: IETF processes (was Re: draft-housley-two-maturity-levels)

From: RJ Atkinson <[email-masked]>
Date: Thu, 28 Oct 2010 12:22:27 -0400

On Weds, 27th October 2010, at 13:56:25 -0700, Bob Braden wrote in part:
> In this environment, the only thing that seems to make sense 
> is for WGs to start usually at Experimental (someone else
> suggested this, I apologize for not recalling who it was).

Agreed.

Most times it would be better if IETF WGs initially create 
an Experimental status RFC, possibly doing so quite rapidly, 
and then later revise that (based on at experience) and 
publish the revision on the IETF standards-track.

Indeed, this is what the LISP WG appears to be doing.
It also is how HIP started out (initially an IRTF HIP RG,
then an IETF HIP WG with Experimental RFCs, now the
IETF HIP WG is working on standards-track RFCs).
I am trying to follow that model with ILNP, which the
IRTF Routing RG has offered to the IRSG for publication
on the IRTF track, as a set of Experimental RFCs of course.

On Weds, 27th October 2010, at 14:48:22 -0700, Bob Braden wrote in part:
> I note that there seems to be some correlation between
> the degradation of the IETF process and the disappearance
> of the Internet research community from the IETF
> (the US government decided that no further R&D funding
> was required, since the Internet was "done".)

Agreed.  

There seems to be strong correlation between those events.  

I believe that the USG (and other research funding bodies 
elsewhere -- to be geographically neutral) were confused 
when they erroneously thought either that the Internet 
was "done" or that "industry would do any research that
might be needed in future".

The IAB's RFC-3869 expressed this concern in 2004, providing
several concrete examples of areas that industry wasn't solving 
and that needed more research.

Yours,

Ran

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 28 Oct 2010 10:29:27 -0700

On 10/28/2010 9:22 AM, RJ Atkinson wrote:
> ost times it would be better if IETF WGs initially create

1. Getting /any/ RFC through the IETF process is very high overhead, including 
Experimental.

2. Why does what you've suggested not qualify for the IRTF rather than the IETF? 
  Shouldn't a standards process be able to sit down and do a standard, rather 
than iterate on experimental designs?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Bob Braden <[email-masked]>
Date: Thu, 28 Oct 2010 10:43:37 -0700

On 10/28/2010 10:29 AM, Dave CROCKER wrote:
>
>
>
> 1. Getting /any/ RFC through the IETF process is very high overhead,
> including Experimental.

Dave,

Excuse me, but just what do you mean by "very high overhead"?

Metrics? Examples? Comparisons with experience with other SDOs?

Bob Braden

---

From: Scott Brim <[email-masked]>
Date: Thu, 28 Oct 2010 13:55:14 -0400

> publish the revision on the IETF standards-track.

Given how many WGs these days in fact seem to start from "let's develop
something and see if we can make it useful", this wouldn't be too bad.

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 28 Oct 2010 10:56:27 -0700

>> 1. Getting /any/ RFC through the IETF process is very high overhead,
...
> Excuse me, but just what do you mean by "very high overhead"?

Quite a lot of work, with unpredictable and unbounded delay, for some of the steps.

> Metrics? Examples? Comparisons with experience with other SDOs?

Comparing with other SDOs is irrelevant to this thread.

As a metric, once a working group has decided to issue a new Internet Draft, it 
takes almost no time to issue it.  Assuming no format hiccups, it's minutes.

In the IETF, to get that same document issued as Experimental requires the full 
IETF approval process with reviews, Last Call, AD support, IESG discussion 
cycles, IESG approval (with the variability of resolving Discusses), and RFC 
editing and approval.

At its very best, this is perhaps a couple of months, with extremely aggressive 
management support.  But it never is at its best.  More typical is 3-6 months.

The only implied criticism in my list is the variability of IESG approval.  All 
of the other components are expensive by design but usually proceed well.

Besides the delay in time, I'd guess that the aggregate cost in people's working 
time is some number of person-weeks. I would not be surprised if it proved more 
meaningful to cast it in terms of person-months.

In other words, as I said, high overhead.

A key point, here, is that the IETF really does not treat Experimental 
differently from Standards Track.  The presumption is that AD evaluation 
criteria are looser, but the default assumption should be that they are not, 
since the IESG has never been that structured or consistent in what prompts a 
Discuss.

d/
-- 

   Dave Crocker

---

From: Eliot Lear <[email-masked]>
Date: Thu, 28 Oct 2010 20:27:59 +0200

Bob,

> Metrics? Examples? Comparisons with experience with other SDOs?

Ran brought up the LISP working group as a positive example of using
EXPERIMENTAL, so let's look at it.  To get even close to being finished,
the group has had to handle 113 issues, some 59 of which were opened by
one person who applied the PS standard to opening an issue (or higher). 
I certainly have seen many MANY PS docs go through with FAR fewer
issues.  As far as I'm concerned, I wouldn't advise someone who wanted
to get to PS to start with Experimental.  And quite frankly, most of the
work done in the IETF is far FROM experimental, but coded and sold.

Eliot

---

From: RJ Atkinson <[email-masked]>
Date: Thu, 28 Oct 2010 14:15:42 -0400

> 1. Getting /any/ RFC through the IETF process is very high overhead,

I believe that publishing an Experimental RFC is
visibly easier than publishing a standards-track RFC.

>    for the IRTF rather than the IETF?  

As my note clearly said, in text you chose not to quote above,
it is already sometimes the case that the IRTF track is used.
HIP and ILNP are recent/current examples of this.  

I believe that most often the IETF often would benefit 
from publishing initially as experimental, then publishing
a revised/clarified specification on the IETF standards-track.

There are a range of examples where this has been done
over the years, with visible success, by various IETF WGs.  
At the moment, the HIP WG is a good example.  As another
example, my understanding, possibly outdated, is that parts 
of the "TCP Multi-Path" work intend to go out directly on
the IETF standards-track, while other parts intend to go 
out initially as Experimental.

> Shouldn't a standards process be able to sit down and do a standard,
> rather than iterate on experimental designs?

The above seems to reflect a mis-reading of what I wrote.

I merely suggested that often an IETF WG would find it 
more productive to go to Experimental RFC first,
then later to the standards-track.  There are a number
of worked examples of this historically, going back
some number of years.  I provided a handful of recent
or current examples.

This suggestion is not in any way novel or strange.
Indeed, I'm merely echoing the suggestions of other 
folks -- this is not an idea that I originated.

Cheers,

Ran

---

From: John C Klensin <[email-masked]>
Date: Fri, 29 Oct 2010 14:23:31 -0400

--On Thursday, October 28, 2010 14:15 -0400 RJ Atkinson
<[email-masked]> wrote:

>> 1. Getting /any/ RFC through the IETF process is very high
>> overhead, including Experimental.
> 

While, based on the recent EAI experience, I think there is
sometimes (not necessarily often) considerable advantage in
going to Experimental first, I don't believe that there is any
significant difference in the time or pain level associated with
getting WG Experimental documents through the IESG as compared
to Proposed Standard documents.  There may be some, e.g., it is
easier with an Experimental document to respond to "DISCUSS: I
don't think this can be implemented, nor that it will work as
intended" with "that is what we are trying to find out and
document", but I've only rarely seen that come up in a DISCUSS
on either type of document.   

As Dave has tried to say in a variety of ways, these things just
depend on the views of the responsible AD and the composition of
the IESG -- generalizations are generally not possible other
than that the mechanisms that produce "slow and/or painful" take
a lot less IESG consensus than moving things along quickly.

The situation might be quite different for non-IETF streams and
possibly non-WG document streams.

>> Shouldn't a standards process be able to sit down and do a
>> standard, rather than iterate on experimental designs?
> 
> 

I think this takes us immediately back to those fundamental
questions of what problem we are trying to solve and why do we
think it would make a difference.  I still remember my very
first task when I became Apps AD: I got to explain to a vendor
that, by implementing and deploying something based on (IIR) a
Proposed Standard, they had only themselves to blame for their
deployed system being incompatible when the WG decided a
particular feature was a bad idea and changed it.   I don't
think it would be possible for an AD to do that today: along
with making it harder to get Proposed Standard, we have started
to treat them as if they are cast in concrete (one can argue
cause and effect either way).

Personally, I continue to believe that the Internet would be
better served by having a lot less difference between Proposed
and Experimental, by changing things (back?) so that getting one
or the other approved and published was really quick, and making
our main standardization/ get the document exactly right /
consensus level the second one.    In that context, I think it
would make less difference whether we had two levels or three
(but I'm the guy who proposed replacing category-levels that are
never quite right with narrative text what could be used to
explain what we really thought about a spec and its maturity).  

But, if we are going to persist in treating Proposed as "already
refined specification to be implemented and deployed" and in
requiring a very high standard of documentation, specification,
and agreement for Experimental, then it seems to me that the
odds are good that trying to insert Experimental "below"
Proposed would, in lots of cases, just push Proposed up (four
levels?) and make things even slower.

And that would, IMO, increase the justification for documents
bearing a WG-only stamp of approval such as the "snapshot" I-D
plan.  I'm not a fan of that plan because I think we should fix
Proposed, including reduce the time and trouble it takes to get
a spec published as Proposed, but, if we really can't do that,
then formal pre-approval by a WG seems like the right way to go.

Strawman (consciously derived from the current procedures of
another SDO who, oddly, modeled those procedures on what they
thought we are doing a decade or two ago):

Suppose we eliminate "Proposed Standard", replacing it with
"IETF Area Specification" or "Preliminary Specification" (there
are probably better names) with the following properties:

	-- No known technical defects (the present criterion for
	Proposed)
	
	-- Review and approval only within the (single) area of
	origin, i.e., WG review and AD review and approval (with
	whatever addition input the AD wanted to ask for, as
	now), but no IESG approval process at all.
	
	-- Published as RFCs, but clearly labeled as
	"preliminary", "possibly incomplete" and "not a
	standard".  I wouldn't expect that to do much good,
	especially if someone were inclined to distort the
	situation for marketing reasons, but we can and should
	try.
	
	-- Instructions to the RFC Editor to favor speed of
	publication and basic comprehensibility over getting the
	text just right.

Presumably this gets things through the IESG faster, if only
because the number of people who can launch a formal DISCUSS is
reduced to two from 15.  It would also reduce IESG workload on
documents that really are preliminary somewhat which, IMO, would
be A Good Thing.

While my instinct is that RFC publication would be desirable, if
that didn't seem workable we could move the idea a bit closer to
the "Snapshot" idea by posting the document in the I-D series
and giving it a gold star.

Then we have a second level that constitutes a real IETF Spec:
satisfaction of various "nits", full community review, including
cross-area review and evidence of community consensus.  I don't
know if we would need one of those levels or two, but an
experiment might be useful.   The advantage of keeping two (or
more) is that a WG could advance something directly to the first
IETF level if they thought the spec was stable and complete
enough, or proceed on the basis of getting an "Area Preliminary
Specification" out as soon as there was agreement on the
technical spec but then move directly (no waiting period) on to
the level of document refinement now needed for Proposed.   

If the WG didn't care about IETF approval, then they don't care.
Nothing stops such a WG from giving up after posting I-Ds today;
with the "Snapshot" idea, they would have equally little (or
much) incentive to move to full IETF review.

If this gets any traction, I'd be happy to write it up or help
someone else do so.  Otherwise, just consider it another
demonstration that it is possible to address the problem of "two
high a barrier to Proposed, few things move to Draft" to which
draft-housley-two-maturity-levels appears to be addressed in a
way that is meaningful, i.e., looking at what happens at the
entry level rather than hoping that the problems with levels 1
and 2 will be magically solved by eliminating level 3.

    john

---

From: SM <[email-masked]>
Date: Fri, 29 Oct 2010 13:20:23 -0700

Hi John,

This is a quick reply to your message.  Please treat it as highly immature.

>explain what we really thought about a spec and its maturity).

Keith posted some comments about document quality.  The ideas are 
good but implementing them will create more work.  It might be 
possible to combine the ideas with other proposals to come up with a 
workable alternative.  I don't know whether Keith will find a 
dilution of the ideas acceptable.

Agreed.

>and giving it a gold star.

It would be difficult to get buy-in if the document is not published 
as a RFC.  Instead of  eliminating Proposed Standard, how about 
allowing the working group output document to be
published as Proposed Standard?  The approval could be done within 
the working group only but that might results in documents of 
questionable quality.  If we take your idea of having an area review 
instead, some problems which a working group might ignore would be 
caught.  It would also look better for labelling purposes.

There will be technical defects and architecturally unsound proposals 
coming out of this.    For those who might say that it is 
unacceptable, I might point out that there it is already the case.

It could work as follows:

   (i)   Working Group Last Call

   (ii)  Area-wide Last Call - This is for the document to get
         more exposure

   (iii) AD review of comments and document

   (iv)  Document published as Proposed Standard with text saying
         that it represents the consensus of Area X and does not
         represent IETF consensus

   (v)   Document goes into "do or die" track

   (vi)  Working Group writes a revised proposal

   (vii) Last Call and IESG evaluation

   (ix)  Document published as Draft Standard.  It represents the
         consensus of the IETF and benefits from IETF-wide review

   (x)   Deployment report required for document to be moved to
         Internet Standard (I am not getting into what the report
         should be about to keep this short).

The above can also be used for Individual Submissions.  The above 
would reduce the load of work of the IESG and they can focus on the 
restricted set of documents that try to make it to Draft or Internet 
Standard.  ADs have less documents to read and IETF participants are 
not overloaded by Last Calls.

>cross-area review and evidence of community consensus.  I don't

There are benefits to having cross-area review of documents.  It is 
unfortunately not possible to have that done for all documents 
currently going for Proposed Standard.  What I wrote above would 
mitigate the problem instead of solving it.

>much) incentive to move to full IETF review.

There is still the issue of timeliness.  It's up to the WG to 
determine whether it wants to use Experimental or have several 
experimental proposals to gain experience in choosing the better 
proposal.  The WG cannot blame the IESG for putting up barriers as it 
has been told that there is an expectation that the issues will have 
be resolved at some point (Draft Standard) or else its work will go 
through a reclassification by default.

There are a lot of details that are not covered in the above.  Fow 
what it is worth, it might even be possible to do the above within 
the bounds of RFC 2026 but that would require some liberal reading of 
the RFC together with some tweaks to other RFCs.

Regards,
-sm

1. http://www.ietf.org/mail-archive/web/ietf/current/msg64345.html

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 29 Oct 2010 16:39:00 -0400

On Fri, Oct 29, 2010 at 01:20:23PM -0700, SM wrote:
> It would be difficult to get buy-in if the document is not published as a 
> RFC.

Supppse we actually have the following problems:

    1.  People think that it's too hard to get to PS.  (Never mind the
    competing anecdotes.  Let's just suppose this is true.)

    2.  People think that PS actually ought to mean "Proposed" and not
    "Permanent".  (i.e. people want a sort of immature-ish level for
    standards so that it's possible to build and deploy something
    interoperable without first proving that it will never need to
    change.)

    3.  We want things to move along and be Internet STANDARDs.

    4.  Most of the world thinks "RFC" == "Internet Standard".

If all of those things are right and we're actually trying to solve
them all, then it seems to me that the answer is indeed to move to _n_
maturity levels of RFC, where _n_ < 3 (I propose 1), but that we
introduce some new document series (call them TRFC, for "Tentative
Request For Comment", or whatever) that is the first step.  Then we
get past the thing that people are optimizing for ("everything stays
as Proposed Standard once it gets published") by simply eliminating
that issue permanently.

Ah, you say, but now things will stick at TRFC.  Maybe.  But we could
on purpose make it easier to get TRFC than it is now to get PS (say,
by adopting John's limited DISCUSS community for TRFC, or one of the
other things discussed in this thread).  Also, the argument about
everyone thinking that RFCs are "standard", and the resulting pressure
to make them perfect and permanent, would be explicitly relieved (at
least for a while), because nobody thinks that TRFCs are standards. 

Note that this is not to denigrate SM's suggestion, which also doesn't
seem wrong to me.  But since one of the issues appears to be that
anything called "RFC" is set in stone, then if we just stop calling
the early-publication documents "RFC" that and introduce something
after I-D (which is formally only on the _way_ to some consensus, and
not actually the product of it), the blockage might be removed.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Martin Rex <[email-masked]>
Date: Fri, 29 Oct 2010 23:03:05 +0200 (MEST)

SM wrote:
> questionable quality.

Essentially, you seem to be asserting that IETF community feedback
should be considered harmful and delayed to much later in the process
where it can have even less impact.

Previous assertions where about the IESG making things too difficult.

To me, that sound a little like giving up.
Changing solutions, fixing protocols and fixing documents is exhausting
and painful, so let's just skip all of that.  Let vendors and implementors
wiggle out how to create interoperable products from shoddy specs
all by themselves -- which is what some of them have been doing for
some time -- implementing defective specs and shipping interoperability-
impaired products long before the standardization work has converged
on a moderately stable Proposed Standard.

To me that looks like one way to obsolete the IETF
(IETF community consensus and IESG review).

-Martin

---

From: Michael Richardson <[email-masked]>
Date: Fri, 29 Oct 2010 21:15:09 -0400

>>>>> "Andrew" == Andrew Sullivan <[email-masked]> writes:
    Andrew> On Fri, Oct 29, 2010 at 01:20:23PM -0700, SM wrote:
    >> It would be difficult to get buy-in if the document is not
    >> published as a RFC.

    Andrew> Supppse we actually have the following problems:

    Andrew>     1.  People think that it's too hard to get to PS.
    Andrew> (Never mind the competing anecdotes.  Let's just suppose
    Andrew> this is true.)

    Andrew>     2.  People think that PS actually ought to mean
    Andrew> "Proposed" and not "Permanent".  (i.e. people want a sort of
    Andrew> immature-ish level for standards so that it's possible to
    Andrew> build and deploy something interoperable without first
    Andrew> proving that it will never need to change.)

    Andrew>     3.  We want things to move along and be Internet
    Andrew> STANDARDs.

    Andrew>     4.  Most of the world thinks "RFC" == "Internet
    Andrew> Standard".

    Andrew> If all of those things are right and we're actually trying
    Andrew> to solve them all, then it seems to me that the answer is
    Andrew> indeed to move to _n_ maturity levels of RFC, where _n_ < 3
    Andrew> (I propose 1), but that we introduce some new document
    Andrew> series (call them TRFC, for "Tentative Request For Comment",
    Andrew> or whatever) that is the first step.  Then we get past the
    Andrew> thing that people are optimizing for ("everything stays as
    Andrew> Proposed Standard once it gets published") by simply
    Andrew> eliminating that issue permanently.

I think this is a workable idea.
But, instead of calling things TRFCXXXX, let's do something less
glamorous and give it hash... maybe based upon the sha1 of the document
or something.  TRFC-ipsec-4d66-1618-00bbd99b.txt :-)

-- 
]       He who is tired of Weird Al is tired of life!           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
	               then sign the petition.

---

From: John C Klensin <[email-masked]>
Date: Sat, 30 Oct 2010 04:39:13 -0400

A few quick observations...

--On Friday, October 29, 2010 13:20 -0700 SM <[email-masked]>
wrote:

>...
>> While my instinct is that RFC publication would be desirable,
>> if that didn't seem workable we could move the idea a bit
>> closer to the "Snapshot" idea by posting the document in the
>> I-D series and giving it a gold star.
> 

If that is true --and it may be-- it would indicate that not
even we can keep track of the difference between "RFC" and
"Standard".  If that were to be the case, discussion of maturity
levels is basically a waste of time.

>  Instead of  eliminating Proposed
> Standard, how about allowing the working group output document
> to be
> published as Proposed Standard?  The approval could be done
> within the working group only but that might results in
> documents of questionable quality.  If we take your idea of
>...

I think there are other issues with your outline, but the key
one is that it would really, really, depend on "do or die"
working.  If it didn't, the IETF would rapidly acquire a
reputation for producing garbage as Standards, and that would
be, IMO, really bad.

But the odds are against "do or die".  We've had that provision
(automatic moves to Historic for Proposed (or Draft) Standards
that are not advanced within a set period) for a very long time.
Unlike the Proposed Standard criteria, which have gradually
evolved to become more and more burdensome in practice, we have
_never_ followed that rule as written and only once (the
"de-cruft" spinoff from the NEWTRK WG) make a serious attempt to
clean out documents no one cared about any more.

I also suggest that the odds are against us, if only because the
IESG will always have higher priorities than reviewing the
status of documents no one cares about any more (either because
they didn't get traction or because they got so much traction
that they represent established practice that no one has
motivation to update them).  In addition, I'm cynical enough to
believe that IESG members would hesitate to kill off documents
that have a few supporters who might put a lot of effort into
complaining to the Nomcom... at least without strong documented
evidence of community support.

Note that we have also made killing things hard: the idea that
it takes putting together and publishing an RFC with details,
justification, and all of the usual sections and boilerplate to
move another RFC to Historic is a post-2026 innovation.  I think
it was caused by the above issues with the IESG deprecating
things because it was obviously the Right Thing to Do.  But,
regardless of the causes, it means a lot of motivation is needed
to kill (or deprecate) something rather than letting it
languish.  IMO, if we wanted "do or die", we'd have to change
that culture too.

best,
  john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 30 Oct 2010 11:59:48 +0200

On 10/30/2010 10:39 AM, John C Klensin wrote:
> If that were to be the case, discussion of maturity

I think it is. The general public perceives RFCs as RFCs, not equally 
weighty, but NOT ACCORDING TO ANY FORMAL CRITERIA.

We might as well get used to that.

Therefore, I'd like to have maturity levels as follows.

1. draft-foo-0: No requirements, not even that it passes idnits.
1b. draft-foo-<nonzero>: Number of nits should decrease.
2. Final draft: No nits, IPR done, Last Call, archival document,
foo-area review, etc.
3. RFC, whether good and weightly, flimsy and ignored, or April joke.

Arnt

---

From: Yoav Nir <[email-masked]>
Date: Sat, 30 Oct 2010 18:23:01 +0200

> least for a while), because nobody thinks that TRFCs are standards. 

I know how you can get it to approve first: Don't take it to the IESG. Require approval only from the ADs for that area. And don't give them a name that makes them look like some slightly different kind of RFC. Call it "blessed draft" or something like that.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 30 Oct 2010 14:31:07 -0400

I think that there are some issues that are not being mentioned, but 
which are important.

In general, there is the issue of impact.  This takes many forms.  But 
the underlying effect is taht protocols which get widely deployed can 
have distinctly negative impact on the net.  Thus, for many years, the 
requirement for routing protocols which could effect changes to core 
behavior was that they had to have demonstrated interoperable 
implementations for PS.  This was acknowledged by all involved to be a 
higher standard than 2026 called for.  It was an area rule (going back 
to about 1992.)  On the other hande, there are plenty of things, even in 
routing, which do not have those kinds of effects and do not need those 
safety measures.  But you can not tell without looking.

Similarly, it is important to make sure we check out the congestion 
behavior of our protocols before we tell folks to go implement them 
widely.  Sometimes this is trivial, sometimes it isn't.  But ignoring 
the question when we are saying "this is ready for deployment" is not a 
good thing.  One of the positive effects of our current system is taht 
because WG knows tha tthye have to clear all the ADs, not just their 
own, they actually think about all these issues.  And usually manage to 
cope with them

To give yet another example, I know of one WG that seriously considered 
publishing an Experimental RFC that flat violated existing standards. 
The issue was found, and is being addressed.  But without cross-area 
review (the issue was a transport issue for an Internet WG) it is all 
too easy to miss things.

Yes, it would be very good to spot all of these things sooner.  I have 
not yet seen a proposal that actually works for doing so.  But letting 
WGs or WGs + ADs approve documents for general advancement is a step 
likely to lead to problems.
If all our WGs handed their ADs high quality documents that they had 
checked for all these issues, then maybe we could look at this 
differently.  But LOTs of examples demonstrate taht WGs do not always do 
this.  (Sometimes WGs give their ADs great documents.  Sometimes not.)

Yours,
Joel M. Hslpern

On 10/30/2010 12:23 PM, Yoav Nir wrote:
>> maturity levels of RFC, where _n_<  3 (I propose 1), but that we
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 31 Oct 2010 08:48:43 -0700

On 10/30/2010 11:31 AM, Joel M. Halpern wrote:
>      One of the positive effects
> of our current system is taht because WG knows tha tthye have to clear all the
> ADs, not just their own, they actually think about all these issues. And usually
> manage to cope with them

A working group is diligent or it isn't.  It gets a range of feedback and 
responds constructively to it... or it doesn't.

The working group behaviors that I have seen that pay explicit attention to the 
specific question of satisfying AD reviews have nothing to do with quality of 
the work and more to do with guessing what will personally bother an AD.  In 
other words, it's about dealing with AD idiosyncrasy rather than with quality.

It is now common to get cross-area reviews and my own observation is that these 
are a) typically quite reasonable and diligent, and b) dealt with constructive 
by the working group.

ADs do sometimes come up with interesting and even important points, but AD 
review is an extremely expensive and often frustrating mechanism that we already 
have a vastly superior replacement for.  Its timing is better and it distributes 
the work far better.

The fact that an AD sometimes catches some important problem is typically taken 
as proof that the AD review and Discuss mechanism is essential.  This is highly 
flawed logic, on two counts.

One is that it does not represent meaningful cost/benefit evaluation.  The cost 
is actually quite high in energy, delay and frustration, and the significant 
benefit overall is quite low (if the wg has been diligent and has gotten 
cross-area reviews.)

The other is that protocol specs have a statistical likelihood of bugs, even 
with the AD review.  We talk about AD review almost as if it ensures perfection, 
but of course we know it does not.

Ultimately, we have to trust the real world to evaluate the safety and efficacy 
of a protocol.  That fact ought to give us permission to balance the cost and 
benefit of the quality assurance efforts we require during specification 
development and approval.

> Yes, it would be very good to spot all of these things sooner. I have not yet
> seen a proposal that actually works for doing so. But letting WGs or WGs + ADs
> approve documents for general advancement is a step likely to lead to problems.
> If all our WGs handed their ADs high quality documents that they had checked for
> all these issues, then maybe we could look at this differently.

We do need quality assurance efforts.  The basic idea that working group efforts 
are subject to outside review prior to approval is a significant value-add by 
the IETF, IMO.  The question is how to provide sufficient review in a reasonable 
way.

I believe that cross-area reviews largely satisfy that requirement.  If 
within-area reviews are also needed, the AD should commission them, not do them 
directly.

d/
-- 

   Dave Crocker