Subject: what is the problem bis

From: [email-masked] (Scott O. Bradner)
Date: Tue, 26 Oct 2010 12:08:57 -0400 (EDT)

while we are the topic of problems

Russ basically proposes too change the maturity warning label on IETF
standard track RFCs -- remove baby before folding carriage -- this
hardly seems like our biggest problem

The IETF publishes a lot of standards track RFCs each year.  Mostly
these are PS (186 in 2009), some DS (3 in 2009), and some S (6 in 2009).  

SOME of these technologies are just what the community needs and just
when the community needs them.  But too many are 
   1/ too late for the market - implementations based on IDs
      deployed or other technologies adopted
   2/ unneeded by the market - does not meet a need that people
      think they have
   3/ broken - flawed in some way that prevents actual deployment
   4/ too complex - hard and costly to correctly implement
   5/ unmanageable - cannot be run by humans

Seems to me that the issue of how the IETF can be better at producing
just what the community needs just when the community needs it is more
important than maturity warning labels.

Scott

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 26 Oct 2010 12:13:42 -0400

> important than maturity warning labels.

Would the first step be to try and get some statistics, to see how many of those ~ 200
standards fall into class 1-6 ?

Regards
Marshall

> 
> Scott
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Ross Callon <[email-masked]>
Date: Tue, 26 Oct 2010 12:32:17 -0400

I don't think that anyone is claiming that the two-maturity-levels draft solves every problem. This draft should not discourage you or anyone else from offering additional proposals to solve the problems that you are mentioning in your email below. 

There are two problems that Russ's draft may very well solve: One issue with our current system is that there is no incentive to go from Proposed Standard to Draft Standard (since you are only going from one "intermediate state" short of full standard to another "intermediate state" also short of full standard). Another issue is that increasingly each of our standards relies on multiple other standards, so that RFCs can only move to Draft Standard if multiple other drafts do also, and it is too much trouble to move multiple drafts all at the same time. 

The downside of Russ's draft is that it is possible that after approving it we might find that nothing changes: Protocol specifications still stay at Proposed Standard; The IESG still takes a lot of time in approving a request to publish a proposed standard RFC. This possible downside seems to be a valid argument that people should continue to think about other improvements to the system (and send comments to nomcom on the candidates for IESG positions). However, this potential possible downside doesn't strike me as a reason to refuse to make the modest step in the right direction that IMHO is expressed in Russ's draft. 

Ross

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Scott O. Bradner
Sent: Tuesday, October 26, 2010 12:09 PM
To: [email-masked]
Subject: what is the problem bis

Scott

---

From: Eliot Lear <[email-masked]>
Date: Tue, 26 Oct 2010 19:18:12 +0200

Ross,

On 10/26/10 6:32 PM, Ross Callon wrote:
> The downside of Russ's draft is that it is possible that after approving it we might find that nothing changes: Protocol specifications still stay at Proposed Standard; The IESG still takes a lot of time in approving a request to publish a proposed standard RFC. 
How will we measure effect?  And what other factors need to be controlled?

(he says, recognizing we are not in an isolated lab)

Eliot

---

From: todd glassey <[email-masked]>
Date: Tue, 26 Oct 2010 10:46:32 -0700

> while we are the topic of problems

No Scott the problem is that the IETF is not a lobbyist organization and 
your blocking the standardization of anything based on whether the 
Internet "needs it or not" makes your IETF the controller of what gets 
routed and what not, it also creates trememdous liability and opens all 
of the parties here to litigation based therein one would think...

Todd Glassey

> hardly seems like our biggest problems
>

-- 
//-----------------------------------------------------------------

This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this for the
addressee, you must not use, copy, disclose or take any action based
on this message or any information herein. If you have received this
message in error, please advise the sender immediately by reply e-mail
and delete this message.

Thank you for your cooperation.

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 26 Oct 2010 10:48:38 -0700

> standards fall into class 1-6 ?

Sure would be nice to have a place for noting the basic data.

What if someone created a wiki... [1]

d/

[1]  http://trac.tools.ietf.org/misc/outcomes/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 26 Oct 2010 10:54:58 -0700

On 10/26/2010 9:32 AM, Ross Callon wrote:
> There are two problems that Russ's draft may very well solve: One issue with
> our current system is that there is no incentive to go from Proposed Standard
> to Draft Standard (since you are only going from one "intermediate state"
> short of full standard to another "intermediate state" also short of full
> standard).

The theory that this change will create this incentive is exactly what I meant 
by charming but unfounded.  Really, the premise here is an appealing fantasy. It 
presumes that the extra label imposes a psychological barrier, but there is no 
evidence that this is true.

What this thinkin really does is to take attention away from the actual 
barriers, which others have cited at length.

Working groups take too long.  The IESG often takes too long and ADs often raise 
unexpected and possibly even arbitrary barriers.  We have moved to an enormously 
heavyweight model.  Timeliness is almost never a factor.

Nothing gets better until that changes.

> Another issue is that increasingly each of our standards relies on
> multiple other standards, so that RFCs can only move to Draft Standard if
> multiple other drafts do also, and it is too much trouble to move multiple
> drafts all at the same time.

This, at least, is a pragmatic point.  I think there has been little effort to 
evaluate it deeply.  It might have some benefit; it might not.  Where is the 
archive of consideration?

d/
-- 

   Dave Crocker

---

From: Ross Callon <[email-masked]>
Date: Tue, 26 Oct 2010 14:27:05 -0400

> heavyweight model.  Timeliness is almost never a factor.

In general I think that there is a lot of truth to this comment. I agree that Russ' proposal does not fix these problems. However...

> What this thinking really does is to take attention away from the actual 

This is where I disagree with you. The simple change that Russ has proposed is not what is taking away from discussion of the actual barriers. What is taking attention away from discussion of the actual barriers is the lengthy debate about Russ's proposed change.

Russ's proposed change is a small step in the right direction. It is not going to solve all problems (with the IETF process, the Internet, various national debts, global warming, or <fill in your favorite problem here>). If we want to solve other more serious problems, then feel free to come up with proposed solutions. However, blocking one small step in the right direction is not a productive way to encourage a different larger step in the right direction. 

The only way that we can find out whether moving to a two stage process will help is to try it out. I agree that there is a possibility that we might find that this causes no real change to the way that standards are done. I don't see that as such a bad downside. The potential upside is that there may be slightly more RFCs moved to full standard status. 

In my opinion the fact that this very simple and straightforward change draws such heavy debate is a disincentive to anyone who would propose other additional changes. 

Ross

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave CROCKER
Sent: Tuesday, October 26, 2010 1:55 PM
To: [email-masked]
Subject: Re: what is the problem bis

d/
-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 26 Oct 2010 11:39:33 -0700

On 10/26/2010 11:27 AM, Ross Callon wrote:
>     What is
> taking attention away from discussion of the actual barriers is the lengthy
> debate about Russ's proposed change.

Here's where the term 'opportunity cost' applies:

    Taking action that does not achieve what is desired consumes energy and time 
and often eliminates the ability to achieve the goal later.  This is a strategic 
and very damaging cost.

> Russ's proposed change is a small step in the right direction.

Folks keep saying that.  However the change is actually quite irrelevant to the 
direction.

I'm a fan of reducing down to 2 levels, too.  But it has nothing to do with how 
overblown the effort to get to Proposed is.  (Well, there's some pretty simple 
psych logic that says that it could actually make the barrier to Proposed 
HIGHER!  After all, fewer steps means that each step is more valuable.)

> It is not going to solve all problems

There is no need to invoke that bit of extreme and unasserted goal.

It is not going to solve /any/ of the major goals it puts forward.

> The only way that we can find out whether moving to a two stage process will
> help is to try it out.

"try it out".  I love this experimental attitude about a fundamental, production 
service for the Internet.

After all, it doesn't hurt anything to try and fail, right?  It's not as if a 
change like this affects an entire, global technical community.

I will again invoke 'opportunity cost'.  Changes like these are very far from 
free, in strategic terms.

> In my opinion the fact that this very simple and straightforward change draws
> such heavy debate is a disincentive to anyone who would propose other
> additional changes.

It is certainly true that our community would rather criticize than enhance.

But the real problem here is that that problem leads folk to ignore legitimate 
criticism and to refuse any sort of comparative analysis of alternative 
approaches to enhancement.

d/
-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Oct 2010 14:51:58 -0400

--On Tuesday, October 26, 2010 10:54 -0700 Dave CROCKER
<[email-masked]> wrote:

>> There are two problems that Russ's draft may very well solve:
>> One issue with our current system is that there is no
>> incentive to go from Proposed Standard to Draft Standard
>> (since you are only going from one "intermediate state" short
>> of full standard to another "intermediate state" also short
>> of full standard).
> 
> The theory that this change will create this incentive is
> exactly what I meant by charming but unfounded.  Really, the
> premise here is an appealing fantasy. It presumes that the
> extra label imposes a psychological barrier, but there is no
> What this thinkin really does is to take attention away from
> the actual barriers, which others have cited at length.
> 
> Working groups take too long.  The IESG often takes too long
> and ADs often raise unexpected and possibly even arbitrary
> barriers.  We have moved to an enormously heavyweight model.
> Timeliness is almost never a factor.
> 

+1

>> Another issue is that increasingly each of our standards
>> relies on multiple other standards, so that RFCs can only
>> move to Draft Standard if multiple other drafts do also, and
>> it is too much trouble to move multiple drafts all at the
>> same time.
> 
> This, at least, is a pragmatic point.  I think there has been
> little effort to evaluate it deeply.  It might have some
> benefit; it might not.  Where is the archive of consideration?

+1.  Moreover, if that is the problem, then we should modularize
things and address that problem.  Sam and I thought we had that
problem under control with RFC 4897, but it has almost never
been used.  Randy and Thomas thought they had it under control
with RFC 3967, but I think that has been applied even less
often.  I think there are real downsides to eliminating the
normative reference rules entirely and would prefer to see
3967/4897 actually tried first, but, if there is real evidence
that inter-document linkages are a problem, we can change that
(again) without that being any sort of proof that the standards
track needs drastic revision.

    john

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 26 Oct 2010 15:24:55 -0400

The problem with the current, failed process is that there is absolutely no
correlation between the standards status of a protocol and adoption.

Most of the documents to reach STANDARD status in recent years have been
SNMP documents. But even though SNMP has its uses, deployment and use hardly
compares with HTTP which is at DRAFT. And nobody should be considering using
the SNMP 2.0 protocol which is the one that is allegedly STANDARD as it has
no security layer.

I can't see the logic of delaying further at this. The only reason being
given for not acting is that it will take time. But not acting has sucked up
an enormous amount of time.

-- 
Website: http://hallambaker.com/

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Oct 2010 15:47:39 -0400

--On Tuesday, October 26, 2010 14:27 -0400 Ross Callon
<[email-masked]> wrote:

> This is where I disagree with you. The simple change that Russ
> has proposed is not what is taking away from discussion of the
> actual barriers. What is taking attention away from discussion
> of the actual barriers is the lengthy debate about Russ's
> proposed change.

Ross, I can't speak for Dave or anyone else, but I think part of
the disagreement here is that some of us don't see this as a
"simple change".  From my point of view, it is quite drastic
because it eliminates a three stage model for those protocols
and groups for which it works, doesn't provide a way to undo the
change if it turns out to be a bad idea, and has not been
demonstrated (empirically or logically) to have any effect on
the number of documents that advance from Stage 1 to Stage 2.

As Dave points out, our history has been that the perception
that it is hard to get to Draft (or that many documents stop at
Proposed) has led to our making it hard to get to Proposed.
There is no evidence that this change would have any effect on
that pattern; it might even make it worse by eliminating the
"extra chance to get it right".

Worse, it seems to combine several different measures, including:

	(1) Dropping Internet Standard and Renaming Draft to
	Internet Standard.  But it doesn't quite do that because
	it also drops the requirements for identification of
	deployment and usefulness that go with Internet
	Standard, substituting only the "two independent
	interoperable" implementations of Draft.
	
	(2) Changing the Normative Reference rules by dropping
	them, with no consideration of side effects.
	
	(3) Posing an alternative between keeping STD numbers as
	they now exist and dropping them entirely, without
	considering a third path, which has been proposed
	several times: assigning those numbers at Proposed (or
	even somewhat earlier).  That possibility has apparently
	been dismissed as addressing a different problem but the
	discussion of STD numbers remains in the document.

A small and simple change would either modularize these and
discuss and make them one at a time or would explain why they
are linked.

> Russ's proposed change is a small step in the right direction.

And we disagree both about whether it is a small step and
whether it is in the right direction.

>...
> In my opinion the fact that this very simple and
> straightforward change draws such heavy debate is a
> disincentive to anyone who would propose other additional
> changes. 

A different way to look at this would be to suggest that the way
this document has been handled -- disallowing both smaller, more
modular, and more incremental changes and more sweeping ones
that might have higher payoff in the process-- is a disincentive
to any serious thinking about what problems we are trying to
solve and how particular approaches might solve them.

    john

---

From: Keith Moore <[email-masked]>
Date: Tue, 26 Oct 2010 15:59:02 -0400

> important than maturity warning labels.

agreed, though we should be careful to not confuse "what the community needs" with "what vendors implement".  both are useful to know, but they're not the same thing.

Keith

---

From: Hadriel Kaplan <[email-masked]>
Date: Tue, 26 Oct 2010 16:01:41 -0400

> HIGHER!  After all, fewer steps means that each step is more valuable.)

I'm not sure it's technically possible for it to take longer to produce a PS in the IETF.  You'd have to invent some sort of time distortion device, or send the docs on trains to Maastricht. ;)

But I agree with you (and Scott et al) on the general point that reducing 3 to 2 doesn't solve a problem anyone has.  For example, I don't think being at only PS has prevented anyone from deploying HTTP or the myriad of other protocols at PS level.  At this point, going above PS is for masochists.

-hadriel

---

From: Keith Moore <[email-masked]>
Date: Tue, 26 Oct 2010 16:02:41 -0400

> I don't think that anyone is claiming that the two-maturity-levels draft solves every problem. This draft should not discourage you or anyone else from offering additional proposals to solve the pr…

I think it's confusing if we change our labeling very often, say more than once per decade.    So if we're going to change the process, or even just the labeling, it's better to get those "right" than to change them merely for the sake of change.

---

From: Keith Moore <[email-masked]>
Date: Tue, 26 Oct 2010 16:04:29 -0400

On Oct 26, 2010, at 1:54 PM, Dave CROCKER wrote:

> Working groups take too long.  The IESG often takes too long and ADs often raise unexpected and possibly even arbitrary barriers.  We have moved to an enormously heavyweight model.  Timeliness is almost never a factor.
> 

and yet, working groups often try to address the wrong problems, because they're so narrowly scoped as to avoid tackling the important problems.

nothing gets better until that also changes.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 26 Oct 2010 16:17:26 -0400

> In my opinion the fact that this very simple and straightforward change draws such heavy debate is a disincentive to anyone who would propose other additional changes. 

Often the reason that "simple and straightforward changes" draw such heavy debate is precisely because people are trying to use "simple and straightforward" changes to take things in a direction that is not well understood, or that doesn't actually address the real problems.  

So people are forced to treat the debate about "simple and straightforward changes" as a proxy for a debate about the direction.  But because the direction is not well understood, the debate is also very muddy and inconclusive.  And because some people will insist that the scope of the debate be confined to the document in question, people have to couch their arguments in those terms - even though their actual concerns are somewhat different.  Arguments like this do not generally lead to either better understanding or consensus.  

Usually the subject of debate is a technical argument, rather than an argument about process.  But it's the same problem.

Keith

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 26 Oct 2010 14:32:38 -0700

Hi -

> From: "Phillip Hallam-Baker" <[email-masked]>
> To: "Scott O. Bradner" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Tuesday, October 26, 2010 12:24 PM
...
...

Fact check - the STANDARD is SNMPv3, not any of the various flavours of v2,
and SNMPv3 has authentication, integrity, confidentiality, and access control.

Randy

---

From: Ofer Inbar <[email-masked]>
Date: Tue, 26 Oct 2010 17:57:01 -0400

On Oct 26, 2010, at 2:39 PM, Dave CROCKER wrote:
> I'm a fan of reducing down to 2 levels, too.  But it has nothing to
> do with how overblown the effort to get to Proposed is.  (Well,

I feel like we already have a 2-level system.

What's the practical difference between Proposed and full Standard?

It may take a lot of effort to make that jump, but what difference
does it make outside of the fact that the document now has a new
label?  Who notices or cares about the distinction?  Does it affect
anything real?

These aren't rhetorical questions.  I feel like there's no practical
effect, but I may be wrong, and I'm asking.
  -- Cos

---

From: Michael Richardson <[email-masked]>
Date: Tue, 26 Oct 2010 15:20:17 -0400

>>>>> "Eliot" == Eliot Lear <[email-masked]> writes:
    >> The downside of Russ's draft is that it is possible that after
    >> approving it we might find that nothing changes: Protocol
    >> specifications still stay at Proposed Standard; The IESG still
    >> takes a lot of time in approving a request to publish a proposed
    >> standard RFC.

    Eliot> How will we measure effect?  And what other factors need to
    Eliot> be controlled?

The obvious measure of number of IDs->PS is interesting.

But, perhaps the average length of time for ID-00 -> PS is more
important, overall to the entire discussion.  That number needs to
additionally have a time point of WG LC, move to IESG and also IETF LC.

-- 
]       He who is tired of Weird Al is tired of life!           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
	               then sign the petition.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 27 Oct 2010 00:03:07 -0400

For which Working Groups does the current system work?

It is completely failing for every one that I have been involved in. The
distinction between DRAFT standard and Internet STANDARD seems completely
arbitrary as far as I can see.

We might as well replace the final step of the process with throwing darts
at a board for all the good it does.

It is not true that there is no way to reverse the change. If we change the
documents to reflect current practice and it turns out to be a bad idea, the
change can be reversed through exactly the same process by which it was
originally proposed.

Now that is obviously not going to happen as the three step process has been
a total failure. But that is entirely due to the merits and has nothing to
do with the lack of a process for reversal.

As for the STD numbers, we have to either get rid of them completely or give
them another acronym. Nobody wants to refer to their work product by an
abbreviation for Sexually Transmitted Disease. I am totally serious about
this.

As a practical matter, protocols are either referred to by name or by RFC
number. I cannot see any context in which I would use the STD number.

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 27 Oct 2010 07:47:26 -0400

> correlation between the standards status of a protocol and adoption.

Why exactly is that a problem?  That's not a rhetorical question.  If
_that_ is what the problem is, I want to know why.  I agree that two
maturity levels might help if it is the problem to solve.

Supposing it is a problem, why do you think that two maturity levels
won't just cause everything to stick at Proposed Standard?

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Yoav Nir <[email-masked]>
Date: Wed, 27 Oct 2010 14:58:29 +0200

This comes back to the question or why have maturity levels at all. Ideally, an implementer should prefer to implement a mature standard over a less-mature one. In practice, adopting the more advanced standard may give you an obsolete protocol, rather than a more stable one. IOW the standardization level of a document does not give a potential implementer any signal as to whether or not this standard is in any sense of the word "good". And if it doesn't signal anything to the "customers" of the documents, what's the point of having these levels at all?

Suppose (picking a random recent RFC) I'm into the Host Identity Protocol, and I would like to implement multi-hop routing. Searching the RFC database (or Wikipedia) I see that there's an RFC for that: 6028. So if I want to implement multi-hop routing, I would implement this, even though it is "Experimental". Even if there was an older specification for the same thing at full standards, I would find the appropriate mailing list and ask which one to implement. I would *not* go for the older document just because it's a full standard.

Given this, why should the authors of RFC 6028 ever bother to advance it to PS, DS or FS?  If this specification needs updating anyway, they might take the opportunity to also advance it, but otherwise what's the point. 

At some point I found that RFC 4478 had 3 independent implementations that interoperated with each other. I asked an AD about advancing it, and his response was along the same lines: why bother? My time (and his) can be better spent on other things.

So in answering you second question, I don't see any reason why things won't keep sticking in PS or even Experimental forever.

I am also not sure that the bar for Experimental is significantly lower than PS. The only difference is that without actual implementations deployed in the field, the ADs feel better about letting a standard go to Experimental. So even if Russ's draft gets adopted, we'll still be with three maturity levels, except that Experimental will be the bottom rung (on second though, it already is)

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Andrew Sullivan
Sent: 27 October 2010 13:47
To: [email-masked]
Subject: Re: what is the problem bis

A

Scanned by Check Point Total Security Gateway.

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 27 Oct 2010 08:13:55 -0700

>So in answering you second question, I don't see any reason why things won't keep sticking in PS or even Experimental forever.

Here's a reason, and possibly the strongest one: author pride. If I wrote a protocol that I was proud of and I had a strong ego (both of those are purely hypothetical, of course...), I would want to see it move forwards and I might spend the necessary effort to make it happen.

Related, but not as strong of a motivator, is corporate motivation. If WhizzyCorp helped create BlatP and implemented it in their hardware, there is some motivation on their part to be able to say "it's a full standard". I think personal pride would be a stronger motivator.

I support going to two steps so that those motivated to get to the second step don't have to spend the effort on a third. It makes the market for "this protocol is more important than that one" waste fewer person-hours on process.

--Paul Hoffman, Director
--VPN Consortium

---

From: [email-masked]
Date: Wed, 27 Oct 2010 17:18:53 -0700 (PDT)

> Here's a reason, and possibly the strongest one: author pride. If I wrote a
> protocol that I was proud of and I had a strong ego (both of those are purely
> hypothetical, of course...), I would want to see it move forwards and I might
> spend the necessary effort to make it happen.

This is a very good point, and it argues in favor of the current proposal.
Consider: Right now the move to draft is, for specifications of any complexity,
a very nontrivial matter. And the reward, after all is said and done, isn't a
"standard", but rather a "draft standard", which to the uninitiated sounds like
it's a step below proposed. You're then faced with a six month wait, followed
by a fairly ill-defined process step with poorly laid out criteria. All of this
significantly lowers the benefit of the proposed->draft step.

I don't have to guess that this is a significant disincentive to bothering with
advancement, I know it is because it's reflected in my own behavior.

> Related, but not as strong of a motivator, is corporate motivation. If
> WhizzyCorp helped create BlatP and implemented it in their hardware, there is
> some motivation on their part to be able to say "it's a full standard". I
> think personal pride would be a stronger motivator.

"full standard", yes. "draft standard", not so much. Names matter.

> I support going to two steps so that those motivated to get to the second
> step don't have to spend the effort on a third. It makes the market for "this
> protocol is more important than that one" waste fewer person-hours on process.

Exactly right.

				Ned

---

From: Eric Burger <[email-masked]>
Date: Wed, 27 Oct 2010 21:26:00 -0400

Actually, my heartache with Russ' proposal is the automatic "If it's Draft, it's now Standard."

I would be quite happy with Proposed and Internet Standard, with NO grandfathering.

---

From: Keith Moore <[email-masked]>
Date: Wed, 27 Oct 2010 21:57:20 -0400

On Oct 27, 2010, at 8:58 AM, Yoav Nir wrote:

> This comes back to the question or why have maturity levels at all. Ideally, an implementer should prefer to implement a mature standard over a less-mature one. In practice, adopting the more advanced standard may give you an obsolete protocol, rather than a more stable one. IOW the standardization level of a document does not give a potential implementer any signal as to whether or not this standard is in any sense of the word "good".

Mostly agree.  The distinction between an older, well-tested, widely-deployed version of a protocol vs. a newer less-tested, less-widely-deployed version with more features is a useful distinction to make.  but the difference between (RFC X, full) and (RFC Y, proposed) where Y >> X only conveys the barest hint of that, and even less in the way of guidance.  I'm thinking specifically of email standards here, where in practice you need to be able to accept RFC 822 messages (because even new mail readers have to be able to deal with old messages) but you should generate messages that conform to 5322 and MIME.

> And if it doesn't signal anything to the "customers" of the documents, what's the point of having these levels at all?

That's why I think we need a different set of labels, e.g.

Protocol-Quality.  We need a statement about the perceived quality of the protocol described in the document.   (Is this protocol well-designed for the anticipated use cases, or does it have significant flaws (including security flaws)?)
Applicability.  We need a statement about the current applicability of the protocol described in the document.  (Is this protocol recommended for general use, not recommended except in specific corner cases, not recommended at all, or strongly discouraged?)
Document-Quality.  We need a statement about the perceived quality of the document itself and whether the protocol description seems to be sufficiently precise to permit implementations to interoperate.   (along with a pointer to errata.)
Maturity.  We need a statement about the amount of actual implementation and deployment experience that the protocol enjoys. 
Completeness.  We need a statement about how accurately the document reflects what is currently believed to be good practice for implementation/use of that protocol, or whether effective implementation requires information not included or referenced in the document.  (e.g. effective implementation of SMTP generally requires some expertise in dealing with heavy loads caused by spam, looping, and denial-of-service attacks which aren't really dealt with in any of the relevant RFCs).
Last-Review-Date.  Date of the last review of these labels for this document.

These would go alongside the existing Updates and Obsoletes labels.  An Applicability-Statement could also be included.

It strikes me that we could establish such a set of labels on an experimental basis, using some sort of community review process for existing RFCs, without making any immediate changes to our proposed/draft/full system of labeling of standards.  IESG could assign the initial labels for new RFCs - the document reviewers are almost doing that anyway.  The existing errata process could be extended to allow (moderated) user comments on these labels, and the labels could be subject to periodic review based on those comments.

If that labeling system turned out to be adequate or could be fixed with some tweaking, we could maybe drop back to two document classes:  Informational, and Standards-Track.  Standards-track would encompass any  former proposed, draft, or full standard which was still in use and for which periodic reviews were still being done.   Former Experimental documents would be reclassified as Informational with appropriate descriptive labels.  Former Historic documents would be reclassified as Informational with Applicability set to Historic.   Standards-track documents would expected to have periodic review of these labels; Informational documents could have some of those labels set to "Undetermined".

Keith

---

From: Yoav Nir <[email-masked]>
Date: Thu, 28 Oct 2010 11:05:55 +0200

That the labels can be updated is a good thing, but you would have to start with something.

People are complaining about the length of time it takes to get anything published. Adding these extra steps of "protocol quality review", "applicability review" etc. will only make that process even longer.

Yoav

On Oct 28, 2010, at 3:57 AM, Keith Moore wrote:

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 28 Oct 2010 07:36:29 -0400

> People are complaining about the length of time it takes to get anything published. Adding these extra steps of "protocol quality review", "applicability review" etc. will only make that process ev…

Much of this is already being done as part of internal review by IESG and the document shepherd.  So no, merely assigning these labels shouldn't make the process longer.

Actually (based on my long ago experience in IESG, which might or might not be informative of the situation today) part of the reason the current process may take so long might be that IESG is essentially trying to shoehorn a number of different concerns into a binary decision of whether something should be Proposed or not - and that decision often involves a lot of angst.  (e.g. when a WG has labored for years to produce a protocol which describes a good idea, but describes it poorly.)   Maybe the IESG could instead do its review, and assign values to these various labels.  If it doesn't think the document yet meets the threshold criteria for standards track, it could offer the WG or author the option to publish as Informational - but still with these labels.  That way, readers could get a sense of the relative quality of the document and protocol and have some improved basis with which to judge whether they should implement it and whether it would work well for them.

Or - here's an idea - what if all that were holding up a document were stale references or normative references to non-standard documents?   The review could come back with Protocol-Quality=good, Applicability=wide, Document-Quality=good, Completeness=incomplete (with the reason given as: references to nonstandard material).  It might still not yet be labeled as standards-track but implementors would know that the document had generally met with favorable reviews.  

Keith

---

From: "t.petch" <[email-masked]>
Date: Fri, 29 Oct 2010 10:54:53 +0200

As an engineer, I do like to know what problem I am required to solve before
proposing a solution:-)  My reading of this thread is that the problem is the
length of time it takes to produce an RFC of any kind, that vendors are off to
the races at the fifth or tenth version of an I-D stage because the market will
not wait for the machinations of the IETF (and as a consequence, to most people,
the differences between PS, DS and FS are irrelevant).

As such, draft-housley-two-maturity-levels seems an irrelevance, a distraction
from the issues facing the IETF.  If there is a problem worth solving in the
space occupied by draft-housley-two-maturity-levels, then Keith's proposal for
an orthogonal set of measurements of document quality seem far more apt.

By contrast, the delays in producing an RFC seem to revolve around WG process,
where Last Call causes people to come out of the woodwork with delaying
suggestions, something a good chair or AD would stamp on, and IESG process,
where certain hot buttons - eg security, flow control - produce some ludicrous
DISCUSS' which delay the process for months.
Tom Petch

----- Original Message -----
From: "Keith Moore" <[email-masked]>
To: "Yoav Nir" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, October 28, 2010 3:57 AM
Subject: Re: what is the problem bis

On Oct 27, 2010, at 8:58 AM, Yoav Nir wrote:

> This comes back to the question or why have maturity levels at all. Ideally,
an implementer should prefer to implement a mature standard over a less-mature
one. In practice, adopting the more advanced standard may give you an obsolete
protocol, rather than a more stable one. IOW the standardization level of a
document does not give a potential implementer any signal as to whether or not
this standard is in any sense of the word "good".

Mostly agree.  The distinction between an older, well-tested, widely-deployed
version of a protocol vs. a newer less-tested, less-widely-deployed version with
more features is a useful distinction to make.  but the difference between (RFC
X, full) and (RFC Y, proposed) where Y >> X only conveys the barest hint of
that, and even less in the way of guidance.  I'm thinking specifically of email
standards here, where in practice you need to be able to accept RFC 822 messages
(because even new mail readers have to be able to deal with old messages) but
you should generate messages that conform to 5322 and MIME.

Protocol-Quality.  We need a statement about the perceived quality of the
protocol described in the document.   (Is this protocol well-designed for the
anticipated use cases, or does it have significant flaws (including security
flaws)?)
Applicability.  We need a statement about the current applicability of the
protocol described in the document.  (Is this protocol recommended for general
use, not recommended except in specific corner cases, not recommended at all, or
strongly discouraged?)
Document-Quality.  We need a statement about the perceived quality of the
document itself and whether the protocol description seems to be sufficiently
precise to permit implementations to interoperate.   (along with a pointer to
errata.)
Maturity.  We need a statement about the amount of actual implementation and
deployment experience that the protocol enjoys.
Completeness.  We need a statement about how accurately the document reflects
what is currently believed to be good practice for implementation/use of that
protocol, or whether effective implementation requires information not included
or referenced in the document.  (e.g. effective implementation of SMTP generally
requires some expertise in dealing with heavy loads caused by spam, looping, and
denial-of-service attacks which aren't really dealt with in any of the relevant
RFCs).
Last-Review-Date.  Date of the last review of these labels for this document.

These would go alongside the existing Updates and Obsoletes labels.  An
Applicability-Statement could also be included.

It strikes me that we could establish such a set of labels on an experimental
basis, using some sort of community review process for existing RFCs, without
making any immediate changes to our proposed/draft/full system of labeling of
standards.  IESG could assign the initial labels for new RFCs - the document
reviewers are almost doing that anyway.  The existing errata process could be
extended to allow (moderated) user comments on these labels, and the labels
could be subject to periodic review based on those comments.

If that labeling system turned out to be adequate or could be fixed with some
tweaking, we could maybe drop back to two document classes:  Informational, and
Standards-Track.  Standards-track would encompass any  former proposed, draft,
or full standard which was still in use and for which periodic reviews were
still being done.   Former Experimental documents would be reclassified as
Informational with appropriate descriptive labels.  Former Historic documents
would be reclassified as Informational with Applicability set to Historic.
Standards-track documents would expected to have periodic review of these
labels; Informational documents could have some of those labels set to
"Undetermined".

Keith

--------------------------------------------------------------------------------

---

From: Keith Moore <[email-masked]>
Date: Fri, 29 Oct 2010 08:13:56 -0400

> DISCUSS' which delay the process for months.

What I get from this is that the entire community needs to be involved in review and input of future RFCs much earlier in the process than Last Call.  In my experience, the problem is generally not that the people "coming out of the woodwork" are making irrelevant suggestions, because IESG is fairly good at ignoring these.  The problem seems to be that by the time a document as reached Last Call, the working group is past the point where it can meaningfully consider input from outside for anything but the most trivial changes - and the problems identified in Last Call are often much more fundamental than that.   Often in my experience, outside reviewers and IESG members have been compelled to try to suggest minor wording changes to address what they saw as fundamental problems of architecture or scope in the proposed protocol.  

And yet, there are far too many Internet-Drafts published for the community to notice and consistently provide early review of every proposal, and far too many WGs even in a particular subject area to permit most community members to keep track of those documents and WGs with which they might be concerned.

I'm not sure exactly where this leads, but I'm sort of thinking it might be nice if there were a "First Call" for community review much earlier in a document's life cycle.   The I-D associated with a First Call should outline the solution that is being proposed, and all of the major considerations (e.g. security) should be dealt with - though perhaps not specified in detail.   There wouldn't be any expectation that the document should be polished, that every aspect of the protocol being proposed should be nailed down, every option defined, every reference included or current, and so forth.  The First Call I-D would not be published as an RFC, unless perhaps the WG died prematurely and there were a desire to publish it as Informational.  It's just an I-D for which the WG requests wide community review.

The WG would be expected to take First Call comments seriously and to report to the responsible AD how it was considering First Call comments into account in the development of its protocol.  The tradeoff is that WGs would not be expected to deal with major structural/scope challenges at Last Call - provided, of course that they hadn't changed the structure/scope drastically from First Call.

Obviously this is not even half-baked yet.  Biggest problem I see is that the community would initially have no idea what level of detail should be specified by First Call.  Some well-written (but incomplete!) examples, perhaps of imagined First Calls for well-established protocols, or early Internet-Drafts for protocols that were eventually standardized, might help.   There are lots of other potential problems also.  But in general I think the idea of getting earlier community feedback is a sound one.

Keith

---

From: John E Drake <[email-masked]>
Date: Fri, 29 Oct 2010 05:32:47 -0700

An I-D usually needs to get WG consensus before it becomes a WG draft.  Getting consensus from the wider community at this point, as you suggest, seems very reasonable.  In my experience, once a document is issued as an RFC, it is considered to be a standard.  The steps beyond that are largely irrelevant.

Sent from my iPhone

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Keith Moore
> Sent: Friday, October 29, 2010 5:14 AM
> To: t.petch
> Cc: [email-masked]
> Subject: what is the problem ter
> 
> 
> What I get from this is that the entire community needs to be involved
> in review and input of future RFCs much earlier in the process than
> Last Call.  In my experience, the problem is generally not that the
> people "coming out of the woodwork" are making irrelevant suggestions,
> because IESG is fairly good at ignoring these.  The problem seems to be
> that by the time a document as reached Last Call, the working group is
> past the point where it can meaningfully consider input from outside
> for anything but the most trivial changes - and the problems identified
> in Last Call are often much more fundamental than that.   Often in my
> experience, outside reviewers and IESG members have been compelled to
> try to suggest minor wording changes to address what they saw as
> fundamental problems of architecture or scope in the proposed protocol.
> 
> And yet, there are far too many Internet-Drafts published for the
> community to notice and consistently provide early review of every
> proposal, and far too many WGs even in a particular subject area to
> permit most community members to keep track of those documents and WGs
> with which they might be concerned.
> 
> I'm not sure exactly where this leads, but I'm sort of thinking it
> might be nice if there were a "First Call" for community review much
> earlier in a document's life cycle.   The I-D associated with a First
> Call should outline the solution that is being proposed, and all of the
> major considerations (e.g. security) should be dealt with - though
> perhaps not specified in detail.   There wouldn't be any expectation
> that the document should be polished, that every aspect of the protocol
> being proposed should be nailed down, every option defined, every
> reference included or current, and so forth.  The First Call I-D would
> not be published as an RFC, unless perhaps the WG died prematurely and
> there were a desire to publish it as Informational.  It's just an I-D
> for which the WG requests wide community review.
> 
> The WG would be expected to take First Call comments seriously and to
> report to the responsible AD how it was considering First Call comments
> into account in the development of its protocol.  The tradeoff is that
> WGs would not be expected to deal with major structural/scope
> challenges at Last Call - provided, of course that they hadn't changed
> the structure/scope drastically from First Call.
> 
> Obviously this is not even half-baked yet.  Biggest problem I see is
> that the community would initially have no idea what level of detail
> should be specified by First Call.  Some well-written (but incomplete!)
> examples, perhaps of imagined First Calls for well-established
> protocols, or early Internet-Drafts for protocols that were eventually
> standardized, might help.   There are lots of other potential problems
> also.  But in general I think the idea of getting earlier community
> feedback is a sound one.
> 
> Keith
>

---

From: Martin Rex <[email-masked]>
Date: Fri, 29 Oct 2010 17:58:36 +0200 (MEST)

> > By contrast, the delays in producing an RFC seem to revolve around

I'm convinced the problem is primarily in the WG processing of the
document, _not_ in the IESG review and neither in the 3 maturity levels.

The current maturity levels "Proposed", "Draft" and "Standard" are MORE
about document quality than protocol usefulness and popularity -- and
they IMHO accurately reflect (lack of) document quality in most cases.

> in Last Call are often much more fundamental than that.

Maybe it would help if the early I-Ds would be more explicit in reflecting
the document status and state of the discussion, for each and
every section and subsection -- so that reviewers get a better picture
about what level of feedback the WG is looking for.

It might also help if the early document would try to get the solution
architecture document accurately before fleshing out the details of
the implementation.  A lot of proposals start out with the implementation
details of the solution and little about the solution architecture.

IMHO, Design flaws that require non-trivial changes are *MUCH* easier
to notice and to discuss at the abstract solution architecture level.
The reason why the TLS renegotiation flaw went unnoticed for so many
years was that noone spent two hours on describing the architecture
of the TLS renegotiation. 

-Martin

---

From: Hadriel Kaplan <[email-masked]>
Date: Fri, 29 Oct 2010 12:20:46 -0400

On Oct 27, 2010, at 9:57 PM, Keith Moore wrote:

The problem with a labeling scheme like this is it's subjective.  "Updates" and "Obsoletes" are not subjective, and to determine whether to apply those two labels is fairly easy.  Labels of the form *-Quality would cause massive debate and need a WG-wide or larger consensus agreement process, if you mean them to be formal labels.

For some of what you describe above, people already produce RFCs to document - RFCs which deprecate a previous RFC, or enumerate issues found, or BCPs, etc.  And obviously errata are already captured by the RFC editor.

What's missing I think is some way to remind/force readers of an RFC to check for errata and updating/obsoleting RFCs, and how.  Since it's a static document when published, I think the most natural way would be to add to the boilerplate a sentence reminding the reader to check for updates, obsoletes, and errata at "http://datatracker.ietf.org/doc/rfcXXXX" where XXXX is filled in by the RFC Editor upon RFC publication.

Or if you want to be really fancy, you could have the IETF auto-create a Wiki type page for each RFC, that allows open community wiki-input about quality and implementation/deployment experience and such, with a big banner indicating the wiki content is not an official IETF position.

-hadriel

---

From: Michael Richardson <[email-masked]>
Date: Fri, 29 Oct 2010 12:36:19 -0400

>>>>> "t" == t petch <t.petch> writes:
    t> By contrast, the delays in producing an RFC seem to revolve
    t> around WG process, where Last Call causes people to come out of
    t> the woodwork with delaying suggestions, something a good chair or
    t> AD would stamp on, and IESG process, where certain hot buttons -
    t> eg security, flow control - produce some ludicrous DISCUSS' which
    t> delay the process for months.  Tom Petch

I think that's unfair, both to the process and to the people involved.
LCs are there because ADs and Chairs often *DO* stamp on the legitimate
concerns of people, which often get ignored in the process.  

In-person meeting time is used regularly for powerpoints rather than
discussion.   This is, I think, a major difference between the IETF
meeting of 2010 and an IETF meeting of 1992.  In 1992, maybe, I think we
had much less access to AV equipment, and thus we actually had to use
our time productively.  Of course, we also didn't have the RFC2026
process at all.  I don't really know: my first meeting was March
1996, although I recall bringing some of my thoughts in the form of
overhead foils (transparencies).

Even the setup of the room matters: the theatre style of seating means
that things are prejudiced towards presenters/listeners, rather than
participation of equals.  
Interium meetings by phone only seem to sometimes be more useful: only
really involved people attend, and everyone is equal disadvantages by
the limited single channel voice channel.

So, it's not that WG Chairs need to stamp on these people, it's that
they need to STOP stamping on them.  Instead, they need to be understood
early on in the process, and their concerns either addressed, or they
need to be made out of scope, at least for that stage of the work.

If you want to remove a roadblock to PS, then I'd say remove the LC.
Move it to from PS->step2.

---

From: Keith Moore <[email-masked]>
Date: Fri, 29 Oct 2010 14:37:11 -0400

On Oct 29, 2010, at 12:36 PM, Michael Richardson wrote:

> In-person meeting time is used regularly for powerpoints rather than discussion.  

+1.  

The single biggest thing that IETF could do to raise productivity in meetings is to remove video projectors from meeting rooms, replace them with white boards and pens, and ban use of PowerPoint and similar tools.
The second biggest thing that IETF could do to raise productivity in meetings is to ban Internet use in meetings except for the purpose of remote participation.

Keith

---

From: "Richard L. Barnes" <[email-masked]>
Date: Fri, 29 Oct 2010 15:24:37 -0400

It kind of seems like we're thinking of this in a 20th-century,  
politburo sort of way.  "We, the illumnati, will decide whether this  
document is awesome."  Could we not just use RFC as a basic threshold  
of quality, then let the community provide open and ongoing feedback?   
Like, with voting buttons at the top of the HTML-formatted RFCs, or  
maybe a comment section at the bottom?

--Richard

On Oct 29, 2010, at 12:20 PM, Hadriel Kaplan wrote:

>> Protocol-Quality.  We need a statement about the perceived quality  
>> of the protocol described in the document.   (Is this protocol well- 
>> designed for the anticipated use cases, or does it have significant  
>> flaws (including security flaws)?)
>> Applicability.  We need a statement about the current applicability  
>> of the protocol described in the document.  (Is this protocol  
>> recommended for general use, not recommended except in specific  
>> corner cases, not recommended at all, or strongly discouraged?)
>> Document-Quality.  We need a statement about the perceived quality  
>> of the document itself and whether the protocol description seems  
>> to be sufficiently precise to permit implementations to  
>> interoperate.   (along with a pointer to errata.)
>> Maturity.  We need a statement about the amount of actual  
>> implementation and deployment experience that the protocol enjoys.
>> Completeness.  We need a statement about how accurately the  
>> document reflects what is currently believed to be good practice  
>> for implementation/use of that protocol, or whether effective  
>> implementation requires information not included or referenced in  
>> the document.  (e.g. effective implementation of SMTP generally  
>> requires some expertise in dealing with heavy loads caused by spam,  
>> looping, and denial-of-service attacks which aren't really dealt  
>> with in any of the relevant RFCs).
>> These would go alongside the existing Updates and Obsoletes  
>> labels.  An Applicability-Statement could also be included.
>>
>
> The problem with a labeling scheme like this is it's subjective.   
> "Updates" and "Obsoletes" are not subjective, and to determine  
> whether to apply those two labels is fairly easy.  Labels of the  
> form *-Quality would cause massive debate and need a WG-wide or  
> larger consensus agreement process, if you mean them to be formal  
> labels.
>
> For some of what you describe above, people already produce RFCs to  
> document - RFCs which deprecate a previous RFC, or enumerate issues  
> found, or BCPs, etc.  And obviously errata are already captured by  
> the RFC editor.
>
> What's missing I think is some way to remind/force readers of an RFC  
> to check for errata and updating/obsoleting RFCs, and how.  Since  
> it's a static document when published, I think the most natural way  
> would be to add to the boilerplate a sentence reminding the reader  
> to check for updates, obsoletes, and errata at "http://datatracker.ietf.org/doc/rfcXXXX 
> " where XXXX is filled in by the RFC Editor upon RFC publication.
>
> Or if you want to be really fancy, you could have the IETF auto- 
> create a Wiki type page for each RFC, that allows open community  
> wiki-input about quality and implementation/deployment experience  
> and such, with a big banner indicating the wiki content is not an  
> official IETF position.
>
> -hadriel
>
>
>

---

From: Hadriel Kaplan <[email-masked]>
Date: Fri, 29 Oct 2010 16:06:40 -0400

>> In-person meeting time is used regularly for powerpoints rather than discussion.  

I've been yelled-at in WG meetings for using the microphone meeting time for discussion, vs. position statements or questions+answers about the slides.
The complainer stated the WG mailing list was for discussion, not the WG f2f meeting. (meanwhile I've gotten complaints about using the mailing lists for discussion, too - some people are never happy)

In fairness, though, there isn't enough time in WG meetings for real discussion on each of the numerous drafts.  I've found virtual interim meetings work really well for that.

> The single biggest thing that IETF could do to raise productivity in meetings is to remove video projectors from meeting rooms, replace them with white boards and pens, and ban use of PowerPoint an…

It's hard to get remote participants to see physical whiteboards, nor to view them in advance. 

> The second biggest thing that IETF could do to raise productivity in meetings is to ban Internet use in meetings except for the purpose of remote participation.

But then how could we read the drafts while they're being presented? ;)

-hadriel

---

From: "Richard L. Barnes" <[email-masked]>
Date: Fri, 29 Oct 2010 16:14:53 -0400

>> The single biggest thing that IETF could do to raise productivity  
>> in meetings is to remove video projectors from meeting rooms,  
>> replace them with white boards and pens, and ban use of PowerPoint  
>> and similar tools.
>

Advance availability of slides can also be helpful for the growing  
fraction of participants for whom English is not a native language.

--Richard

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 29 Oct 2010 13:23:10 -0700

On 10/29/2010 1:14 PM, Richard L. Barnes wrote:
> Advance availability of slides can also be helpful for the growing fraction of
> participants for whom English is not a native language.

+1

Also, the style of the slides makes a difference.

I was originally taught to produce very cryptic slides that have only a few key 
words per bullet.  These work well as a visual reference for a native-language 
audience with a speaker providing the actual content.

For English as a second language audiences, a more verbose form is more helpful. 
  The trick, of course, is to still be relatively terse.  So, whole readable 
thoughts, but not extended prose.

This also helps when the slides are circulated without the speaker...

d/
-- 

   Dave Crocker

---

From: Keith Moore <[email-masked]>
Date: Fri, 29 Oct 2010 17:47:18 -0400

On Oct 29, 2010, at 4:05 PM, John C Klensin wrote:

> 
> 
> --On Friday, October 29, 2010 12:20 -0400 Hadriel Kaplan
> <[email-masked]> wrote:
> 
>> 
>> Protocol-Quality.  We need a statement about the perceived
>> quality of the protocol described in the document.   (Is this
>> protocol well-designed for the anticipated use cases, or does
>> it have significant flaws (including security flaws)?)
>> Applicability.  We need a statement about the current
>> applicability of the protocol described in the document.  (Is
>> ...
> 
> Hi.  
> 
> It is difficult to imagine how these sorts of idea would work in
> conjunction with RFCs given that those are explicitly archival,
> never-changing documents and your suggestions seem to imply
> evolving classification and comment systems.

well, of course those classifications wouldn't be in the RFCs themselves (any more than our current maturity levels are).
and it's quite natural that the various measures of quality/relevance/maturity of an RFC will fluctuate over time.

> However, a number of similar ideas --including effectively
> replacing Standards-Track Maturity levels with more descriptive
> text and finer-grained comments were incorporated into a
> proposal to the NEWTRK WG in 2005-2006.   I think it is safe to
> suggest that the reasons why the proposal never went anywhere
> remain controversial, but you might find it interesting
> recreational reading:
> 
> https://datatracker.ietf.org/doc/draft-ietf-newtrk-repurposing-isd/

thanks for the pointer; I'll take a look.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 29 Oct 2010 18:04:44 -0400

On Oct 29, 2010, at 4:14 PM, Richard L. Barnes wrote:

> Advance availability of slides can also be helpful for the growing fraction of participants for whom English is not a native language.

If there's a desire for more than about ten "slides" per meeting, there's something seriously wrong with the meeting agenda.

But I do acknowledge that PowerPoint is a good tool for constructing outlines.  So what people should do is take those slides, add comments to them, send the slides (in PDF) to the mailing list well before the meeting, and expect meeting attendees to have read them before the meeting.  The slides should not be shown at the meeting, except perhaps in the rare case that one of them contains a complex drawing that needs to be referred to.  (even then, it's pretty hard to make a decent drawing with PowerPoint.)

Keith

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 29 Oct 2010 18:54:41 -0400

I am finding this discussion difficult to parse.

Here we have a post that says 'I can't understand the purpose of this
proposal', then suggests solving a completely different problem and declares
the proposal irrelevant.

There are multiple reasons for the two step proposal, we do not all have to
agree on them all. But it should be fairly obvious that the problem two
track is intended to address is the number of times a document is required
to go through the RFC process. It is not an attempt to address the length of
time the process takes.

I have yet to see a single poster state that they think that the three step
process is working. On the other hand we are managing to issue RFCs.

On Fri, Oct 29, 2010 at 4:54 AM, t.petch <[email-masked]> wrote:

> > And if it doesn't signal anything to the "customers" of the documents,
> what's
> the point of having these levels at all?
>
> looping, and

---

From: "Glen Zorn" <[email-masked]>
Date: Sat, 30 Oct 2010 15:01:17 +0700

Keith Moore [mailto://[email-masked]] writes:

> discussion.
> The single biggest thing that IETF could do to raise productivity in
> meetings is to remove video projectors from meeting rooms, replace them
> with white boards and pens, and ban use of PowerPoint and similar tools.

I couldn't agree more.  It's easy, too, just requiring a little bit of
backbone...

> The second biggest thing that IETF could do to raise productivity in
> meetings is to ban Internet use in meetings except for the purpose of
> remote participation.

Harder to do & not clearly an improvement: it clear out meeting rooms a bit,
but on the other hand people who (for example) just read email in meetings
aren't really harming productivity too much.

---

From: Keith Moore <[email-masked]>
Date: Sat, 30 Oct 2010 07:34:36 -0400

> aren't really harming productivity too much.

I'm not sure about that.   If you're in a room with ten people who are participating in a discussion, it's easy to know whether those ten have achieved consensus among themselves.  Also, chances are good that each of those ten people has had a chance to ask questions, voice objections, or otherwise make contributions to the discussion.    

But if you're in a room with a hundred people (mostly staring at laptops) and only ten active participants, it's much harder to know whether there is consensus in the room.  And because there are so many people not obviously doing anything, those who have something to say are more likely to feel inhibited.  After all, most people are saying nothing (and not paying much attention), and we humans (okay, most of us) tend to take cues for what is socially acceptable by watching the behavior of those around us.

In the early-to-mid 1990s, IETF WG meetings used to be good places to actually discuss concerns about a document, and hash out potential solutions.  I remember several occasions when a WG would schedule two meeting sessions in a week, one on Monday and another on late Wednesday or Thursday.  The Monday session would discuss the document(s) on the table, identify problems, suggest solutions.  Then a couple of WG participants and the authors would sit up late one night and revise the document in time for review at the second meeting (or at least, to be able to report to the second meeting what changes they had made, and get feedback on those).   I think it led to much faster convergence than what we usually see now.  And often the face-to-face review/revise/review sessions resulted in getting the document in a state where there were only a few nits remaining.  I don't think this would work the way we have meetings now, because there's nowhere nearly enough time for discussion.

Keith

---

From: Joel Jaeggli <[email-masked]>
Date: Sat, 30 Oct 2010 09:29:03 -0400

This discussion has a periodicy about 6 months. The premise is asinine, we can't go back to the early to mid 90s. 

Joel's widget number 2

On Oct 30, 2010, at 7:34, Keith Moore <[email-masked]> wrote:

> In the early-to-mid 1990s, IETF WG meetings used to be good places to actually discuss concerns about a document, and hash out potential solutions.  I remember several occasions.when a WG would schedule two meeting sessions in a week, one on Monday and another on late Wednesday or Thursday.  The Monday session would discuss the document(s) on the table, identify problems, suggest solutions.  Then a couple of WG participants and the authors would sit up late one night and revise the document in time for review at the second meeting (or at least, to be able to report to the second meeting what changes they had made, and get feedback on those).   I think it led to much faster convergence than what we usually see now.  And often the face-to-face review/revise/review sessions resulted in getting the document in a state where there were only a few nits remaining.  I don't think this would work the way we have meetings now, because there's nowhere nearly enough time for discussion.
> 
> Keith
>

---

From: Yoav Nir <[email-masked]>
Date: Sat, 30 Oct 2010 18:25:55 +0200

> aren't really harming productivity too much.

They do by skewing statistics. ADs gauge WG position by the feel of the room. 100 people in the room make it look like there's a lot of interest, when it fact only the document authors and one of the WG chairs have read the drafts. Same for hums. Who knows what drives someone staring at a laptop to hum one way or another?

As an example, the IPsecME meeting in Maastricht had over 100 people in the room. Most never looked up from their laptop screens.

---

From: Joel Jaeggli <[email-masked]>
Date: Sat, 30 Oct 2010 09:38:12 -0700

On 10/30/10 9:25 AM, Yoav Nir wrote:
>>> The second biggest thing that IETF could do to raise productivity
>>> in meetings is to ban Internet use in meetings except for the
>>> purpose of remote participation.
>> 
>> Harder to do & not clearly an improvement: it clear out meeting
>> rooms a bit, but on the other hand people who (for example) just
>> read email in meetings aren't really harming productivity too
>> much.
> 
> They do by skewing statistics. ADs gauge WG position by the feel of
> the room. 100 people in the room make it look like there's a lot of
> interest, when it fact only the document authors and one of the WG
> chairs have read the drafts. Same for hums. Who knows what drives
> someone staring at a laptop to hum one way or another?

the final arbiter of any test in the room is on the mailing list.
disagreement if it is exists is trivially easy measure... ambivalence is
harder to measure. in the ipsec case there are potentially quite a lot
of people who aren't as concerned about the outcome as they are there is
one and that they're going to need to implement.

if the filter you apply is that only those with strong opinions should
participate that's going to drive outcomes to the margins.

> As an example, the IPsecME meeting in Maastricht had over 100 people
> in the room. Most never looked up from their laptop screens.
> 
> _______________________________________________ Ietf mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Keith Moore <[email-masked]>
Date: Sat, 30 Oct 2010 19:30:13 -0400

> This discussion has a periodicy about 6 months. The premise is asinine, we can't go back to the early to mid 90s. 

What's asinine is to dismiss out-of-hand something has worked well in the past.

The only reason we can't change the way we have discussions is that too many people are in the habit of thinking we have to do things the way we are doing them.  To anyone with an engineering background, that's not a reason.  (and what are the rest of you doing here anyway?)  :)

We had serious scaling problems with IETF in the late 1990s and we had to adapt because of the huge number of attendees we had.  But attendance is way down since then.  

Powerpoint inhibits thinking.  As it's nearly always used, it inhibits discussion.  It makes IETF meetings very expensive, because they tremendously lower the value returned in exchange for the money invested to get people there.  People who actually want to get work done have less incentive to attend IETF meetings than they used to.  So the people who attend IETF meetings these days are less likely to be those with the best technical talent, and more likely to be "goers" who are just there to tout the company line and/or to watch what other people are doing. 

The amount of harm caused by IETF's overuse of powerpoint is huge.

Keith

---

From: Keith Moore <[email-masked]>
Date: Sat, 30 Oct 2010 19:34:38 -0400

> the final arbiter of any test in the room is on the mailing list.

True.  But a room with a high ratio of active participants to total attendees makes a much better sounding board for providing constructive feedback, than a room with a low ratio.  You get convergence on a sound proposal more quickly if you can get good feedback from face to face meetings.  And (provided the attendees at such meetings represent a decent cross-section of interests), proposals that have enjoyed such feedback are more likely to be enthusiastically accepted by a mailing list.

Keith

---

From: Julian Reschke <[email-masked]>
Date: Sun, 31 Oct 2010 10:27:43 +0100

On 29.10.2010 18:20, Hadriel Kaplan wrote:
> "Updates" and "Obsoletes" are not subjective, and to determine whether
> to apply those two labels is fairly easy. Labels of the form *-Quality
> would cause massive debate and need a WG-wide or larger consensus
> agreement process, if you mean them to be formal labels.
> ...

It's only sometimes easy. It get's really hard when the "old" RFC 
combines too many different things (like a new HTTP method, one 
application for it, and a registry because previously there was none), 
or when the new spec is actually a set of RFCs.

> found, or BCPs, etc. And obviously errata are already captured by the
> RFC editor.
>
> What's missing I think is some way to remind/force readers of an RFC to
> check for errata and updating/obsoleting RFCs, and how. Since it's a
> static document when published, I think the most natural way would be to
> add to the boilerplate a sentence reminding the reader to check for
> updates, obsoletes, and errata at
> "http://datatracker.ietf.org/doc/rfcXXXX" where XXXX is filled in by the
> RFC Editor upon RFC publication.

All RFCs published since last January have that pointer in the 
boilerplate, it goes to

   http://www.rfc-editor.org/info/rfcNNNN

(now if we published specs in (X)HTML, people might actually click on it)

> Or if you want to be really fancy, you could have the IETF auto-create a
> Wiki type page for each RFC, that allows open community wiki-input about
> quality and implementation/deployment experience and such, with a big
> banner indicating the wiki content is not an official IETF position.

That might be a good thing to have.

Best regards, Julian

---

From: Hadriel Kaplan <[email-masked]>
Date: Sun, 31 Oct 2010 11:51:26 -0400

>   http://www.rfc-editor.org/info/rfcNNNN

Sweet. :)
I actually checked the boilerplate I use for I-D's before saying it, but I guess my boilerplate is old.

-hadriel

---

From: Julian Reschke <[email-masked]>
Date: Sun, 31 Oct 2010 16:54:48 +0100

> I actually checked the boilerplate I use for I-D's before saying it, but I guess my boilerplate is old.

Oh, it's only in RFCs...

Best regards, Julian