Subject: Re: draft-housley-two-maturity-levels

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 1 Oct 2010 23:07:50 -0400

Back when the 3 step process was originally proposed, part of the rationale
for a low barrier to entry step one seems to have been the fact that in some
cases it is necessary to have code points assigned by IANA before work
actually gets done.

While there are some code points that are sufficiently scarce that a
standards action remains the appropriate criteria, there are really very few
for which that is true.

At this point there is no value to three steps because the requirements for
passing the first step are now much higher than they used to be are
 arguably higher than they used to be for the second step.

That creates something of a potential circle in which to get a draft ready
for PROPOSED Standard there is really no substitute for real world
deployment and in some cases that is not really viable until after you have
the code points.

I suggest that WGs and proto-WGs be encouraged to make use of the
EXPERIMENTAL and INFORMATIONAL status drafts for code point assignment in
the case that this is necessary to gain real world experience and that this
be recognized as an appropriate use of these stages.

---

From: Russ Housley <[email-masked]>
Date: Mon, 25 Oct 2010 14:36:38 -0400

I'd like to hear from the community about pushing forward with this
proposal or dropping it.

At least one other proposal was raised.  My reading of this mail list is
that the proposal in draft-housley-two-maturity-levels has more support.

Since the -00 cut-off for IETF 79 has passed, I am assuming that no
other proposals are going to be raised.

Should I be seeking a sponsor for this draft?

Russ

---

From: Barry Leiba <[email-masked]>
Date: Mon, 25 Oct 2010 20:22:54 -0400

> I'd like to hear from the community about pushing forward with this

I see disagreement with the proposal, but we'll see disagreement with
any proposal.  I see enough support to continue pursuing it, in my
opinion, and trying to find a balancing point that enough of us can
agree with.

I'd like to see work continue.

Barry

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 Oct 2010 13:35:23 +1300

> opinion, and trying to find a balancing point that enough of us can

I am happy to agree to what the draft currently says. We've sliced
and diced this many times over the years, and this seems very close to the
least-unpopular view. That's the best we can hope for, imho.

    Brian

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Eric Burger <[email-masked]>
Date: Mon, 25 Oct 2010 20:37:57 -0400

Ever since Proposed Standard became Draft Standard for all practical purposes, might as well make it official.

---

From: Keith Moore <[email-masked]>
Date: Mon, 25 Oct 2010 22:30:50 -0400

I'm against it.  I think it's a step in the wrong direction.

Keith

---

From: John Levine <[email-masked]>
Date: 26 Oct 2010 02:44:00 -0000

>least-unpopular view. That's the best we can hope for, imho.

+1

R's,
John

---

From: "James M. Polk" <[email-masked]>
Date: Mon, 25 Oct 2010 22:37:03 -0500

> >least-unpopular view. That's the best we can hope for, imho.

I'm not in love with the 3 maturity levels, especially when I was 
asked by an AD during Maastricht to provide proof of 2 independent 
implementations just to have an ID I was presenting be considered to 
become a WG item.

That bar is just WAY too high.

That said, I think the only part I'm concerned about with your 
proposal is allowing Internet Standards to reference Proposed 
Standards. Given that they can change so much - or more likely - they 
can have parts of them that just aren't ever implemented, but still 
have one or more of these un-implemented parts that is a critical to 
the Internet Standard.

I guess if this clears the logjam of all the other issues, I'm 
willing to agree to this.

James

---

From: Ross Callon <[email-masked]>
Date: Mon, 25 Oct 2010 23:37:51 -0400

I support the draft in its current form. IMHO moving to two maturity levels is a significant step in the right direction. 

Thanks, Ross

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: Monday, October 25, 2010 8:35 PM
To: Barry Leiba
Cc: [email-masked]
Subject: Re: draft-housley-two-maturity-levels

    Brian

---

From: Michael Richardson <[email-masked]>
Date: Tue, 26 Oct 2010 09:09:55 -0400

exec summary, but comments inline: I am in favour of two maturity levels.

>>>>> "James" == James M Polk <[email-masked]> writes:
    >> >least-unpopular view. That's the best we can hope for, imho.

    James> I'm not in love with the 3 maturity levels, especially when I was asked by an
    James> AD during Maastricht to provide proof of 2 independent implementations just
    James> to have an ID I was presenting be considered to become a WG
    James> item.

But, that's just wrong, regardless of 2 or 3 maturity levels.
This AD has just created a 5 maturity level system! (ID without interop,
ID with interop, PS, DS, FS).   Do they really know what they asked for?
This is perhaps the fundamental problem.

    James> That bar is just WAY too high.

    James> That said, I think the only part I'm concerned about with your proposal is
    James> allowing Internet Standards to reference Proposed Standards. Given that they
    James> can change so much - or more likely - they can have parts of them that just
    James> aren't ever implemented, but still have one or more of these un-implemented
    James> parts that is a critical to the Internet Standard.

    James> I guess if this clears the logjam of all the other issues, I'm willing to
    James> agree to this.

I agree with you, but I think we can work this out.

I am in favour of two maturity levels.

(see I can top-post and inline post at the same time!)

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 26 Oct 2010 12:13:19 -0400

On Mon, Oct 25, 2010 at 10:37:03PM -0500, James M. Polk wrote:
> I'm not in love with the 3 maturity levels, especially when I was asked 
> by an AD during Maastricht to provide proof of 2 independent  

I agree, but I see precious little evidence so far presented that the
two-maturity-levels proposal under discussion will solve that problem.

If the problem we need to solve is that it's too hard to get an ID
published as Proposed Standard, then we should tackle that.  The
problem in that case does not need tweaking of standards levels and so
on, but hard rebukes of WGs (and, frankly, IESG members) who insist on
making that publication hard.

If the problem is that things linger in the standards track without
progressing, then _maybe_ reducing the number of maturity levels
helps, because it might solve a public relations problem the IETF has
(i.e. if the real problem of lingering is that the IETF has a
resulting PR problem).

The draft as it stands claims that the reason the bar is so high for
PS is because things linger there.  That is an incentive to make the
resulting RFC as good as possible because it will be the only document
ever published.  This amounts to a causal argument about how things
got to be the way they are.  I am a little sceptical that this story
is true.  (One can tell just as compelling a story in the other
direction: once the PS bar was high enough, there was no incentive to
revisit an RFC and try to advance it.)  But suppose the
two-maturity-levels draft's story were right: as the draft stands, it
will do absolutely nothing to solve that problem.

If things are currently languishing as PS in a three-track system,
there is no reason to suppose that removing one (later) level of
maturity will do anything about those documents that never advance
beyond PS.  If documents do not so advance, then the incentives to
make publication as PS hard remain exactly the same.  We should
therefore expect the same result, which is that things linger at PS
and WGs and the IESG make it hard to publish that initial, PS-level
RFC.

In fact, the draft as it stands will actually make that incentive
stronger, because it explicitly permits downrefs from STANDARD to
PROPOSED STANDARD.  I get the reasoning, but one likely effect of this
is going to be that there is even more pressure to get the PS
documents close to perfect.  Even if there are significant problems
with a given PS, if there are downrefs to it there will be a lot of
pressure to avoid changing it substantively, because of those
references, which means that revisiting PS RFCs won't pay off.  

Because of the above, I don't think the proposal is a good idea in
principle.  As a matter of fact, however, I don't think its adoption
would make much practical difference: the RFCs that manage to clear
the initial high bar will mostly continue to stick at PS for want of
energy to move them along the track.  Sometimes, some people will have
the energy to move things along, and then they will.

Best regards,

Andrew

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Martin Rex <[email-masked]>
Date: Tue, 26 Oct 2010 19:05:42 +0200 (MEST)

> That bar is just WAY too high.

Were you document author or WG chair?

I think that the problem is not about the maturity levels, but more
about the document maturity when processing it.

Making a document an official WG item can make any further changes
to the document extremely painful because of the formal "WG consensus"
prodecure, especially if there is bias in the system (AD, WG chair,
document editor or several of them have a personal interest).

When an I-D is still fairly immature, it makes sense to fill out
the white spaces with little bureaucratic overhead (and more author
discretion, and optionally counter-proposals), before formally
adopting the document as a WG item.

I'm not at all surprised that applying maintenance mode change control
at the prototyping stage significantly impairs the evolvement of
new proposals/documents/I-Ds.

Eric Burger wrote:
>
>       since Proposed Standard is the new Draft Standard, and since the
> IESG has to make sure any proposal is beyond bullet-proof, the industry
> has long since implemented draft-mumble-21, which has not changed for
> over a year, and few in industry cares if the document publishes as an
> RFC, because from their point of view, if something has been working in
> the field for three years, has 18 independent implementations, and has
> not yet crashed the Internet, it is probably ready, whether the IETF
> formally says so or not. That is the fast track to making the IETF
> irrelevant.

The only way to make non-trivial technical proposals bullet-proof
is to implement them!  Look at how many documents the IESG has to
review--it is simply possible for the IESG to make an interop
assessment and/or a proof-of-concept for a non-trivial technical I-D.

What they're probably looking at is
 - overall architectural issues touched by the I-D
 - clarity and consistency of the document (structure and language)
 - references to other standards

What worries *me* is why there are 18 independent implementations
of a draft, but none of the implementors has helped getting the
document ready for publication.  Getting the I-D finalized should
be more important for vendors than shipping implementations of it.
Unfortunately, it seems to be the other way round.  That is not
something the IETF can change, it is vendors which have to
change their attitude about participating the standardization
process if they have an interest in that technology.

-Martin

---

From: Ted Hardie <[email-masked]>
Date: Tue, 26 Oct 2010 12:41:55 -0700

> that the proposal in draft-housley-two-maturity-levels has more support.

I'm assuming that the "other proposal" you mean is
draft-hardie-advance-mechanics;
if I missed a different one, please correct that mistake on my part.

First, I'm not sure that the two actually compete, except in that the community
might be unwilling to make two changes at once; mine was written with
explicit statement that it could be implemented with either a two or three
step process.

Second, I agree that draft-hardie-advance-mechanics does not have
enough support
to go forward at this time. There was some discussion of it, and I
likely should produce a new
draft.  I did not do so for this round because I did not sense anything like
enough support for it to indicate that the community had enough
interest and energy
to discuss thoroughly the change in working style (despite my personal belief
that it largely followed the current work style on the ground).

> Should I be seeking a sponsor for this draft?
The document currently has an open issue (in Section 8).  Does this need
to be resolved prior to seeking a sponsor?

regards,

Ted

---

From: Michael Richardson <[email-masked]>
Date: Tue, 26 Oct 2010 15:13:15 -0400

>>>>> "Andrew" == Andrew Sullivan <[email-masked]> writes:
    >> asked by an AD during Maastricht to provide proof of 2
    >> independent implementations just to have an ID I was presenting
    >> be considered to become a WG item.
    >> 

    Andrew> I agree, but I see precious little evidence so far presented
    Andrew> that the two-maturity-levels proposal under discussion will
    Andrew> solve that problem.

    Andrew> If the problem we need to solve is that it's too hard to get
    Andrew> an ID published as Proposed Standard, then we should tackle
    Andrew> that.  The problem in that case does not need tweaking of
    Andrew> standards levels and so on, but hard rebukes of WGs (and,
    Andrew> frankly, IESG members) who insist on making that publication
    Andrew> hard.

Given a finite amount of resources, if we reduce the number of levels,
it seems that we might have more resources left to deal with reducing
the time. 

If advancing from PS to step-two is easier, then maybe the "quality" of
a PS could go down.  I am also not convinced, but I prefer to try this,
since in effect, it's just documenting current practive anyway.

-- 
]       He who is tired of Weird Al is tired of life!           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
	               then sign the petition.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 26 Oct 2010 18:59:47 -0400

+1

Ship it, its ready.

-- 
Website: http://hallambaker.com/

---

From: Lars Eggert <[email-masked]>
Date: Wed, 27 Oct 2010 10:16:01 +0300

Hi,

> implementations just to have an ID I was presenting be considered to 

I was that AD, and your characterization is not accurate. (I've pointed this out privately before.)

The issue was that it was difficult to judge - due to lack of WG interest and review - whether interoperable specifications could be built based on the document in question. I said that *if there were* multiple implementations, that that would certainly demonstrate this nicely. But there are other ways, such as more WG review activity, etc.

Lars

From the Maastricht minutes:

    Lars: One way is to get people in TSVWG to work on and review the
          documents (and say they will do). There needs to be people doing
          this to ensure STD-track progression. Right now, it is hard for
          me to judge if an RSVP implementer can interoperate using this
          specification. If there were two independent implementations,
          this would clearly demonstrate the implementability of a Spec.

A bit later, Gorry followed up on that:

   Gorry: The RSVP directorate has been contacted in the last few
          days and I am hoping they will soon get back to me.
          -: The old IETF ethos of "rough consensus and running code"
          requires two implementations. This seems a good idea.
   Gorry: I like that.

    Lars: I like it too. This seems to be one way to show that the
          specifications are readable and it can be implemented.

---

From: "James M. Polk" <[email-masked]>
Date: Wed, 27 Oct 2010 11:21:26 -0500

>pointed this out privately before.)

I'll cop to the use of the word "proof" was stronger than it needed 
to be. It was still suggested though, even with another TSVWG author 
saying it was a good idea (Randy Stewart with SCTP).

>The issue was that it was difficult to judge - due to lack of WG 
>interest and review - whether interoperable specifications could be 
>built based on the document in question.

I think this gets to the core of what many of us have mentioned on 
the thread about Russ's proposal about two-maturity-levels: that 
whether or not two implementations can interoperate is the purpose of 
the PS level getting to the DS level, and should not be a basis for 
stalling an individual ID progressing to WG item (which you do admit 
to here in this email).

>I said that *if there were* multiple implementations, that that 
>would certainly demonstrate this nicely. But there are other ways, 
>such as more WG review activity, etc.

that (WG review) was ultimately (seemed to be) agreed to, but not 
right away - and it requested 5 full reviews before being considered, 
which is a high bar for either the Transport or RAI areas (that I'm 
familiar with over 11 years at the IETF).

***To everyone not familiar with TSVWG, we are chartered with 
progressing at least 5 different protocols and mechanisms. Room 
consensus is never easy for the chairs to realize due to the 5 
protocols and mechanisms are not close to each other topologically, 
other than they are all within the Transport realm. What I mean here 
is that if we have ~10 people in a room of 150 that claim interest in 
any of the charter items we chairs are ecstatic.***

There's another thing in play here, and that's who some (i.e., at 
least one AD) believe this extension is for (Military), and I'll 
repeatedly said I haven't heard this requirement from them (ever!), 
and haven't focused on that market segment for a couple of years now. 
Lars - you mentioned this target market when answering Fred during 
the (somewhat heated) TSVWG discussion in Maastricht.

We can talk more about that whenever you want, say in 2 weeks?

James

---

From: "James M. Polk" <[email-masked]>
Date: Wed, 27 Oct 2010 11:36:54 -0500

At 11:21 AM 10/27/2010, James M. Polk wrote:
>whether or not two implementations can interoperate is the purpose 
>of the PS level getting to the DS level, and should not be a basis 
>for stalling an individual ID progressing to WG item (which you do admit to

I inadvertently left out the word "suggesting" right here in the sentence.

>here in this email).
>right away - and it requested 5 full reviews before being 
>considered, which is a high bar for either the Transport or RAI 
>areas (that I'm familiar with over 11 years at the IETF).
>
>is that if we have ~10 people in a room of 150 that claim interest 
>in any of the charter items we chairs are ecstatic.***
>
>and haven't focused on that market segment for a couple of years 
>now. Lars - you mentioned this target market when answering Fred 
>during the (somewhat heated) TSVWG discussion in Maastricht.
>

---

From: SM <[email-masked]>
Date: Wed, 27 Oct 2010 11:18:56 -0700

Hello,

>Should I be seeking a sponsor for this draft?

I ask for your indulgence as I could not resist:

  "If you wish to seek Area Director sponsorship for an
   individual submission, the best solution is to contact the
   most relevant Area Director directly, with an explanation of
   why the draft is appropriate for IETF publication."

  "Also, please consider the normal IETF publication path
   through an existing working group, or consider proposing
   a BoF at a future IETF meeting."

In Section 1 of draft-housley-two-maturity-levels-02:

  "In the current environment, many documents are published as Proposed
   Standards and never advance to a higher maturity level.  Over time,
   this has resulted in IETF working groups and IESG members providing
   much more scrutiny than is called for by RFC 2026 [1] prior to
   publication as Proposed Standard."

Quoting a message from James M. Polk [1]:

  "I'm not in love with the 3 maturity levels, especially when I was asked by
   an AD during Maastricht to provide proof of 2 independent implementations
   just to have an ID I was presenting be considered to become a WG item."

That does not sound like IESG members providing much more 
scrutiny.  It sounds more like an undocumented rule being applied by 
IESG members.  A clarification about this has been posted [2].

Quoting the message:

   "If there were two independent implementations, this would
    clearly demonstrate the implementability of a Spec."

It is not a stretch too far to say that people will take that case 
and read it as a rule.

In Section 1:

  "One desired outcome is to provide an environment where the IETF
   community is able to publish Proposed Standards as soon as rough
   consensus is achieved."

That lowers the bar from "consensus" to "rough consensus" instead of 
setting consensus as the end-goal and falling back to rough consensus 
if people "agree to disagree".

In Section 2:

  "The requirements for Proposed Standard are unchanged; they remain
   exactly as specified in RFC 2026."

Does that mean that IESG members will not ask for two independent 
implementations?

In Section 5:

   "Lack of this review has not revealed any ill effects on the
    Internet Standards Process."

That means that evaluating the viability of the standardization 
effort and the usefulness of the technology is not important.

In Section 6:

   "The rules that make references to documents at lower maturity
    levels are a major cause of stagnation in the advancement of
    documents."

I would be grateful if anyone could point me to specific cases of 
that which could not have been addressed under current rules.

One of the bottlenecks of the process is the IESG.  It is not 
practical to ask IESG members to do a line by line review of hundreds 
of pages of Internet-Drafts before each IESG evaluation.  The number 
of Internet-Drafts seeking to attain Gold (proposed) Standard is 
quite high.  This proposal cannot change that.

The issue of timeliness of specifications has been mentioned.  If the 
actual specification is going to be delivered in two years or more, 
people will fall back to the Internet-Draft whatever the IETF 
says.  If the IETF is concerned about the IETF Standards Process, it 
could consider whether that can be brought down to one year.  This 
might entail:

  (i) one month for discussing what problem the author is trying to solve

  (ii) one month to haggle about process details

  (iii) nine months to go from Internet-Draft to Last Call

Review the document in a year, nit pick on it and refine it.  The 
IESG can ask for the implementation report at that stage.  Turn Draft 
into where the IETF tries to get it right.  The author would have 
some less than painful experience of the process by then to handle 
this.  The last stage could be seen as documents that have withstood 
the test of time.  If the Internet did not crash by then, the 
document is good enough.

The above is to encourage a "do or die" approach and leave it up to 
the community to go and write code to keep the IETF label.  It is 
infeasible to have IESG members catch all the bad ideas that are 
submitted for evaluation.  The IETF cannot protect the Internet.

   "The benefit of a standard to the Internet is in interoperability -
    that multiple products implementing a standard are able to work
    together in order to deliver valuable functions to the Internet's
    users."

The IETF could turn quality into:

    "the ability to express ideas with enough clarity that they
     can be understood in the same way by all people building
     systems to conform to them, and the ability (and willingness)
     to describe the properties of the system well enough to
     understand important consequences of its design, and to ensure
     that those consequences are beneficial to the Internet as a whole."

As mentioned in RFC 3935:

   "A part of being relevant is being timely - very often, documents
    delivered a year after core decisions have been taken are far
    less useful than documents that are available to the
    decision-makers at decision time."

As nothing has changed since the year 2004, it would be disingenuous 
to place the blame on the current IESG.

At 13:01 26-10-10, Hadriel Kaplan wrote:
>solve a problem anyone has.  For example, I don't think being at 
>only PS has prevented anyone from deploying HTTP or the myriad of 
>other protocols at PS level.  At this point, going above PS is for masochists.

The HTTP RFC is at Draft Standard level.

Regards,
-sm

1. http://www.ietf.org/mail-archive/web/ietf/current/msg64201.html
2. http://www.ietf.org/mail-archive/web/ietf/current/msg64298.html