Subject: RE: Appeal Response to Abdussalam Baryun regarding draft-ietf-manet-nhdp-sec-threats

From: <[email-masked]>
Date: Wed, 3 Jul 2013 00:37:10 +0100

Do we have any statistics on how many appeals to the IESG fail and how many succeed?

If I knew that 97% of appeals get rejected, I wouldn't even bother writing one...

(On the other hand, that might simply be because 97% of the appeals are written by loons. Statistics can't tell us everything.)

Lloyd Wood
http://sat-net.com/L.Wood/

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of The IESG [[email-masked]]
Sent: 02 July 2013 23:24
To: [email-masked]
Cc: [email-masked]
Subject: Appeal Response to Abdussalam Baryun regarding draft-ietf-manet-nhdp-sec-threats

The IESG has reviewed the appeal of Abdussalam Baryun dated June 19,
2013 on the subject of inclusion in the acknowledgments section of
draft-ietf-manet-nhdp-sec-threats:

http://www.ietf.org/iesg/appeal/baryun-2013-06-19.txt

This is a dispute about a matter in a working group. The same matter has
previously been raised with the working group chairs and responsible
Area Director, as specified in RFC 2026 Section 6.5.1.

Writing acknowledgments sections is largely a matter of editorial
discretion, where good sense and general attribution practices are the
primary guidelines, although RFC 2026 Section 10.3.1 has some specific
rules regarding acknowledgment of major contributors, copyright, and
IPR.

After reviewing the appeal, including the associated list discussion and
draft revisions, the IESG concludes that the authors made a reasonable
editorial choice that was well within their discretion and that none of
the messages at issue fall under the required acknowledgment rules of
RFC 2026 Section 10.3.1 and RFC 5378 Sections 5.6a and 1c. The IESG
finds that the chairs and responsible AD handled complaints about the
matter appropriately.

---

From: Bob Hinden <[email-masked]>
Date: Tue, 2 Jul 2013 17:08:55 -0700

Lloyd,

> Do we have any statistics on how many appeals to the IESG fail and how many succeed?

Appeals are listed at:

   https://www.ietf.org/iesg/appeal.html

Bob

---

From: Randy Bush <[email-masked]>
Date: Wed, 03 Jul 2013 09:19:28 +0900

> If I knew that 97% of appeals get rejected, I wouldn't even bother

i have never considered writng one.  sour grapes make bad wine.

randy

---

From: Russ Housley <[email-masked]>
Date: Tue, 2 Jul 2013 22:23:36 -0400

http://www.ietf.org/iesg/appeal.html

Every appeal ever submitted to the IESG and its response can be found here.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 03 Jul 2013 15:10:23 +1200

> Every appeal ever submitted to the IESG and its response can be found here.

...since late 2002, that is. There were appeals earlier in history. The
first one I recall reached the IAB in 1995, and had presumably already
been rejected by the IESG. However, statistics since 2002 are probably
enough.

I'd say that rejected appeals quite often lead to clarification of
procedures for the future; therefore they have value, even if it
isn't immediately obvious.

   Brian

---

From: Dave Crocker <[email-masked]>
Date: Tue, 02 Jul 2013 20:43:52 -0700

> been rejected by the IESG. However, statistics since 2002 are probably

I believe I submitted the first appeal.  It was to break a logjam with 
the the IETF's failure to sign an agreement with Sun, to get NFS brought 
into the IETF.  I had been the cognizant AD when they approached the 
IETF, but the ball got dropped after that.

So it wasn't so much 'rejected' by the IESG as it was cast in terms of 
IESG failure, although really no one was quite sure who needed to sign 
the contract with Sun and everyone was afraid of doing it.

My reading of the appeal was that it succeeded, in that the agreement 
with Sun was signed shortly after that and the IETF took over the NFS 
specification.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Jari Arkko <[email-masked]>
Date: Wed, 3 Jul 2013 07:19:20 +0200

> i have never considered writng one.  sour grapes make bad wine.

Errors do happen, for everyone and for all organisations. We do not treat appeals as sour grapes at the IESG, IAB or other places that receive them. We consider them an opportunity to review whether something was missed. At the same time, we do not intend to give special treatment to an argument just because it is labeled as an appeal. Sometimes legitimate differences of opinion are just that, and consensus was rough.

Jari

---

From: William McCall <[email-masked]>
Date: Wed, 03 Jul 2013 05:39:57 -0500

> i have never considered writng one.  sour grapes make bad wine.
I used to read the appeals for my own education. Some pretty hilarious 
stuff in there. I feel this contributor's frustration though (even 
though the IESG is right).

I'd tend to agree with you Randy... save it for something worth 
advocating for.

---

From: Pete Resnick <[email-masked]>
Date: Wed, 3 Jul 2013 11:32:37 -0500

> Do we have any statistics on how many appeals to the IESG fail and how many succeed?

My quick read of http://www.ietf.org/iesg/appeal.html:

Accepted: 6
Denied: 25
Withdrawn: 1

One appellant appealed 12 times and all of the appeals were denied. One 
appellant appealed 4 times, all denied. One appellant appealed 3 times, 
all denied.

At least two of the accepted appeals resulted in a different remedy than 
requested by the appellant (i.e., adding an IESG Note to a document 
instead of making other changes or rejecting the document).

At least two of the denied appeals were on strictly procedural grounds; 
one came over two months after the action, one was appealing an IAB 
decision that was out of jurisdiction for the IESG to decide.

Interpret the above as you see fit.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 3 Jul 2013 09:54:26 -0700

Jari, *:

Disclaimer: see signature (i do not know the details of this specific case).

To me the problem seems to be going back to the means the IETF has for providing recognition
to participants contributing by review/feedback. As long as recognition for that contribution
is primarily left to the disgression of the listed draft authors, it will negatively impact
the amount of especially critical feedback/review the IETF will see. Unless a contributor has
a specific business reason to reject or help to improve a drafts, its most likely not worth
their time to fight / improve documents without better means of recognition than how its
defined today. Especially if their job role lives off showing recognition for their contribution
to their employer/sponsor.

As much as i hate overboarding processes, an explicit review tool tracking feedback
and approval/disapproval of documents may be able to help here. Especially given how
there is already tooling to show some form of IETF score based on explicit
authorship. You know who's tool i am talking about ;-)

Not claiming i am persuaded that the problem is significant enough to invest into an
explicit review tool, just saying its more than just difference of opinions or rough
consensus as you seem to claim (if i undestood you correctly).

Cheers
    Toerless

-- 
---
Toerless Eckert, [email-masked]
It's much easier to have an opinion if you do not understand the problem.

---

From: Warren Kumari <[email-masked]>
Date: Wed, 3 Jul 2013 13:02:36 -0400

On Jul 3, 2013, at 12:32 PM, Pete Resnick <[email-masked]> wrote:

> One appellant appealed 12 times and all of the appeals were denied. One appellant appealed 4 times, all denied. One appellant appealed 3 times, all denied.
> 
> At least two of the accepted appeals resulted in a different remedy than requested by the appellant (i.e., adding an IESG Note to a document instead of making other changes or rejecting the document).
> 
> At least two of the denied appeals were on strictly procedural grounds; one came over two months after the action, one was appealing an IAB decision that was out of jurisdiction for the IESG to decide.
> 

Thank you -- another worthwhile thing to do is look at who all has appealed and ask yourself "Do I really want to be part of this club?"

Other than a *very* small minority of well known and well respected folk the http://www.ietf.org/iesg/appeal.html page is basically a handy kook reference.

W

--
It is impossible to sharpen a pencil with a blunt axe.  It is equally vain
to try to do it with ten blunt axes instead.
    --  E.W Dijkstra, 1930-2002

---

From: John C Klensin <[email-masked]>
Date: Wed, 03 Jul 2013 14:10:47 -0400

--On Wednesday, July 03, 2013 13:02 -0400 Warren Kumari
<[email-masked]> wrote:

> Thank you -- another worthwhile thing to do is look at who all
> has appealed and ask yourself "Do I really want to be part of
> this club?"

I am honored to be a member of that club.   Remembering that
appeals, as others have pointed out, a mechanism for requesting
a second look at some issue, they are an important, perhaps
vital, part of our process.  We probably don't have enough of
them.  Effectively telling people to not appeal because they
will be identified as "kooks" hurts the process model by
suppressing what might be legitimate concerns.

In addition, it is important to note that the page does _not_
list every appeal since 2002.  If one reads Section 6.5 of RFC
2026, it describes a multi-step process for appears in each of a
collection of categories.  The web page lists only those that
were escalated to full IESG review.  That is important for two
reasons:

* The majority of appeals, and a larger majority of those that
are consistent with community consensus or technical
reasonableness, are resolved well before the issues involved
come to the formal attention of the full IESG.  If an issue is
appealed but discussions with WG Chairs, individuals ADs, or the
IETF Chair result in a review of the issues and a satisfactory
resolution, then that is an that is completely successful in
every respect (including minimization of IETF time) but does not
show up in the list on the web page or statistics derived from
it.

* A few minutes of thought will probably suffice to show you
that appeals that have significant merit are far more likely to
be resolved at stages prior to full IESG review.   By contrast,
a hypothetical appeal that was wholly without merit, or even
filed with the intent of annoying the IESG, is almost certain to
reach the IESG and end up on the list, badly distorting the
actual situation.

best,
   john

p.s. to any IESG members who are reading this: community
understanding of the process might be enhanced by putting a note
on the appeals page that is explicit about what that list
represents, i.e., only appeals that reached full IESG review and
not all appeals.

> Other than a *very* small minority of well known and well
> respected folk the http://www.ietf.org/iesg/appeal.html page
> is basically a handy kook reference.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 03 Jul 2013 11:18:44 -0700

> Interpret the above as you see fit.

As with most 'social' analyses, it's usually a good idea to look for a 
bit more than an entirely trivial numbers game, such as by trying to 
find some criterion that helps to distinguish amongst the appellants.

In this case, I think that a reasonable distinction could be made 
between real participants in the community, versus, ummmm... others. A 
plausible-if-simplistic criterion could be noting whether the appellant 
had authored at least one RFC.

As with most heuristics, it doesn't provide a guarantee.  Still, it 
looks like a useful filter.

d/

---

From: Sam Hartman <[email-masked]>
Date: Wed, 03 Jul 2013 14:46:27 -0400

>>>>> "John" == John C Klensin <[email-masked]> writes:

Strong agreement.
I'm not currently a member of that club, although if I stick around the
IETF long enough it's bound to happen.
I've certainly received and reviewed appeals that I thought were a valid
contribution to the process.

Don't appeal if there is a better way to address your concerns, but if
an appeal is the right approach, then file one.

---

From: SM <[email-masked]>
Date: Wed, 03 Jul 2013 10:24:50 -0700

At 03:39 03-07-2013, William McCall wrote:
>I used to read the appeals for my own education. Some pretty 
>hilarious stuff in there. I feel this contributor's frustration 
>though (even though the IESG is right).

The decision of the respected members of the IESG was 
predictable.  There may be a minor issue.  I cannot comment about that.

The appellant mentioned that he worked hard and he has been 
excluded.  In my opinion any other contributor might be frustrated in 
similar circumstances.  The IETF is a place of many 
misunderstandings.  Maybe one day something good will come out of all that.

>to their employer/sponsor.

Yes.

There are more incentives not to perform a critical review of a draft 
instead of doing the reverse.  If contributors operate solely for 
business reasons it can lead to IETF structural issues.

>I am honored to be a member of that club.   Remembering that

:-)

>suppressing what might be legitimate concerns.

Yes.

>every respect (including minimization of IETF time) but does not

Agreed.

Sometimes a gesture of goodwill is all that it takes.

Regards,
-sm

---

From: Ted Lemon <[email-masked]>
Date: Wed, 3 Jul 2013 19:16:24 +0000

On Jul 3, 2013, at 2:18 PM, Dave Crocker <[email-masked]> wrote:
> As with most 'social' analyses, it's usually a good idea to look for a bit more than an entirely trivial numbers game, such as by trying to find some criterion that helps to distinguish amongst the appellants.

Yup.   E.g., it's worth reading the IESG response to John Klensin's appeal, and also the IESG response to Dean's most recent appeal.   I wasn't on the IESG for either of these, so I have no attachment to the text as written, but in both cases it seems very clear to me that the right thing was done, and that the responses were well reasoned and thorough.

---

From: Pete Resnick <[email-masked]>
Date: Wed, 3 Jul 2013 14:31:24 -0500

On 7/3/13 1:10 PM, John C Klensin wrote:
>> Other than a*very*  small minority of well known and well respected folk thehttp://www.ietf.org/iesg/appeal.html  page is basically a handy kook reference.
>>      

I think this is a bit overstated. There are 14 unique names of 
appellants (2 of which are groups of appellants). As I stated, 3 of 
those appellants account for 19 appeals, all denied. Perhaps you don't 
want to be part of the club with those 3 who make up 60% of the 
appealing, but if you simply remove those, you get:

13 appeals for 11 appellants (2 of them appealed twice, with years in 
between appeals)
1 appeal withdrawn before the IESG decided
6 appeals accepted
6 appeals denied.

So the small minority are actually the repeat appealers. Of the rest, 
over half I would instantly recognize as well-known and long-time 
participants, and (without naming names) half of *those* folks were 
denied and half were accepted.

So appeals that get to the level of the IESG from the group of 11 are 
accepted half of the time. That means that these folks are bringing 
issues to the IESG that, after having gone through the WG, the chairs, 
and the cognizant AD, half the time are still accepted by the IESG. That 
is, there's a 50/50 shot they've found a serious problem that the IESG 
agrees the rest of us in the IETF have missed.

I'd be part of that club.

> suppressing what might be legitimate concerns.

Agreed. In any dispute process, there will be some folks who are 
outliers that make up an awful lot of the total load. But that shouldn't 
take away from those who are using it for its designed purpose.

> were escalated to full IESG review.

Interestingly, 2026 6.5 only refers to things that get to the IESG, IAB, 
or ISOC BoT as "appeals". The rest of the "discussions" are simply part 
of "dispute" or "disagreement" resolution.

But John's central point still stands: Most of the dispute resolution 
takes place before it ever gets to the IESG, IAB, or ISOC BoT as a 
formal appeal.

> represents, i.e., only appeals that reached full IESG review and

Good idea.

pr

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 3 Jul 2013 15:41:33 -0400

+1

And don't lets forget that plenty of people have proposed schemes that WGs
have turned down and then been proven right years later.

If people are just saying what everyone else is saying here then they are
not adding any value. Rather too often WGs are started by folk seeking a
mutual appreciation society that will get through the process as quickly as
possible. They end up with a scheme that meets only the needs of the mutual
appreciation society.

On Wed, Jul 3, 2013 at 3:31 PM, Pete Resnick <[email-masked]>wrote:

> **
> --On Wednesday, July 03, 2013 13:02 -0400 Warren Kumari<[email-masked]> <[email-masked]> wrote:
>
>
>
> Other than a **very** small minority of well known and well respected folk the http://www.ietf.org/iesg/appeal.html page is basically a handy kook reference.
>
>
>
> I think this is a bit overstated. There are 14 unique names of appellants
> (2 of which are groups of appellants). As I stated, 3 of those appellants
> account for 19 appeals, all denied. Perhaps you don't want to be part of
> the club with those 3 who make up 60% of the appealing, but if you simply
> remove those, you get:
>
> So the small minority are actually the repeat appealers. Of the rest, over
> half I would instantly recognize as well-known and long-time participants,
> and (without naming names) half of *those* folks were denied and half were
> accepted.
>
> accepted half of the time. That means that these folks are bringing issues
> to the IESG that, after having gone through the WG, the chairs, and the
> cognizant AD, half the time are still accepted by the IESG. That is,
> there's a 50/50 shot they've found a serious problem that the IESG agrees
> the rest of us in the IETF have missed.
>
> Agreed. In any dispute process, there will be some folks who are outliers
> that make up an awful lot of the total load. But that shouldn't take away
> from those who are using it for its designed purpose.
>
>
> or ISOC BoT as "appeals". The rest of the "discussions" are simply part of
> "dispute" or "disagreement" resolution.
>
> takes place before it ever gets to the IESG, IAB, or ISOC BoT as a formal
> appeal.
>
>
> Pete Resnick <http://www.qualcomm.com/~presnick/> <http://www.qualcomm.com/~presnick/>

-- 
Website: http://hallambaker.com/

---

From: Eliot Lear <[email-masked]>
Date: Wed, 03 Jul 2013 22:08:46 +0200

No hat on and I'm not commenting on the specific case at hand.

On the general point, I think it's better to err a bit toward the
generous side.  As an author I use as a rule of thumb whether or not I
or the working group has taken someone's suggestion and put it into
text.  And it has to be more than editorial.  A missing "," doesn't get
you into acknowlegments, but highlighting confusion or clarifying
language probably does.

Eliot

---

From: Michael StJohns <[email-masked]>
Date: Wed, 03 Jul 2013 17:04:06 -0400

Before we go down this rathole too far again -

1) If you want to second guess the working group, AD and IESG, then the best approach is to probably review the bidding by reading the emails on the working group list and then forming an opinion based on that record.  I have and I'm pretty content with the current result.

2) With respect to credit, I would add that even to be considered for it a contributor's balance of payments on a document, standard etc needs to be substantially net positive.  If you're trying to claim credit for a reasonable suggestion, but its one of 10s or 100s of unreasonable ones, and you've pretty much disrupted the working group while making said suggestions..... well why reward that?

3) I think we need to continue to give each WG and each AD deference to their established ways of proceeding - unless and until there is some determination supported by facts that there is a problem.  

Mike

---

From: Warren Kumari <[email-masked]>
Date: Wed, 3 Jul 2013 18:58:56 -0400

On Jul 3, 2013, at 3:41 PM, Phillip Hallam-Baker <[email-masked]> wrote:

> +1 
> 
> And don't lets forget that plenty of people have proposed schemes that WGs have turned down and then been proven right years later.
> 
> If people are just saying what everyone else is saying here then they are not adding any value. Rather too often WGs are started by folk seeking a mutual appreciation society that will get through the process as quickly as possible. They end up with a scheme that meets only the needs of the mutual appreciation society.
> 
> 
> 
> 
> On Wed, Jul 3, 2013 at 3:31 PM, Pete Resnick <[email-masked]> wrote:
>>> Other than a 
>>> *very* small minority of well known and well respected folk the http://www.ietf.org/iesg/appeal.html
>>>  page is basically a handy kook reference.
>>>     
>>> 
> 
> I think this is a bit overstated.

Yes. It was a flippant response and there should probably have been a smiley somewhere in it...

> There are 14 unique names of appellants (2 of which are groups of appellants). As I stated, 3 of those appellants account for 19 appeals, all denied. Perhaps you don't want to be part of the club with those 3 who make up 60% of the appealing,

Yup, that is the club I was meaning.

> but if you simply remove those, you get:
> 
> 13 appeals for 11 appellants (2 of them appealed twice, with years in between appeals)
> So the small minority are actually the repeat appealers.

Yeah, you are right.
I was simply looking at the list of repeats. 

> Of the rest, over half I would instantly recognize as well-known and long-time participants, and (without naming names) half of *those* folks were denied and half were accepted.
> 
> So appeals that get to the level of the IESG from the group of 11 are accepted half of the time. That means that these folks are bringing issues to the IESG that, after having gone through the WG, the chairs, and the cognizant AD, half the time are still accepted by the IESG. That is, there's a 50/50 shot they've found a serious problem that the IESG agrees the rest of us in the IETF have missed.
> 

Yup, fair 'nuff -- as would I.

> Agreed. In any dispute process, there will be some folks who are outliers that make up an awful lot of the total load. But that shouldn't take away from those who are using it for its designed purpose.

Agreed. The dispute / appeals process is important, and needed -- it has served, and I'm sure will continue to serve, a useful purpose. 

But, before filing an appeal I think one should take a step back, wait a day or three to calm down and ask oneself:
A: is this really worthy of an appeal? 
B: how / why did we end up here? 
C: does my appeal look more like the club of 3, or the club of 11? 
D: have I tried to resolve this without resorting to appeals? really?
E: do I actually understand how this IETF thingie works?  
F: was there any sort of process violation or am I simply annoyed that no-one likes / listens to me?
G: have I filed more appeals than actual contributions?
H: does my appeal text Contain Randomly capitalized Text or excessive exclamation marks? Have I made up words?
I: am I grandstanding?
J: am I simply on the rough side of consensus?
K: is this really worthy of an appeal? 

W
> Interestingly, 2026 6.5 only refers to things that get to the IESG, IAB, or ISOC BoT as "appeals". The rest of the "discussions" are simply part of "dispute" or "disagreement" resolution.
> 
> But John's central point still stands: Most of the dispute resolution takes place before it ever gets to the IESG, IAB, or ISOC BoT as a formal appeal.
> 
> 

--
"Build a man a fire, and he'll be warm for a day. Set a man on fire, and he'll be warm for the rest of his life." -- Terry Pratchett

---

From: <[email-masked]>
Date: Thu, 4 Jul 2013 01:20:29 +0100

> C: does my appeal look more like the club of 3, or the club of 11? 

I think there's a new club of one.

Lloyd Wood

---

From: Doug Barton <[email-masked]>
Date: Wed, 03 Jul 2013 17:33:56 -0700

> I think there's a new club of one.

Wait, so now instead of voting we're using clubs? I think I need to pay 
more attention to this thread ...

---

From: Dave Cridland <[email-masked]>
Date: Thu, 4 Jul 2013 01:36:01 +0100

Yeah, but we don't actually count the clubs, so it's okay.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 4 Jul 2013 01:09:19 +0000

On Jul 3, 2013, at 8:33 PM, Doug Barton <[email-masked]> wrote:
> Wait, so now instead of voting we're using clubs? I think I need to pay more attention to this thread ...

If you don't read ietf, you don't get to participate in the consensus... ;)

---

From: Jari Arkko <[email-masked]>
Date: Mon, 8 Jul 2013 14:34:45 +0200

Toerless, SM, and others who commented on the importance of recognising people who made contributions: I fully agree, of course. Giving credit for contributions, be it about being the developer of a major protocol, having your name on the author list, or being mentioned in the acknowledgments is one of the currencies that help draw people into doing work at the IETF. Along with people's need to get an internetworking problem solved, of course. And the needs for their software to interwork with others. And the needs of their users being met…

Anyway, back to acknowledgments. We should, of course, give credit for contributions. I hope we all think about this long and hard when we write our documents, and do the right thing. Erring on the side of being inclusive is probably a better strategy for most cases.

The issue in this case though was where to draw the line. As an example, for my documents, I've mostly used a strategy where I acknowledge the significant contributions. I've occasionally used another approach, essentially listing everyone who had done any work relating to the document, no matter how small. I think either model is defensible, but there will always be a question of what contributions meet the criteria for being included. Should I acknowledge someone if they post a review that said everything is OK? A comment on the mailing list that they support this document? A suggestion that did not result in a change in the document? A minor editorial fix? A question? An argument? We do not have a definition of what kinds of things should result in your name being listed in the acknowledgments. And I don't think we should formalise that either. It is a better model to have the authors make common sense decisions about these matters. And, as with any topic, if there is a mistake there are several opportunities to rectify the situation if after analysis it seems that a mistake was made. But only if it were a clear mistake - I think it would be a bad model if the IESG or someone else were to micromanage this. The documents are WG's documents and author's documents.

Jari