Subject: Re: making our meetings more worth the time/expense (was: Re: setting a goal for an inclusive IETF)

From: [email-masked] (Noel Chiappa)
Date: Tue, 30 Jul 2013 12:35:02 -0400 (EDT)

> From: Keith Moore <[email-masked]>

Great message. One idea:

    > WG meeting sessions aren't scheduled to encourage discussion, but to
    > discourage it. At meeting after meeting, in several different areas, I
    > see the lion's share of the time devoted to presentations rather than
    > discussion.

Easy fix: 'slide' (well, nobody uses real slides anymore :-) rationing.

E.g. if a presenter has a 10 minute slot, maximum of 3 'slides'
(approximately; maybe less). That will force the slides to be 'discussion
frameworks', rather than 'detailed overview of the design'.

	Noel

---

From: Bob Braden <[email-masked]>
Date: Tue, 30 Jul 2013 10:47:21 -0700

> frameworks', rather than 'detailed overview of the design'.

Noel,

I tried the 3 slide limit in the End2end Research Group some years ago, 
and it did not work very well.
Presenters just can't discipline themselves that much, no matter how 
hard you beat on them.

Bob Braden

---

From: Keith Moore <[email-masked]>
Date: Tue, 30 Jul 2013 22:30:13 +0200

On Jul 30, 2013, at 7:47 PM, Bob Braden wrote:

> I tried the 3 slide limit in the End2end Research Group some years ago, and it did not work very well.
> Presenters just can't discipline themselves that much, no matter how hard you beat on them.

Maybe the first step is to stop having "presenters".

Keith

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 31 Jul 2013 08:38:26 +1200

> Presenters just can't discipline themselves that much, no matter how

It's been pointed out before that in a group with very diverse languages,
written words are usually better understood than speech. It's a fact of life
that you can't have a full-speed cut-and-thrust discussion in a group
of 100 people, half of whom are speaking a foreign language. Sitting in
a circle does not fix this.

Also, remote participants need full text slides; the soundtrack simply
isn't enough.

The old days are gone.

   Brian

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 30 Jul 2013 22:23:28 -0400

> a circle does not fix this.

While that is true, I think it misses the point of the objections to
the sit-and-watch-PowerPointTV.

First, I observe that we already _have_ a great deal of written words:
the drafts.  I continue to believe that altogether too much time in WG
meetings is spent "introducing", "presenting", or otherwise showing
off ideas in an existing draft to participants in the WG.  I
acknowledge that (particularly in early stages of WG life, in topics
with a lot of different work, and in cross-WG presentations) these
"intro" presentations are a fact of life.  But I think we are
extremely bad at holding the reigns on them.

In a WG meeting, I think such "intro" presentations about drafts
really can be kept to three pieces of information: the name of the
draft, a slogan describing the problem it is supposed to solve, and a
pointer to the beginning(s) of discussion thread(s) on the draft.  If
the person promoting the draft can't give the elevator pitch, they
don't know their own draft well enough to summarize it and shouldn't
be presenting it.  Any additional discussion in the presentation ought
to be exploring, as much as possible, one or more of the following
topics:

    - a particular issue
    - is $issue a real problem
    - alternatives for solving $issue
    - motivation for $issue solution choices

Each such slide, it seems to me, ought to encourage at most a couple
minutes of exposition and then some discussion.  The _reason_ to get
together in a big room with other people is to use the high-bandwidth
opportunity to hash out the extent of a problem.  The back and forth
of "you forgot this", "no that won't work because it explodes foo",
and so on, is the value here.  

Notice that none of that includes complicated flow-chart diagrams that
explain in detail a proposal.  There _is_ a place for those, however:
an actual presentation that gets made after significant discussion on
the list has made it clear that nobody understands the proposal.  At
that point, those 10-15 minute presentations of some proposed
mechanism are important, if only to inspire commenters to go back to
the list and say, "Ok, _now_ I get what you were trying to say, and
your text needs to be improved along the following lines."  But these
full explanation presentations happen too often when there has not
been such confusion.

Of course, all of the above depends on us going back to the list and
working out the details there, and it depends on people having read
the drafts and having a list of questions themselves that have been
deferred from the list for the face to face discussion.

I believe presentations in meetings are also sometimed useful if they
are exploring a problem space.  In that case, I believe what one needs
is _short_ presentations of the sort, "Here's what I think the
problems are," and then a lot of well-moderated discussion.

Unfortunately, actually running meetings this way is a lot of work,
requires fairly careful planning, and requires an indifference to
nasty remarks on the part of presenters who would much rather listen
to themselves for 20 minutes than to others.  But I think it'd make
for better meetings.  (Yes, along with room layouts that were more
suited to getting people to the mic.)
 
> The old days are gone.

Yes, and we need to figure out how to use meeting time effectively
here in the new days.  That effective use does not, I think, involve
expanding to fill all the time in the year with 20 minute low-content
presentations summarizing the draft that you can read in the span of
the time it takes to get through the presentation.  (Perhaps I'm
wrong.  Perhaps people find that the only time they have now to read
the drafts is during the presentation of the draft.  I sure hope not.)

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Scott Brim <[email-masked]>
Date: Wed, 31 Jul 2013 07:22:36 +0200

http://tools.ietf.org/wg/intarea/trac/wiki/MeetingTimePrioritization

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Jul 2013 08:35:50 +0200

> a circle does not fix this.

Yes, but most of the people in a typical WG meeting today aren't really participating in the meeting anyway.   They're not contributing input, they're not paying attention.  Their noses are in laptops.   It's hard to tell how many of them would be participating if the meeting were more useful, but the very fact that the room contains so many nonparticipants is itself a deterrent to getting work done in the meeting.   If nothing else, whenever someone tries to get a sense of the room, it's very misleading - people may be humming when they haven't even been listening, or it may appear that there's no significant support for something when there really is significant support among those who are interested in the topic.

> Also, remote participants need full text slides; the soundtrack simply

You seem to be assuming that the purpose of WG meetings is to have presentations.   I emphatically disagree.

If we decide to make WG meetings fora for interaction and discussion, we can adopt or invent disciplines and tools to better accommodate interaction and discussion between people of diverse languages and including those at other locations.   But the disciplines and tools that we've adopted at the moment are designed to accommodate an audience, not active participants.

> The old days are gone.

It sounds like you are saying that IETF is doomed to become irrelevant because it's stuck in habits that do not work.  I hope you're wrong about that.

Keith

---

From: Abdussalam Baryun <[email-masked]>
Date: Wed, 31 Jul 2013 12:52:39 +0100

IMHO, The presenters are MUST, but the time channel for presenting is the
problem or boring factor. I mentioned before that we need short
presentations 5 minutes, and more discussions.

AB

---

From: Abdussalam Baryun <[email-masked]>
Date: Wed, 31 Jul 2013 13:18:58 +0100

comments below

> extremely bad at holding the reigns on them.

The presenter SHOULD focus on taking WG feedback by asking WG Questions
like : do you think explaining in a section XXX will be good? then the WG
hummms,

> and so on, is the value here.

Please note that we SHOULD not only blame the presenter/author for this
problem of boring and presentation format, but I also blame the WG
participants, why the don't READ, READ, READ, the DRAFTS under AGENDA. If
they do READ then they can input please take my questions, 1, 2, 3 and
please take my recommendations 1, 2,3, and please take my
requests/comments, 1,2,3.

> full explanation presentations happen too often when there has not

I agree with this above point.

> deferred from the list for the face to face discussion.

I agree with you,

> problems are," and then a lot of well-moderated discussion.
I agree as if you ment short as less/equal than 5 minutes. The IETF Chair
and WG Chairs SHOULD consider these issues you raised.

> suited to getting people to the mic.)

I will add that We need with the author/presenter, to know the reviewers of
the draft, and get their short comments, so that will help the WG to decide
when asked for their opinion. So the reviewers are as a supervisor to the
WG, and the WG Chair is arranging their input in the session to make the
meeting valuable.

> the drafts is during the presentation of the draft.  I sure hope not.)

New draft proposals can be done in the last times of the session, and will
need to be discussed first on the list before presented. However, even
thoes proposals I will think they should be less than 5 minutes, and
discussions for them less than 10. For WG drafts the valuable
discussions SHOULD be more than 10 minutes, otherwise, we maybe done
side/hidden meetings.

AB

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Jul 2013 14:48:34 +0200

There are occasions when presentations are appropriate, but they should be the exception rather than the rule or default assumption.

Sent from my iPhone

On Jul 31, 2013, at 1:52 PM, Abdussalam Baryun <[email-masked]> wrote:

> IMHO, The presenters are MUST, but the time channel for presenting is the problem or boring factor. I mentioned before that we need short presentations 5 minutes, and more discussions.
>  
> AB
> 
> 
> On Tue, Jul 30, 2013 at 9:30 PM, Keith Moore <[email-masked]> wrote:
>>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 01 Aug 2013 08:55:41 +1200

Hi Keith,

On 31/07/2013 18:35, Keith Moore wrote:
>> It's been pointed out before that in a group with very
>> diverse languages, written words are usually better
>> understood than speech. It's a fact of life that you can't
>> have a full-speed cut-and-thrust discussion in a group of
>> 100 people, half of whom are speaking a foreign language.
>> Sitting in a circle does not fix this.
> 
> Yes, but most of the people in a typical WG meeting today
> aren't really participating in the meeting anyway.   They're
> not contributing input, they're not paying attention.  Their
> noses are in laptops.   

The last sentence there doesn't imply the previous one. I can
only speak for myself (and I'm not in Berlin), but I am often
looking at the slides, the draft and the jabber session while
listening to the speaker; no need to look up in order to pay
attention. Alternatively I am working on something else while
waiting for the topic I'm interested in to start; or I'm
checking whether it's time to move across to a clashing session.

Of course I have no idea what fraction of attendees are doing this.

> It's hard to tell how many of them
> would be participating if the meeting were more useful, but
> the very fact that the room contains so many nonparticipants
> is itself a deterrent to getting work done in the meeting.
> If nothing else, whenever someone tries to get a sense of the
> room, it's very misleading - people may be humming when they
> haven't even been listening, or it may appear that there's no
> significant support for something when there really is
> significant support among those who are interested in the
> topic.

That's true; I am much more suspicious of "sense of the room"
consensus calls than I am of "sense of the mailing list" calls.
But pretty much all WG Chairs do what they're supposed to by
confirming the consensus on the list.

>> Also, remote participants need full text slides; the
>> soundtrack simply isn't enough.
> 
> You seem to be assuming that the purpose of WG meetings is to
> have presentations.   I emphatically disagree.

The purpose is to communicate, including communication with
remote participants. Sometimes that means explaining things that
are, perhaps, badly explained in the draft - and getting back
comments that show what needs to be changed in the draft.

> If we decide to make WG meetings fora for interaction and
> discussion, we can adopt or invent disciplines and tools to
> better accommodate interaction and discussion between people
> of diverse languages and including those at other locations.
> But the disciplines and tools that we've adopted at the
> moment are designed to accommodate an audience, not active
> participants.

I don't think it's fair to blame the tools. We should be asking
questions about how we use the tools.

>> The old days are gone.
> 
> It sounds like you are saying that IETF is doomed to become
> irrelevant because it's stuck in habits that do not work.  I
> hope you're wrong about that.

No, I mean that we have to deal with a very diverse community
and the style of discussion that was successful when you or I
first started coming to the IETF isn't going to come back. I
don't, unfortunately, have the magic formula.

On 31/07/2013 14:23, Andrew Sullivan wrote:

> be presenting it.  

That's a bit hard on people early in their IETF career who may
be among the most creative. So I disagree, when we are talking
about a piece of new work that has attracted interest on the
mailing list - we should be reasonably liberal at that stage.
Once. After that, I agree; repeat intros and non-substantive
discussions shouldn't be given meeting time. But I've been in
vey useful sessions where people have showed slides to
illustrate open issues, and got very productive feedback that
clarifies the options. I've been in other sessions where after a
few seconds I switch off after 2 slides and wait for the chairs
to call time.

<deleted several paragraphs that I strongly agree with>

...
> suited to getting people to the mic.)

I absolutely agree.

> the drafts is during the presentation of the draft.  I sure hope not.)

I agree here too. It really all depends on strong session
control by the chairs and a much stronger "this is not a
conference" message to participants.

   Brian

---

From: Keith Moore <[email-masked]>
Date: Wed, 31 Jul 2013 23:22:01 +0200

> confirming the consensus on the list.

When binding decisions are being made, this is of course the right thing to do.   But we still need an accurate sense of the room (from active participants) just to make good use of meeting time - so (for example) we can know whether or not it appears that a problem has been discussed sufficiently and/or addressed.   Of course we have to consult the mailing list in order to be sure we have rough consensus, but it's also very important to take advantage of face-to-face time to attempt to work out solutions to thorny problems.   So the failure to get an accurate sense of the room might not necessarily mean that the wrong decision is made, it might mean that the decision doesn't get made until much later - maybe after there have been another one or two meetings.  

> comments that show what needs to be changed in the draft.

Sometimes, yes.   But I doubt that should occupy the bulk of our meeting time.

> questions about how we use the tools.

Some people don't see this, but I'm very convinced that the tools do have a significant effect on how we choose to convey our messages.   Partly because of the hardware, partly because of the software, these tools are optimized for certain things and pessimized for others.    It's pretty obvious that a laptop with a full-size keyboard and a poor pointing/drawing device is going to be much better at putting text on a screen than drawing diagrams.    It's very easy to do what the tool makes it easy to do (typically put a few words of text on a screen) while losing sight of your purpose (typically to facilitate an interactive discussion with input from several parties).   Part of the problem is indeed the tool's fault, or perhaps our fault for choosing the wrong tool for the job out of habit.   

It's not that visual aids are inherently bad.  Properly chosen visual aids can really help facilitate a discussion.   But a few words in large type on a small screen is rarely effective, and these screens aren't big enough and the projectors often lack the resolution needed to display adequate diagrams.   Also, the tools are generally designed to produce static content, rather than let you manipulate the content while it's on screen (say to illustrate the answer to a question).

> don't, unfortunately, have the magic formula.

I'm not arguing that we need the same style of discussion that worked for IETF when it was ~300 people.   You're right that we have a much larger and more diverse community now; we also have a much more diverse set of needs to consider in our designs.     Because of all of this diversity, it's even more important that we let diverse voices be heard.   We certainly need a better way to moderate discussions than either having the loudest people speak whenever they want, or forcing people to queue up at microphones.    And yet, I think what we're doing now is actually far worse (even for our current circumstances) than what IETF was doing in 1990.   We've taken huge steps in the wrong direction.

Keith

---

From: Melinda Shore <[email-masked]>
Date: Wed, 31 Jul 2013 13:34:21 -0800

I have to say that I was very impressed with how the oauth
session went.  There was minimal presentation and maximal
discussion, and the discussion was not interrupted until it
started getting circular.

But, I suspect that this is a reflection of the fact that
there's some substantial disagreement about not just
narrowly-defined issues, but some architectural models that
people are working from, that there's a very large number
of interested parties, there are time pressures and
expectations from external bodies, etc.  The discussion
would have been lively no matter how the meeting was
managed.  But, I thought it was very well-done.  The
challenge is how to have meetings that productive when
there are far fewer interested parties, fewer disagreements,
etc.  It may be the case in some instances that if
it's going to be nothing but presentations there may not
be a need for a working group to meet at all.

Melinda

---

From: Keith Moore <[email-masked]>
Date: Thu, 1 Aug 2013 04:35:27 +0200

On Jul 31, 2013, at 11:34 PM, Melinda Shore wrote:

> It may be the case in some instances that if

+1.   If nothing else, when a WG agenda starts to shape up like this, this should be a big red flag for chairs and ADs.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Thu, 01 Aug 2013 07:50:40 +0200

On 7/31/2013 4:23 AM, Andrew Sullivan wrote:
> First, I observe that we already_have_  a great deal of written words:

I'm a fan of powerpoint, but with careful discipline in its use.

"The drafts" are the bible for detail, of course.  They must be the 
ultimate arbiter of current truth.  But while consulting the drafts for 
very specific concerns is often essential, using them during a 
discussion often is distracting.  Forest-vs-tree distinctions come to mind.

I see properly done slides as serving two important purposes:

    1. Concise summary of issues

    2. Helpful synchronization for those not fully up to speed and
       especially those not native English speakers

The first requires the presenter to distill things beforehand and in 
their head, and that's always useful discipline for a discussion and 
even more so for a debate.  The exercise of producing the distillation 
serves to ensure that the speaker, at least, has a concise sense of the 
issues.

The latter is just good group management, especially for a group trying 
to be... inclusive.

The challenge is how to balance detail with summary -- verbosity with 
terseness -- in the slides. And how to use the slides.

I used to do slides that were very cryptic.  Just keywords.  As I had to 
do presentations with non-native speakers, I was taught to make the 
slides stand alone so that they were useful even without anyone speaking 
to them.  "Telegraphic" language is roughly where that landed.  More 
than just keywords but much, much less than full narrative.

As for how to use the slides, I try not to read them, but instead to 
speak in a way that covers what the slides cover, albeit in full 
sentences.  So there should be correlation, but not verbatim recitation. 
  (I said "try".)

 > http://tools.ietf.org/wg/intarea/trac/wiki/MeetingTimePrioritization

I might argue that that specific list is overly fussy and possibly 
Procrustean, but the gist of it definitely looks like the right kind of 
thinking, to focus wg face-time on resolving things rather than chatting 
and teaching.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net