Subject: Re: Fwd: Re: IP: Microsoft breaks Mime specification

From: "Kyle Lussier" <[email-masked]>
Date: Sat, 26 Jan 2002 18:14:56 -0800

>   *> But the use of a trademark, which stands for "complies with RFCs"
>   *> could be incredibly valuable.

> I suggest that you read RFCs 1122 and 1123 from cover to cover, and
> then ponder whether the nice-sounding phrase "complies with the RFCs"
> has any useful meaning.  Perhaps you will begin to understand why the
> IETF Way is interoperability testing, not conformance testing But you
> are free to make your proposal at IAB plenary of the next IETF.

Thanks for the comments Bob!  I think there is very much
a misconception as to what I am proposing.

As I've mentioned, I absolutely, positively do not want 
conformance testing, of any kind!

Purely an IETF endorsed logo. If you *want* to use a logo, you send 
in your $50-$100, sign the agreement that says your product works 
with the RFCs, and you get permission to use the trademark.

Procedures would have to be in place to provide a "logo yank"
process in eggregious abuses.  It shouldn't be easy to yank
a logo, it should be thoroughly peer reviewed.  I wouldn't
even mind if it took 12 months+ to yank a logo.  

What I am fundamentally looking for here is a procedure by which 
there is a control mechanism for defining a vendor trying to
be interoperable (which is a huge consumer, customer, and vendor
benefit) vs. a vendor that is using taking standards and abusing
them in the marketplace.

When you yank the logo, it's not like you can't still sell
your product.  

It's just for us, as a vendor, having something like this allows us 
to contract to supporting "interoperable" third party vendors that 
are well behaved, and we get an "opt-out" on vendors whom the
IETF community has put a big red "X" on.

Zero, and I repeat Zero conformance testing.  The reality is,
standards and RFCs are going to get it only "mostly right"
the majority of the time, and standards need to change.

But the good faith intentions of a vendor towards interoperability
should not change.

The very simple logo idea I am proposing is purely a visible rating
system at to the good faith intentions of a vendor to be interoperable.

I am just saying, we need to reward intoperable vendors with the
logo, and give CIOs the option to sign deals with vendors who
are truly faithful to standards.

I think this idea could help all of the markets significantly in
terms of giving everyone a visible mark of interoperability.  You
get the mark until you absolutely, positively aggregiously abuse
it.  For 99% of the companies supporting IETF this will be 
extraordinarily valuable, and help all of us sell our products
as well as get some money to have some IETF parties. :)

This will only be a pain in the butt for the 1% of particularly
powerful vendors who are unwilling to support IETF standards.

Kyle Lussier
AutoNOC LLC

---

From: [email-masked]
Date: Sat, 26 Jan 2002 22:19:38 -0500

> IETF community has put a big red "X" on.

There's problems here:

1) Two logo'ed products can still fail to interoperate.  Remember - this
thread was started by a failure to interoperate.  But by the time the
IETF even *heard* about the MIME bug that started this discussion, the
vendor had already acknowledged it was a bug, and assigned a bug ID to it.
So the vendor is being responsive, keeps the logo - and it didn't tell
you anything about the product.

2) Two X-out'ed products can still manage to interoperate.

3) If there's *no* conformance testing, what does it *actually* tell
you other than "the company had $100 and bothered sending it in?". 

4) What do you do if you spec that logo on an RFP, and only one
vendor has a logo'ed product - and it's the worst of the bunch?

I have in my bedroom a night light, which I purchased at a local
grocery store.  It has a UL logo on it, which doesn't tell me much
about its suitability as a night light (I can't tell if it's bright
enough, or if it's too bright, or what its power consumption is),
but it *does* tell me 2 things:

1) It has been *tested* and found free of any known safety design problems.
It may not *work* as a night light, but it won't shock me when I go to
throw it in the trash can because it's not suitable.

2) A high enough percentage of night light manufacturers get UL listed
that I can afford to be suspicious of any company that doesn't have
the logo on their product.

Ask yourself this - if there's no up-front testing of minimum compliance,
how is the $100 any different from the money customarily paid in
some parts of the world to the representatives of the local <insert
ethnic-based organized crime syndicate>?

				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

---

From: grenville armitage <[email-masked]>
Date: Sun, 27 Jan 2002 16:17:30 +1100

Kyle Lussier wrote:
	[..]
> conformance testing, of any kind!
	[..]
> benefit) vs. a vendor that is using taking standards and abusing

Interoperable with what?

Interoperability testing occurs between implementations, and doesn't
require reference to a document or specification. Conformance testing
is, essentially, interoperability testing against an implementation
that has previously been declared standards-compliant - the reference
implementation.

Your process for yanking a logo requires a vendor's implementation to
fail an interoperability test against a known standards compliant
implementation. Anything less would make the logo meaningless. That
smells dangeoursly like conformance testing. And that's why you're
getting such push-back.

cheers,
gja

---

From: John W Noerenberg II <[email-masked]>
Date: Mon, 28 Jan 2002 09:01:39 -0800

>the logo on their product.

Underwriters Laboratories, Inc.  is a non-profit corporation that was 
founded in 1894.  This 
<http://www.ul.com/about/otm/otmv3n2/labdata.htm>article describes 
the process UL uses for developing their standards.  Many UL 
standards receive ANSI certification.  According to the article, UL 
relies on information from a number of sources while developing a 
standard.

UL tests products submitted by its customers for *conformance* to its 
standards.  UL's reputation depends on the rigor and independence of 
their testing.  I don't know how it costs to submit a product for 
testing, but obtaining UL certification isn't free.  UL's 
certification program is successful, because when consumers like 
Valdis (and me) see a UL label, they believe in its value.  As Valdis 
points out, the value of the label has limits.

Certification isn't the work of a volunteer organization like the 
IETF.  It could be the work of an organization like Underwriters 
Labs.  This would be a good thing for Internet standards, imho.

One idea proposed multiple times in this meandering discussion is 
that those advocating testing should put up or shut up -- create a 
testing organization or move on to other topics.  I concur with both 
those suggestions.  I'm sure you'll all be pleased this is my last 
word on the topic.

best,
-- 

john noerenberg
[email-masked]
   ----------------------------------------------------------------------
   While the belief we  have found the Answer can separate us
   and make us forget our humanity, it is the seeking that continues
   to bring us together, the makes and keeps us human.
   -- Daniel J. Boorstin, "The Seekers", 1998
   ----------------------------------------------------------------------

---

From: Einar Stefferud <[email-masked]>
Date: Mon, 28 Jan 2002 21:25:34 -0800

Well now, an idea blinked on here;-)...

As Paul Hoffman noted, it costs a small fortune for an entire set of 
vendor products to be tested against all other interworking products 
(N**2 pairs is the estimate) and there is no proffered business model 
for doing this for the entire involved industry..

But, maybe someone can devise a business model for testing one 
product against all the others that claim to conform to the standard 
under test.

I know that HP did this ounce for their Internet products by hiring a 
person to do it from one of their customer's sites on the Internet. 
It does not matter here who or where it was done.

But, this puts the burden on the vendors that wish to be able to 
claim inter-workability with all others, or with some subset of their 
choice.

Or they can identify those that do not interwork for the benefit of 
those that want to know such stuff.

This then becomes an individual company decision, and does not 
require massed agreement, or require synchronized work schedules. 
Just put your system on the net and find someone out there to test 
against.  Doing it on the real net is just fine for this testing 
model.

Of course, the vendors that do this can brag or not, as they wish.

And here is no great concern for whether every vendor does it or not.

And the market can make up its mind by itself.

For my view, I have trouble believing that all those vendors are not 
vitally interested in inter-working among their products.

And, in addition, I would hope that someone might mount an open 
discussion mailing list for people to use to post their private 
experiences with what does or does not work.

And last:  This is no longer a useful IETF discussion;-)...\Stef

At 09:01 -0800 28/01/02, John  W Noerenberg II wrote:
>Underwriters Laboratories, Inc.  is a non-profit corporation that 
>was founded in 1894.  This 
>UL tests products submitted by its customers for *conformance* to 
>its standards.  UL's reputation depends on the rigor and 
>independence of their testing.  I don't know how it costs to submit 
>a product for testing, but obtaining UL certification isn't free. 
>UL's certification program is successful, because when consumers 
>like Valdis (and me) see a UL label, they believe in its value.  As 
>Valdis points out, the value of the label has limits.
>

---

From: John C Klensin <[email-masked]>
Date: Tue, 29 Jan 2002 19:17:26 -0500

John,

One addition to your description -- a small, but important,
point...

ANSI (of which both UL and the normative standards on which
their more detailed testing/evaluation standards are based are
members and accredited SDOs) makes a careful decision between
"safety standards" and other sorts of things.  By the
definitions they use, the IETF has never done a safety standard.
That is probably  A Good Thing.

The safety standards tend to be rigidly normative, specifying
exactly what is permitted and what is not.  There is no need for
our sort of interoperability testing, because things are
required to conform to an explicit set of specifications and
requirements.  And, where that isn't done, there are usually
requirements for approval by the local authority or use of
"approved" equipment.  In both cases, the term "to approve"
implies some sort of inspection or certification entity.

The characteristic that these things have in common is that they
are designed to be incorporated into legislation.  The National
Electrical Code, with which many readers of this list are
probably familiar (my apologies to those, especially out of the
US, who are not -- you already knew that we do things in odd
ways here), is a good example.  The Code itself is nothing more
than an ANSI Standard.  Conformance is voluntary, right?   Well,
the text is full of references to approved devices and approval
by local authority.  

Then various jurisdictions come along, take that "voluntary"
standard, and pass laws saying that it is illegal to do
electrical things any other way.  In the process, they specify
the local approval authority (your friendly neighborhood
electrical inspector in most cases) and the list of bodies that
can approve "approved" devices.  The latter is a list that
usually has only one entry on it, and that entry is UL.  If
Valdis can buy a non-UL-certified night light in Vermont, he
gets a choice.  In many jurisdictions, it is illegal to sell
such things, or household fire insurance is scrap paper if there
is a fire traced to a device without UL certification.  And, for
some devices, the codes themselves require that only certified
devices get installed.

Now, in our business, partially because we don't do safety
standards, we rely on external certification processes,
including a lot of self-certification, rather than these
elaborate drills that prevent selling or installing things that,
in the judgement of some organization, are non-conforming.
Stef's most recent notion of people doing their own
interoperabiity testing and announcing what they find if they
want to is exactly self-certification.  And it has been around
for years.  

But that brings us right back to where this series of thread
started: the company in question has never, to my knowledge,
made a loophole-free public claim that it conforms to anything
the IETF has produced, especially at the applications level.  If
they had made such a claim, and obviously didn't conform, then
someone might have a reasonable cause of action against them,
with or without a public announcement process.  But they are
doing exactly what they claim to be doing (read the licenses) --
delivering software that may or may not work and may or may not
be good for anything.  If one doesn't like that, one should
presumably go elsewhere or figure out why there isn't an
"elsewhere" and do something about it.

    john

--On Monday, 28 January, 2002 09:01 -0800 John  W Noerenberg II
<[email-masked]> wrote:

>> I have in my bedroom a night light, which I purchased at a
>> local grocery store.  It has a UL logo on it, which doesn't
>> tell me much about its suitability as a night light (I can't
>> tell if it's bright enough, or if it's too bright, or what
>> its power consumption is), but it *does* tell me 2 things:
>> 
>> 1) It has been *tested* and found free of any known safety
>> design problems. It may not *work* as a night light, but it
>> won't shock me when I go to throw it in the trash can because
>> it's not suitable.
>> 
>> 2) A high enough percentage of night light manufacturers get
>> UL listed that I can afford to be suspicious of any company
>> that doesn't have the logo on their product.
> 
> Underwriters Laboratories, Inc.  is a non-profit corporation
> that was founded in 1894.  This
> <http://www.ul.com/about/otm/otmv3n2/labdata.htm>article
> describes the process UL uses for developing their standards.
> Many UL standards receive ANSI certification.  According to
> the article, UL relies on information from a number of sources
> while developing a standard.
> 
> UL tests products submitted by its customers for *conformance*
> to its standards.  UL's reputation depends on the rigor and
> independence of their testing.  I don't know how it costs to
> submit a product for testing, but obtaining UL certification
> isn't free.  UL's certification program is successful, because
> when consumers like Valdis (and me) see a UL label, they
> believe in its value.  As Valdis points out, the value of the
> label has limits.
> 
> Certification isn't the work of a volunteer organization like
> the IETF.  It could be the work of an organization like
> Underwriters Labs.  This would be a good thing for Internet
> standards, imho.
> 
> One idea proposed multiple times in this meandering discussion
> is that those advocating testing should put up or shut up --
> create a testing organization or move on to other topics.  I
> concur with both those suggestions.  I'm sure you'll all be
> pleased this is my last word on the topic.
> 
> best,
> -- 
> 
> john noerenberg
> [email-masked]
> 
> --------------------------------------------------------------
> --------    While the belief we  have found the Answer can
> separate us    and make us forget our humanity, it is the
> seeking that continues    to bring us together, the makes and
> keeps us human.    -- Daniel J. Boorstin, "The Seekers", 1998
>

---

From: Mark Adam <[email-masked]>
Date: Wed, 30 Jan 2002 12:22:31 -0500

Since interoperability on a one-to-many scale would be a problem,
perhaps approaching it from the many-to-one point of view would be
better.

Einar's ideas are good, but still difficult to implement. What happens
when a company fails to find every device it should be tested against?
It almost seems that what we need is the concept of a reference
platform.

Having a reference platform allows for a single point of contact for
everyone wanting "IETF Certification".

I would also suggest that the task of implementing such a platform
should be up to the WGs creating the standards or the companies
authoring the standard. This would also give you a group that could
administer the platform. Of course there would have to be some rules of
conduct so that nobody could be excluded from performing their
interoperability testing. (Do I smell a BOF here?) I'm sure groups
holding reference platforms could find some way to make money off of
this without breaking the rules.

I'm not saying this would be easy to implement, but it might be worth a
thought. 

mark---------------