Subject: Re: Yes, conformance testing required... Re: Fwd: Re: IP: Microsoft breaks Mime specification

From: "Kyle Lussier" <[email-masked]>
Date: Sat, 26 Jan 2002 23:09:53 -0800

> Your process for yanking a logo requires a vendor's implementation to
> fail an interoperability test against a known standards compliant
> implementation. Anything less would make the logo meaningless. That
> smells dangeoursly like conformance testing. And that's why you're
> getting such push-back.

Well, this comment is undoubtedly going to cause some more 
push-back. :)

I seem to be getting two conflicting viewpoints: 

  #1 Vendors can only be trusted to be interoperable on their own,
     and can not be forced to conform.

  #2 Vendors absolutely can't be trusted to be interoperable,
     without conformance testing.

I guess everyone approaches things in different ways.  

And that's why I made the proposal.  Because this idea works with
either viewpoint.

Personally, in this particular kind of massively distributed, diverging 
objectives scenario, I say "trust everyone to do what's right"
and then use the logo yanking process to (1) identify ill behaving
vendors / products, (2) give them double reasonable opportunity
to correct, and then in the absence of any good faith effort
(3) publicly (but nicely) flog them by yanking the logo.

Trust everyone to do what's right.  Reward the people who do the
right thing (by allowing them to use the logo).  And people who
do the wrong thing can lose it.

I'm not really a believer in conformance testing, because the
space of the Internet is so rapidly evolving, anything you
test against is a moving target, and because something conforms
at one point, it may not next week.  I think that sentence addresses
the majority of "problem-type" criticism the idea has had.
I am absolutely on everyone's side and agree with everything
posted as such.  Everyone has listed problems, but no one has
said they can't be worked around.

I'm just looking for a solution that creates significant, immediate
benefit for people who try to follow standards.  And when bad
vendors come around and start doing bad things to hurt interoperability
(an incredible benefit to customers, consumers, you name it), the
IETF makes it easier for

Mostly, I'm looking for some level of easy-in product segmentation
for contractual, customer visibility, and CIO empowerment type things.

If you are a vendor, and your customer gets pissed at you and says
you aren't being a good vendor, and you said you would be, it
gives them an angle to push.  A slow, bureaucratic one, but a way
to lead vendors, through reward, to do the right thing.

Kyle Lussier
AutoNOC LLC

---

From: grenville armitage <[email-masked]>
Date: Mon, 28 Jan 2002 10:52:12 +1100

Kyle Lussier wrote:
	[..]
>      without conformance testing.

Kyle, in all kindness, you're missing the most fundamental
viewpoint expressed here recently: The IETF isn't the place,
nor is it the organization, that could or should take on the
role of interoperability-cop.

cheers,
gja

---

From: Paul Hoffman / VPNC <[email-masked]>
Date: Sun, 27 Jan 2002 10:27:35 -0800

>      without conformance testing.

Missing is "#3  Vendors can't be trusted to be interoperable without 
interoperability testing." VPNC performs conformance testing for 
IPsec, and there are plenty of examples of our members who conform 
but do not interoperate without a lot of knob twiddling and an 
occasional bug fix.

The long experience with IPsec interoperability events (which VPNC 
does not hold) has shown that A, B, and C might conform, and A and B 
can interoperate fine, but A and C cannot interoperate. This is 
usually due to administrative interfaces either not having the right 
knobs, the defaults for C being valid for conformance but not for 
interoperability, or weird magic.

Is interoperability testing needed for end users? Possibly, but it 
won't happen until someone comes up with a good business model for 
the testing agency. When anyone comes up with one, I'd love to hear 
it. VPNC was originally formed to do good interop testing for the 
IPsec industry, but when we figured out what that would cost all of 
the members, there was no longer any interest. The basic problem: 
either each of the 35 members is responsible for running and 
debugging the test with the other 34 members, or they are willing to 
pay someone to run and debug the 1225 (35^2) tests for them. In the 
former case, the best statement of why that was not attractive was 
"if I have a staff person who has that much skill with our product 
and the at least 100 hours it will take, I have much more important 
work for them". In the latter case, there was immediate history of 
another interop testing agency who both charged a large amount of 
money to do the tests and a fair amount of vendor staff time to do 
debugging in order to do about one fifth the number of tests. Thus, 
VPNC is left doing conformance testing with verifiable results, which 
is admittedly not nearly as valuable to end users. (See 
<http://www.vpnc.org/conformance.html>, particularly near the end, 
for more details.) VPNC also does some small-to-medium sized interop 
demos, but these are not formal interop tests with formal results.

Of course, this is not to say that formal interoperability testing is 
impossible. There are examples of where it happens today in the 
Internet industry. But there are probably one or two orders of 
magnitude of examples of where it does not happen. Informal 
interoperabilty events, where there are no results published but lots 
of good interaction between vendors, have helped the industry a great 
deal but are largely invisible to end users (and still don't produce 
the level of interoperability that people in this discussion say is 
"required").

Does the IETF or ISOC want to get into either conformance or 
interoperability testing? It is fairly safe to say this not going to 
happen without a business model to pay for the short-term and 
long-term costs. So far, no business model has appeared in the 
discussion.

--Paul Hoffman, Director
--VPN Consortium