As tech lead of Cloudera Altus Director, I’ve often been asked whether some product feature, or use of a particular version of some external software, or use of some external service – some it – is supported. I found that it’s really hard to answer these questions. There are different factors involved, like how much testing we’ve done or continue to do, or what we recommend to customers, or say we’ll let them at least get away with. The answer is rarely a binary, absolute yes or no.
So I spent some mental cycles thinking about the question of whether something is supported. Thinking about the different angles, I came up with a hierarchy of support levels. It’s certainly a “1.0” sort of artifact. But, I think it’s a good starting point, and it already helps me answer “Is it supported?” questions today.
Level 0. It doesn’t work
This is pretty much a “no”, to the extent that we know it will cause a failure. Maybe we’ll support it someday.
Level 0.5. It might work by accident
This doesn’t really deserve its own integer designation, because from a point of view of expectations it’s equivalent to Level 0. In practice, it might happen to work by pure luck, or because it doesn’t change much from what works before. It could be assessed and then elevated to a higher level of support, or more likely it could lead to a small amount of engineering work instead. Until then, though, you can’t say it’s reliably supported in any sense.
Level 1. Implementation for it is done but it might not work
You wouldn’t stay here long. This means that work was done for it but it is essentially untested. A “no, but check back later” response.
Level 2. It has been tested to work
This is the first level that approaches a “yes” answer. There is implementation work in place, and developers and those mythical, unicorn-like QA people have tested it, either manually or automatically or both, to see it functioning correctly.
This is not an uncommon stopping point, but it’s not up to the standard we should aspire to for a “supported” thing.
Level 3. It is documented and/or stakeholders have been trained
Documentation is really a requirement for support. If you don’t publicly talk about something, then it’s going to remain a hidden feature that will slowly rot from disuse; customers, or your own conscience, won’t force you to keep it working.
Along with documentation, stakeholders like support and field personnel need to understand it, so that they can use it, explain it, and fix uses of it. The understanding of support personnel seems like a prerequisite to claim that something is supported.
This is the first point at which you can answer “yes”, while it’s still kind of a “yes but”.
Level 4. It is approved by all stakeholders
It works and is tested, and information about it is spread around, and even better, everyone agrees that it works properly and in a “supportable” fashion. To get to this level, engineers have to have feedback from support personnel especially that it is not only working, but something that they can deal with.
At this point, I think you can confidently say “yes”, although it isn’t wholly satisfactory. Still, most features land here.
Level 5. It is continually tested
You could argue that this level belongs at a lower number in the hierarchy, but I think most features don’t get to enjoy constant testing. There are only so many permutations of a product that you can test on a continual basis, and in the face of finite testing resources, not every feature makes that cut. The important ones do, and you have to pick them. The rest stay one level down, where your answer has the unfortunate caveat of “it has worked in the past whenever we’ve tried it.”
Disregarding where continual testing should fall in the hierarchy, as it stands this is the nirvana of the hierarchy. Still, you need to watch those tests, or else it might fall back to level 4 without you realizing it.