Beta Testing

I'm sure as you've cruised around the web you've run across the
term "beta release" or "beta test version" or something to that
effect. You may also have heard "alpha test" or "public preview"
and other similar terms. What the heck do these terms mean?

All well managed product development projects are split into
multiple phases, each distinct and each with it's own goals.
Generally you begin with a proof of concept, request for proposal,
short analysis or things like this. You follow this short cycle
with a longer analysis, then a design specification, and finally
you implement the program. This consists of the actual coding of
the programs and might include documentation as well.

Now you've got a finished project, and what you are supposed to do
next is a quality assurance step. Generally, you want to test and
test and test until the program (or application system) exactly
meets the design specification (which should be treated as a
sacred document) and associated standards manuals. It is critical
to note that testing is done against the specification and
standards manual and nothing else at this phase of the project.

When and only when you complete this testing and the product 100%
does what it was designed to do, then you create what is called an
alpha test. This consists of sending the code to a select group of
users who will pound on the product. Their job is to ensure that
the product works in the field under conditions outside of your
lab.

Once your product has been tested by some alpha testers, you may
want to take some time to correct any issues, then perhaps send it
out to the alpha testers again. You might repeat this step one or
two times until the number of bugs found is significantly reduced.

Your next step is to widen the group of testers greatly. This is
called a beta test, and in it's purpose is to ensure your product
works on an even wider base of computers. The theory is you cannot
duplicate all conditions in your lab, and the alpha testers are
purposely kept to a small, very manageable group.

Beta testers are generally just a number of users who, in exchange
for a reduced fee or a free copy of the software, agree to install
the program (or system) and use them. They are to report any
errors, and understand that the program may have bugs, can crash
and might corrupt data.

A program or system might go through one, two or even three rounds
of beta testing. The developers would be in constant communication
with the testers so as to produce the best possible product.

This is the way it worked, and this is the method the better
product managers still employ. I've been a beta tester many times,
most seriously a dozen years ago with operating systems such as
RSX, RT11, RSTS/E and OpenVMS. One of the major reasons why the
Digital Equipment Corporation operating systems were so good (and
in many ways are still unsurpassed) - they really knew how to run
a solid series of beta tests.

The original concept was very simple, but large companies such as
Microsoft have corrupted it until now Beta testing is not really
testing. You see, Microsoft has changed Beta testing to really be
a "pre-release" or "public preview". What does this mean? It means
Microsoft has the option to deliver unfinished or inadequately
tested code to tens of thousands of users.

Microsoft does not tend to go through a lot of trouble to get
feedback from this vast army of beta testers. Oh yes, they do
surveys and send forms, but to do a real beta test you need to be
in firm control and be sure your testers are doing something
useful.

You see, beta testing is supposed to be part of the development
process, not the marketing and promotional process. And,
truthfully, that's one of the primary reasons why Microsoft (and
other large companies) have had such poor quality assurance in the
past few years.

So, basically, beta testing helps ensure that a product is more
reliable and works in environments outside of the lab. And that's
all it is supposed to do.