Recently I released a product to one of my customers for a preview (e.g. Alpha Release) and not too long after I starting getting calls with complaints regarding incomplete functionality... As you can image I stated it was Alpha and I might as well have been speaking elvish because they just didn't understand. Lucky for me, this term was spelled out pretty clearly in the Software Requirements Specifications (SRS) I wrote at the beginning of the project which we all agreed to. This type of thing is crucial and you really need to define threshholds for your releases in writing and do as much as possible to make the definitions comprehensive. Over the years I have found a lot of different definitions for caveats such as Alpha, Beta, Candidate Release, Etc... However you define it, write it down and make sure everyone agrees. Here is what I wrote in my SRS...
PRODUCT TESTING AND SOFTWARE PRERELEASE PROVISIONS
Testing shall occur using a test team coordinated by the customer as well as internal testing coordinated by the software development team. The team shall test the product(s) and evaluate its functionality and capabilities as they apply to the current state of the development life cycle. The pre-release test group(s) shall provide information concerning software changes (enhancements and defects) through the Configuration Control Board (CCB) and test event hot-wash meetings as applicable. For details on project test procedures and methods, refer to the project Test Plan. The following pre-release testing phases are defined for this project:
- Alpha Release testing is the first phase of testing in a software development process. This phase includes unit testing, component testing, and limited system testing. This is generally an internal test cycle and is accomplished using an Alpha Release of the product.
- Beta Release testing is the phase of software testing in which a sampling of the intended audience uses the product(s). This phase of testing may also include a number of specialized tests designed to identify areas of the product needing improvement (Ex. Security, Performance, Scalability, etc…). Beta testing is considered "pre-release testing" and is accomplished using a Beta Release of the product.
- Candidate Release testing (A.K.A. Production/Acceptance Test Releases) are software releases prior to the final production of the software. In this phase of testing all stated requirements should have been met. The customer’s acceptance testing is often conducted with this release and is accomplished using a Candidate Release of the product.
The aforementioned external tests are designed; among other things; to verify and/or validate the products development. For the purposes of this document, verification is meant to answer two primary questions with respect to the product being tested.
- "Are we building what we were asked to build?"
- "Are we building it right?"
Verification is accomplished as part of unit testing, integration testing, systems testing, Function Qualification Test (FQT), and more. For the purposes of this document, validation is meant to answer two primary questions with respect to the product being tested.
- "Are we building what the operator needs?"
- "Are we building the right thing?"
Validation is primarily accomplished as part of acceptance testing although valuable insight can be gained by including a diverse set of end-users in Beta testing.
The provisions below identify the scope of each pre-release product and the test phases they support.
Alpha Release
Alpha releases are software releases made prior to any Beta release or Beta test phase. This release of the product is not yet stable and will lack many features. Alpha software is only released to provide for targeted analysis, design reviews, prototyping, or early product demonstrations. Alpha releases shall not be used to fully assess any level of compliancy to stated requirements or goals; however this release may be used to validate the approaches selected. Alpha releases are internal releases and should never be released to third parties. After some testing and some revision, the product will assume beta status.
- Alpha Test Phase
Alpha releases are versions of a new product still in development and represents the first phase of testing in a software development process. This phase includes unit testing, component testing, and limited system testing. This is an internal pre-release test cycle. The first Alpha release marks the beginning of the Alpha Test Phase.
Beta Release
Beta releases are versions of a new product still in development, which is provided to external operators for testing. Beta releases are not produced until sometime after bug convergence has been observed. Bug convergence is the point at which the development team has made measurable progress against the active bug count; which is the point at which the rate of bugs resolved exceeds the rate of bugs reported. Many bugs are usually eliminated at the earlier Alpha stage of development and testing, however bugs will still be present in Beta releases. Beta testers use the software to do real work and report any bugs or badly implemented features they find. Beta releases may be used to assess levels of compliancy as it is related to the stated requirements and/or goals; however, it is often limited to a subset of requirements for which the beta was released. While Beta releases are; by definition; releasable to the “public” they should only be distributed to a limited number of expert operators in the earliest phases. Distribution of beta versions allows operator testing and feedback to the developer, so that any necessary modifications can be made before final product release. The term Beta release is given to a product that is not ready for public consumption, but is good enough for a wider testing scope. By the end of a beta test, all major bugs should have been discovered and repaired. Generally, beta testing is considered to be the final pre-release stage of the tests, and includes experienced testers external to the developing organization.
- Beta Test Phase
The Beta test release marks the beginning of the Beta Test Phase. Beta Testing is the second phase of testing in which a sampling of the intended audience tries the product out. Beta testing is considered an external pre-release test cycle.
Candidate Release
The Candidate Release (A.K.A. Production/Acceptance Test Release) is a software release made prior to the final production of the software. In this phase of development all stated requirements should have been met or mitigated to a final status. Candidate Releases are not produced until the project has observed a Zero-bug bounce. Zero-bug bounce is the point when development has resolved all bugs raised by testing and no active bugs remain. The customers acceptance testing is often conducted with this release, which may include a wide range of testing models (e.g. usability, security, performance, etc…) needed to fully validate the product. This is still considered a pre-release product pending final release approval from the customer.
- Production Test Phase
The Production test phase is the final phase of testing prior full release of the product in which the customer performs final acceptance testing. Particular attention is placed on media (re)production methodologies, software installation routines, and documentation.
Tags: requirements, processes