A Pocket Guide to Beta Testing – Part 1

A couple of years back, a team I was coaching wanted to organise a Beta testing phase for their newly developed product. Since I personally never set-up such an activity, I created a guide for them based on material I found online plus company specific information. Now, as a product owner, I want to try out for myself if what I provided back then actually works. So I thought I’d share some of my old findings and check how they hold up today.

Based on the resources I used, I identified four stages of Beta testing and created a questionnaire to guide my readers through it.In this blog post you can find out what I came up with for the first stage, Setting up activities.

Before starting the Beta phase, try to define its scope. Question the purpose of the testing as well as the timing and duration.

When should my product be Beta tested?

If the participants have a bad first experience with the product, your chances of getting them to continue putting in effort, will be much lower. Thus, it is recommended that you have some kind of readiness checklist before starting.

• The development team has verified that all of the components of the product are ready, so that Beta testing can begin.

• Auxiliary components (documentation, etc.) have been assembled into a single package which represents exactly what will be sent to testers.

• The out of the box experience has been successfully reviewed, including setup, installation, and supporting documentation.

• Basic product functionality has been successfully reviewed (all key features are working) by product management.

• Known bugs which could not be addressed prior to Beta have been clearly documented and communicated to testers.

• The uninstall process (if applicable) has been verified.

Setting appropriate goals: What do we need to learn?

To conduct a successful Beta, you need to decide beforehand what you want to accomplish by doing this activity. The questions below can help you crystalise your goals and expectations.

• Do you need a few customers to validate readiness with some hands-on customer usage?

• Do you want to discover corner case bugs?

• Do you want to check interface acceptance?

• Do you want to test product functionality in real-world environments?

• Do you want to test support infrastructure (documentation, customer support etc.)

• Do you want to collect feature requests?

• Do you want to test production infrastructure (alerting, error logging etc.)?

How long should the Beta testing last?

It is generally recommended that tests are no shorter than 2 weeks and no longer than 12, with most Beta tests having between 4 and 8 weeks of test time.

How many Beta testers do I need?

There are some factors to consider when deciding how many testers you need for example

• the length of the test

• the goal of the test

• the size of the system under test (one new feature vs a complete new product)

When deciding on the number of testers, keep in mind that some estimates report that only 20% of Beta users send back useful information*. To get back substantial feedback you will need for

• Consumer products: 50 – 200 people

• Business products: 10 – 50 people

These were the questions I thought could help a product begin with Beta testing. In an upcoming post, I will add the questions for the second stage, Onboarding the Testers, focusing on giving the right amount of information to the testers to get started.

Resources

• Instabug Beta Testing Blog

• Centercode Resource Library

• Joel Spolsky’s Top Twelve Tips for Running a Beta Test

• Brian Lawley’s How to Run an Effective Beta Program

*I have looked to find these “estimates” but my research hit a wall. If anyone has a good source, please let me know.