A Pocket Guide to Beta Testing – Part 2

A couple of weeks back I posted a blog on how to set up a Beta testing phase, based on resources I found online a couple of years back. In this post I continue with the second stage of Beta testing, Onboarding the Testers, focusing on giving the right amount of information to the testers to get started.

Beta testers are investing their company’s and their own time to provide feedback for your software. The chances are that the easier it is to get started using the system and the more seamless it is to continue using it, the more information you will extract from their testing. When preparing your onboarding documentation try coming up with answers to the following questions that your testers might ask.

How can I start using the system?

Provide information about the system to use, log in credentials, registration specifics, role required etc.

How do I log a bug?

Provide link to the bug tracking system and log in credentials if required. If the system is a special tool (in contrast to let’s say plain email) include a link to the documentation or do a demo before the testing phase starts.

Put a bug report template in place to extract the information that would make the issue reproducible.

Bug report example layout

Title: Short description

System Information: Device, operating system, browser etc.

Perceived Severity: Very high, high, medium, low

Actions: Describe the steps that led to the appearance of the bug

Expected Result: What you expected to happen

Actual result: What actually happened. Add screenshots or short recordings if available

How do I submit a suggestion or feature request?

The tester might have suggestions on how to improve an existing functionality or to request an additional missing one. Having a different template for such requests first of all indicates that the tester differentiates this from a bug in your system. Additionally, it helps sorting and prioritizing the feedback received.

Suggestions example layout

Title: Short description of the new feature

Urgency: As soon as possible, Near future, Later

Relates to: The part of your software that is relevant for the request

Use Case Description: Let the user describe what they want to accomplish with the new functionality.

How long should I use the system? Does it need to be all in one go or can I do it at my own pace?

You should set the time frame of testing depending on your goals. For example, if you need to monitor the productive system under stress you should have all the testers using it at a specific point of time. If on the other hand, you want to collect information about its overall quality you can allow the testers to do their activities on their own pace.

How can I get help?

You can decide whether you want to include a link to the documentation right away during onboarding or exclude it so that you can get the feedback on how easy it is to find. Nevertheless, make sure that in the onboarding you establish communication channels between the testers and the development team. If the testers can’t get an answer fast, chances are that they will stop testing and be frustrated with the experience and eventually your product.

By having answers to these questions I thought I could provide a solid starting point for the testers to try out the product. In the next post of this series, I will add the questions for the third stage, Conducting the Testing, giving some guidance on how to get the right information from the testers.

Resources

• Instabug Beta Testing Blog

• Centercode Resource Library

• Joel Spolsky’s Top Twelve Tips for Running a Beta Test

• Brian Lawley’s How to Run an Effective Beta Program