After a small break, I continue with the final part of my Beta testing findings, from some research I did a couple of years back. This one is about how to guide your users through the Beta testing phase and how to ramp up the activities at the end.
Once the Beta testing phase has started, it will be critical that you communicate regularly with both the participants and your internal team. If the participants hear from you regularly they will be reassured and be much more likely to continue putting in time and effort to help you with the product.
Most beta testers will try the program when they first get it, and then lose interest. Keep your beta testers motivated! On the other hand, try not to overwhelm the testers. Limit the surveys to at most 1 per week, always having less than 20 questions.
What kind of questions should I ask my testers?
You can address at least 2 types of questions to your testers.
-
Multiple choice questions
-
Open end questions
With multiple choice questions, you encourage your testers participation by making it easier for them to do so. Try to quantify your testers answers on a numerical scale of 1 to 5. Here is an example question for inspiration.
-
Did you find the instructions and messages given on the computer screen helpful?
-
Following the instructions, I ended up in dead-ends
-
I had trouble understanding the instructions
-
When I read the instructions carefully I was able to proceed with my tasks
-
The instructions helped me get through my tasks
-
I barely noticed the instructions as everything was very intuitive
-
On the other hand, some of the information you seek can’t be quantified. Start with a general aspect you want to know more about, break it down into identifiable parts, and ask about those parts. For example,
-
For which of your daily tasks is the app most useful to you?
-
Which of the app’s features do you find the most useful?
-
Which of the app’s features do you find the least useful?
How can I get information from the testers while I sit next to them?
Let the testers interact with the system in their own way and note down the paths they use in order to achieve their goals. Even though you might be tempted to show them all the cool features of your application just sit back and see if the testers can find them on their own.
One key feedback aspect that cannot be easily communicated through a questionnaire, is the testers’ emotional experience of using your software. Note down if they feel frustrated when they cannot accomplish a task or they feel that with the documentation at hand they can continue with their work.
You may even want to go their company if they are local and watch them get up and running so that you can figure out where any of the likely bumps in the road might be for other participants.
What should we do internally during the beta testing phase?
Make sure to communicate periodically internally with your team. Provide a weekly status report including number of bugs reported, usage by participants, whether you are meeting the stated goals and what you need from the team to continue to be successful.
It’s critical that you keep the team informed so that as the beta program gets ready to wrap up everyone knows the status and there aren’t any surprises or disconnects about the results.
The beta testers will be eager to provide feedback in case the perceive something as a bug. You want to reply to them about the validity of the claim, a potential fix or its prioritization if is a missing functionality. Make sure that you filter out the noise, so the tickets you have at the end of the testing phase are valid and prioritized items.
Monitor your system during the beta phase. It is a great opportunity to test your alerting, monitoring and logging mechanisms. The testers might interact with your app in different ways that the development team giving you a preview of the system’s behaviour in real life scenarios. You can see how the system recovers from alerts and recover from failure and take preventive measures to improve your process.
Finally trust your testers. They are the people using the system so if they think that something is a bug pay close attention to why they make the claim. Is it because the system is not doing what they expect it to or because they did not understand the correct way to complete the task. After all
A bug is anything that threatens the value of the product.
Bach & Bolton
When you are ready to end the beta phase the participants complete a short exit survey. One of the questions is usually the Net Promoter Score (NPS), a metric commonly used throughout the market measuring customer experience and predicting business growth. It sums up to a question along the lines as it is shown below:
How likely are you to recommend the app you tested to other colleagues?
From a 0-10 scale, those who respond with a 0 to 6 are referred to as “Detractors”, while people who reply with 7 or 8 are called “Passives”, and 9 or 10 are “Promoters”. Using an nps calculator you can then check your program’s score.
Take the results of the survey and deliver a final report to your team. This can include a short summary, bug trend information, whether you met the original agreed upon goals and a summary of customer opinions and feedback. Most importantly give your team members the credit they deserve for pulling off the program. Deliver this report to the team prior to making the decision to ship the software – it will prove to be a useful and fact-based tool to make an informed choice.
Finally, document the experience, include what worked well and what didn’t and get ready for the next round of customer testing.
More Questions Than Answers 
One thought on “A Pocket Guide to Beta Testing – Part 3”