The Pros and Cons of conducting Usability Testing with multiple participants at the same time

Our lead Usability Testing course trainer Cathy Carr recently encountered an interesting question from one of her attendees: ‘Would you ever test with more than one participants in a single session?’. The short answer is ‘yes’. Surprised?

Traditional user testing is conducted with a single participant. The participant is given specific scenarios to carry out on a product or service, while their actions and behaviour are observed by a moderator.

Mobile-Tablet_Usability-testing-rig_user-testing

A ‘think aloud’ dialogue is encouraged, where the participant verbalises their thoughts and actions as they go about completing scenarios. They may talk about what they are looking at, thinking, doing and feeling.

The ‘think aloud’ dialogue gives moderators insight into the cognitive processes used by the participant to complete the task. ‘Think aloud’ may uncover expectations, misconceptions or misinterpretations of the service and how these came about.

12801178_10156603118360230_4756248782288016946_n

In paired testing, is carried out with multiple participants (usually in pairs) simultaneously. The two participants work together to complete the scenarios asked of them. An open and free dialogue between participants is encouraged, instead of the ‘think aloud’ dialogue.

Just like any methodology, there are pros and cons of paired testing:

Pros:

  • Participants can feel more at ease, helping them relax and talk more comfortably. This is particularly relevant when testing with people that know each other well (e.g. friends, siblings, partners, family members) and share an interest that relates to the experience you are researching. This can be really effective when testing with young children or conducting research on more sensitive topics.
  • Less probing and prompting is required from the moderator, enabling them to remain neutral – participants are likely to expose more in the conversations between each other than they do with the more traditional ‘think aloud’ approach.For example, by listening to the problem solving conversation between the participants it becomes clear where the issues are without requiring probing.
  • You’re likely to identify more usability issues when testing in pairs, enabling you to identify more usability issues in a shorter timeframe.

Cons:

  • With two people you have more to observe, more data to capture and therefore potentially more to miss. Note taking in paired sessions requires more effort and skill from the moderator. It strengthens the case for employing a separate note taker.
  • Similar to the limitations of focus groups, you may have one participant dominating the session, which can mean you miss the inputs and insights from the second participant.
  • Participants cooperating with one another can feed off each other’s ideas and observations. They may both achieve more than they would do individually. This can result in false positives.
  • You may be creating a more unnatural setting, especially if the two participants are strangers; this may limit their responses or even cause them to adapt their behaviour.You need to ensure that you fully understand the natural context of use of the product or service: if it’s generally used by multiple users then paired testing is valid. If this is not the case, then individual user testing sessions are recommended.
  • It may put extra strain on recruitment: recruiting participants that match the recruitment brief can be tricky enough on its own, never mind adding the additional requirement that two participants need to be available at the same time.

When to use paired testing

  • When conducting usability tests with young children, as they feel more comfortable discussing their experiences with one another. The conversations between the children and the language they use to describe a product or service can provide better insights than you may have gathered otherwise.
  • When testing services that are inherently multi-user focused, in order to understand the experience from the various viewpoints e.g. multiplayer games, chatting services, social networking services etc.
  • When testing a service which covers a sensitive subject, as enabling participants to have conversations between themselves can uncover insights that may have been impossible to capture otherwise.
  • When scenarios that you initially think of as being completed by a single user may actually involve or be heavily influenced by multiple users. For example:
    • Interacting with a TV interface has a primary user (the person with the remote control) but may also have multiple secondary users, such as the other people in the household who all take part in the decision making of what to watch next.
    • Buying a holiday usually involves the primary user completing the purchase on a website but often this is done after much deliberation with the other secondary users.
  • Occasionally constraints around recruitment, participant availability and scheduling mean that having multiple participants within one session is the only way to gain access to the full set of participants. However, this should only happen in extreme cases. 

Want to know more?

Bunnyfoot has a wealth of experience of conducting paired testing in lab- and field-based settings so, if you’re considering it as an approach for testing your product/service, get in contact and we can talk through your requirements.

Leave a comment

Your email address will not be published.

Read enough? Get in touch...

Contact Caroline Bentley to discuss your needs:
0207 608 1670 more@bunnyfoot.com

Or come visit us, we have offices in Oxford, Sheffield and London.