The trouble with focus groups

— Published June 4th, 2006  |  by John Wood

Yet some organisations continue to use focus groups to test the usability of their websites. It seems that the limitations of this technique are still not as widely understood as they should be.

Recently, we had the opportunity to participate in a project in which the client used both focus groups and user testing to test a new and very expensive website. This gave us a rare opportunity to make a direct comparison between the results produced by both techniques.

How the techniques were deployed

The focus groups used in this project each involved a facilitator and six users of the website. Each user had access to the website on a PC and were guided through a series of activities by the group facilitator. The session was videotaped and the reactions of the participants were analysed and incorporated into a report.

The user groups that we organised for the project each involved one user of the website. This user was asked to attempt some typical tasks on the website by a facilitator, while an observer watched the test and recorded the findings. Each test was also videotaped and the findings were incorporated into a report.

So what’s the difference?

On the surface, the only difference between these techniques seems to be that the focus groups involved six people at a time whereas the user tests involved only one. In fact, the focus group team referred to the user tests as the ‘one-to-one interview technique’ throughout the project.

But the real difference between the techniques is in the data recorded and how it is analysed.

In a focus group, the facilitator is interested in what people say. They show them the website and let them use it and ask what they thought of it. These reactions are recorded, analysed and presented as findings. In contrast, user testing is about watching what someone is doing rather than listening to what they are saying. We see what users do, analyse what we saw and report these observations as findings.

What people say and what they do may differ

The key strength in user testing -- and the main weakness of focus groups in this context -- is that there is often a disconnect between what people do and what they say. In fact, if you sit through even a few user tests you will learn that what people say and what they do are often directly contradictory.

It is not unusual to watch someone struggle to fill in an online form for 15 minutes, watch them accidentally delete their work and start over several times, and then hear them say that they had no problems and found the form very easy to use. You soon learn not to take what people say at face value.

Why is this so? Well, there are lots of reasons. People are naturally polite and are at pains not to offend anyone they associate with the website by criticising it too harshly. People also tend to assume that any problems they are having are their own fault, and not the fault of the website’s design. As a result, people often berate themselves for their own stupidity during tests, when in fact the design is simply terrible. For these reasons and many more besides, asking questions and recording the answers is too simplistic an approach.

Observation yields better results

When the focus group and user test were completed, the differences between the techniques was clear.

The focus group report reflected consumer reaction to the website: the specific likes and dislikes of the test group. By concentrating on the expressed opinions of the test group, the focus group identified a number of problem areas in the website. But user opinion was often contradictory, and the precise causes of issues, as described in the user's own words, was often vague.

In contrast, the user testing was able to bring a lot more precision and analysis to the table. The report identified which features had problems, why they were problems and what the client should do to fix them. By closely observing what users were actually doing, rather than relying only on what they were saying the user test report offered much more detail and was more directly actionable.

So, having gone through this exercise, we would concur with Nielsen's opinion that focus groups have their place. We are certainly less negative about the merits of the technique than some of our industry peers (as in the "Shoot the Focus Groups" Business Week article), but what is clear is that focus groups are unquestionably not suited for usability evaluations.

If you want to know if your users can use your website or not, don't just ask them. The answers you get will be vague at best and may be directly misleading in the worst cases. You need to watch what they do -- you need to user test.