When to not use an online survey

– posted February 21st, 2008 by Laurence Veale Comments (7)

The Center for Universal Design, North Carolina State University provides this example of where a survey might not be the best idea, particularly in light of the actual insight it will give them.

Q1: Please tell us where you are from?

Question 1 asking What country are you from?

Q2: Please tell us the name of the country where you live

Question 2 asking What country are you from? in a different way

Thanks!

And that's it. Essentially just one question

All they want to know is what country you are visiting from!

Analytics anyone?

A decent analytics package would give far more accurate information, without inconveniencing the visitor. The screenshot below a map overlay from our Google Analytics account.

Map overlay on Google Analytics

Digg this post Submit to del.icio.us Submit to technorati

Categories Spotted, Usability, Web analytics

7 comments so far

1. Michele on Feb 21st, 2008 - 13:39

Err no. Analytics won’t tell you accurately, as many users are on corporate networks that report themselves as being country X, while the user is actually in country Y. The same happens with satellite providers

2. Brian on Feb 21st, 2008 - 15:24

@Michele — But how much more accurate will a survey be? At least with analytics you’re sampling virtually everyone, whereas with a survey you have an unknown selection bias (who tends to answer these surveys?).

The point isn’t that analytics is 100% accurate — nothing is. The point is that it’s probably much more accurate than anything else (at least certainly more so than an unrepresentative survey), and you don’t need to bug your users to get a good indication of the answer.

3. Laurence on Feb 21st, 2008 - 17:13

That’s it Michele! I’ve had enough. We’re moving everything over to Joyent

4. Brad C on Feb 25th, 2008 - 17:46

Just to play devils advocate for a minute. Maybe they were looking for visitors from certain geographic locations and if you were from one of them there may have been more questions.

Although if that’s the case, it’s unfortunate that they wasted your time.

5. Lar on Feb 28th, 2008 - 12:42

Hi Brad,

thanks for dropping by. I had tested all those possibilities.

If you’re US, then it’s just state and ZIP they ask for, otherwise it’s just country.

I’m not too worried about my wasted time, just pointing out that there are better ways of getting that information without inconveniencing their visitors, i.e. analytics

The center for UD is highly regarded internationally so I would have expected more from them.

6. Tom Doyle on Mar 7th, 2008 - 09:41

As much as I think the survey is a ridiculous idea, I would tend to agree with Michele that analytics are going to be just as inaccurate.

Anaytics packages are not an accurate way of measuring geographics of a websites users at all.

The most obvious example of this for me is AOL users. An Irish AOL user is likely to be regarded as being from the US.

For the moment I think we just got to take geographic statistic with a pinch of salt!

7. Colm on Mar 7th, 2008 - 10:42

@Tom: Geo data from Google Analytics tends to be quite accurate, it can even tell you what exchange people are connecting on from ISPs.

As for AOL - they wouldn\’t be sending their traffic from Ireland to the US - it just wouldn\’t make sense both in terms of performance and bandwidth, instead they would piggyback on the local providers albeit with their own IP address range.

The one advantage of a package like GA over some of the other analytics packages is the wealth of geoinformation Google has - with everyone in the world using google they\’ve got presenting a localised homepage down to a tee and can transfer this information to the analytics end of things.

8. Tom Doyle on Mar 10th, 2008 - 16:21

@ Colm: With due respect…

AOL routes all it\’s traffic through proxy servers (unless configured otherwise).

Generally AOL users will appear to come from the US, however they could possibly look like they come from France, Germany, Japan, Australia and Brazil. An AOL user to report as being from Ireland, is unlikely.

An AOL user could also possibly appear to be coming from a number of different locations because it gives the member a multiple IP address for a session.

It\’s also not likely to report the exchange people are connecting at accurately, because as per above, they could appear to be coming from numerous locations.

So all in all, the statement that Analytics would be more accurate than a web survey is very debatable, no matter how impressive GA is in other areas.

Tom