Two examples: Remote user testing in action

I’ve been wanting to write about the challenges of remote user testing since UXMag reviewed the book Remote Research last spring. I’ve long talked about the advantages of remote research as part of a testing plan, especially a low-budget plan.

The book is full of excellent guidance, and I recommend it. But I have a couple of cautions about the authors’ advice, which I’ll address as I share a examples of testing we’ve done.

1. Using surveys to conduct remote tests

The authors say that surveys are ineffective for remote testing. That’s absolutely true if you use them the way surveys are usually designed — typical survey questions don’t give insight into actual behavior.

But that doesn’t mean they can’t. I’ve participated in and used surveys to collect very useful user testing-style data.

I first got the idea of testing this way taking surveys sent by the Washington Post. They use the survey format to mimic an interview. In this example, they showed an image, then on the next screen asked questions about what you noticed:

Taking one of these surveys is eerily like actually being interviewed. In some, they even circled parts of the screen shot and asked what you thought would happen if you clicked on there.

At EDF a few years back, we used a stripped-down version of this to do rudimentary navigation testing. We set up a task for each question, such as “You are looking for guidance on what fish are safe for you to eat and healthy for the planet.” We offered choices that matched the navigation on our site. We also asked people how confident they were that their choice would work.

A sign of a well-designed test is when you learn something new — but that makes perfect sense. That’s exactly what happened. For the fish question, we knew people would click “Oceans,” and about a third did. We also guessed people would click on “Health,” and sure enough, a third did. But what got us was that a third of respondents clicked on “Land, Water and Wildlife” — internally, we know that’s about projects in water management and protecting land-based wildlife, but to outsiders, it looked like where they would find information about fish.

Nowadays, tools have evolved so that we don’t have to resort to using survey tools in this way. In fact, we’re in the midst of using TreeJack to test navigation for EDF’s new site design. TreeJack is designed for navigation testing and gives richer results than a survey. For example, it indicates how long people struggled with a task, and shows how they backtracked on the way to their answer.

Nowadays, is it ever worth considering a simple survey tool for remote testing? Yes, it is. A basic one-month plan for SurveyMonkey or similar survey tools are only 20% of the cost of a specialized tool like TreeJack (40% if you qualify for TreeJack’s nonprofit discount). Depending on how tight your budget is, that could matter — or your organization might already have a subscription, making it free. If you do go this route, though, structure your questions carefully and keep in mind the limitations of the data you collect.

2. Interviews via instant message – fast and cheap

The authors also point out that remote testing shouldn’t be used simply to cut costs and time from a project. That strikes me as naive, especially in the nonprofit sector or for small businesses. So often, the choice is between quick, cheap testing or none at all. If remote testing lets you do some testing rather than none, go for it!

I like this example of when Porter Mason, a producer at EDF, was resourceful at testing a new feature with no notice and no budget.

It was late in the day, and we were struggling with a new way for site visitors to tweet our content. We suspected it wasn’t clear yet, but were stuck on how to improve it. I was ready to give up, launch the feature as-is, and try to get feedback later. Porter answered, “Just give me an hour!”

He wrote a simple script, and in about an hour, tested the feature on three people via IM.

The first interview helped him identify what exactly was confusing about the new tweet function, and the next two confirmed that the fix worked. After launch, the tool was used frequently.

Some snippets from the interviews. Porter started by soliciting general reactions to the page, then moved on to the tweet function:

Porter: OK, so move your mouse over to one of those links (don’t click yet), and let me know if what happens is what you expect
Tester #1: it turns orange and “Tweet This” pops up
Porter: is that about like something you’d expect to happen?
Tester #1: normally, i would expect it to possibly change color, but i would think that it would be a link to another web page with relevant info
Tester #1: so in other words,
Tester #1: i would expect clicking the text to take me to a web page, and clicking the orange button to allow me to tweet
Porter: gotcha, so go ahead and click it
Tester #1: it takes me to the Twitter
Porter: so this would be what you expected for the button, but not for the text, is that what you’re saying?
Tester #1: correct
Porter: OK, now did you notice some orange links or buttons spread through the text?
Tester #2: i didn’t
Tester #2: but i notice them now
Porter: OK, cool, so move your mouse around those (don’t click anything yet) and let me know what you think these are
Tester #2: it tweets the specific quotes
Tester #2: that get highlighted
Tester #2: when you run over them
Porter: gotcha, so assuming you wanted to tweet one of those quotes, go ahead and do that
Porter: let me know if what happens is along the lines of what you expected
Tester #2: it just took me to twitter
Tester #2: didn’t actually put the stuff in my tweet box
Tester #2: wait
Tester #2: did it again
Tester #2: and it put it in there
Tester #2: i think it messed up before
Tester #2: because i wasn’t signed into twitter
Tester #2: and i messed up my passwrod the first time


Porter: OK, cool, so move your mouse around those (don’t click anything yet) and let me know what you think these are, and how they work
Tester #3: the button shows highlighted text
Tester #3: that you can tweet via Twitter
Porter: gotcha, so assuming you wanted to tweet one of those quotes, go ahead and do that
Tester #3: OK, now that is what I expected
Tester #3: in fact, it’s better than I expected
Porter: how so?
Tester #3: because the text is already in nice Twitter format

This exercise worked because:

  • Porter has both observed skilled moderators and conducted and analyzed tests himself. He is familiar with the experience he was replicating via IM.
  • His interviewees were extremely articulate and at home on IM. This can be true for testing in a web-savvy crowd, but I wouldn’t recommend it for all audiences.

Even with these limitations, a quick remote test let us improve usability on the site, with little time and no budget.

So, to wrap up, Remote Research is a valuable guide to conducting research remotely, but there are probably more opportunities for bootstrap-style than the authors identify. Test as much as you can!