SXSW 2012: DIY Mobile Usability Testing

Belen Barros Pena
Open Source Technology Center (Intel), Interaction Designer

Bernard Tyers
Nokia Siemens Networks, Packet Core Engineer

Presentation Description

Usability testing is an interaction designer’s bread and butter, but applying it to the study of mobile applications and websites brings considerable challenges. Which device should we use for testing? Can we use an emulator? How do we prototype for mobile? Can we just recycle the tasks we use for desktop software tests? Do we test in the lab or in the wild? How do we record screen, fingers and facial expressions?

We don’t intend to answer all those questions in just one session: that would be madness! We’ll focus instead on the last one.
Follow us in our quest to set up a mobile usability testing environment on a tight budget. We’ll show you how others do it. We’ll roam around electronics and professional video stores searching for brackets and webcams. We’ll put our DIY skills to the test and waste a lot of silicon trying to build our mobile recording device. We’ll scour the Internet for free software, and we’ll finish off building the lab and running a usability test in front of your eyes.

If we can do it, so can you! You’ll come out of this session knowing exactly what you need to do to run and record usability tests with mobile devices.

Presentation Notes

Slides: diymobileusabilitytesting.net/doku.php?id=diy-mobile-usability-testing-sxsw2012

Record mobile interaction for both a memory aid, but it’s also a powerful communication tool to prove to the clients/owners of the software that people do visually struggle using their product. Intel records both the actions of what they are intending and actually do, and as well as the reaction of the person.

Usability tests are pretty much the same on mobile devices as they are on desktop computers, except… Before you run usability tests on mobile devices you need to ask the following to produce the goals of your test:

  • Which Phone?
  • Which Context?
  • Which Connection?

Handset usability affects test results. If a user is used to an iPhone and you give them an Android, then you’re going to have a learning curve and cause issue. To get around this always make sure you run tests against users with the phone they are used to. If you cannot do this, make sure to use training and warm-up tasks which allows the participant to get used to using the device first.

Should we run tests in the field or the lab? Well, with desktop usability testing it doesn’t really matter. But with mobile devices we use phones on the toilet, well lit, and dark settings. In all seriousness, no one really knows right now which test is best to do. However, we do know that testing in the field is resource intensive and expensive. Even if you just test in the lab, it’s better to do that than nothing at all.

If you must do field testing:

  • Do it late, because your in-lab tests will get most of the usability concerns first
  • Plan and run pilot tests
  • Be prepared, such as if it rains

Recommendations:

  • Never test over wi-fi, as you’ll loose a lot of value running over a slower network
  • Cover participants’ data costs who are doing the tests for you

So how do we record the experience?

  • Wearable equipment like hat-cameras
  • Document cameras; but those are not cheap, and have the disadvantage of requiring participants to keep within the camera’s range and this just isn’t natural feeling
  • Mountable cameras which allow for natural interaction with the phone, if they don’t get too heavy
  • Screen capture software; but no one likes you installing stuff on their phones, and no application will support all platforms
  • Remote tool such as mouseflow.com (records visits to your website without people knowing). It supposedly also works on mobile devices. It seems as though this doesn’t fully work yet on all phones though

If it would be possible you’d want

  • Easy to put together
  • Cheap
  • Repeatable
  • Allows holding the device
  • Allows one-handed use
  • Supports all form factors
  • Runs tests with participants’ phones
  • Captures screen, face and fingers
  • Gives enough video quality

Photograph of PresentationIntel took the 5 recording methods and found that the mounted devices were the best solution. But that was too expensive, so they instead built their own using Erector Sets, cheap web cams, poster putty (BlueTag, which also helps protect the phone), and bolts. They then run this through a windows machine with both of the cameras showing up, and just simply screen capture.

Posted in SXSW 2012.

Leave a Reply

Your email address will not be published. Required fields are marked *