The Hacker's Guide to Usability Testing

Starr Horne bio photo By Starr Horne

What’s the use in writing great code if nobody uses it? In this talk at Cascadia Ruby I demonstrate a simple method for usability testing that engineers can and should use:

Here’s the transcript:

Starr Horne: Hey, guys. Where did that come from? There are ghosts in here. I just got one question for you, guys. Are you ready to rock? [audience approval]
Starr: I thought everybody was going to be silent. I had this whole shtick worked out. Anyway, let us talk about usability testing. Now, I have written a lot of code that nobody uses. It is kind of a weird thing to say, right? I imagine I am not the only guy here who can say that. A couple of years ago, I was working for this company, this start-up that did software for real-estate agents. Overall, this project was technically successful. We had a good designer on board, it was pretty, and the engineering team did well. We were delivering things on time, under budget. We just had this one big problem. Nobody actually wanted to use the software. After a while, this started to bug me because I'm lazy. I pride myself on this. I don't like doing things that I don't have to do. Here, I'd spent two years of my life working really hard to build something when I could have been playing Xbox. It really irked me that I missed such a great opportunity to slack off. In this frame of mind, this agitated frame of mind, I started noticing little things about the place. I had never actually met one of these real estate agents. I didn't even know what these customers of ours looked like. It was pretty interesting because the development team had asked about this. We'd asked to be able to shadow these users, to be able to go and see how they actually used the software. There was always something more important. There was always a fire to be put out. There was always some new feature. [emphatically] "This time it's just going to make everybody flock to the app." We just never got around to doing that. In the end, we wound up making an app that technically was a nice app in and of itself, but just fundamentally wasn't connected with how these people actually did their work, on a day-to-day basis. From a careful reading of Slashdot and Tech Crunch, this seems to be the rule, rather than the exception. People seem to be doing this all the time. This is on the top of my mind, lately, because I know I worked for that company, I work for myself now, I work at my own company. I am a co-founder of Honeybadger. We monitor Ruby applications and productions, send you alerts when bad things happen. This is really important to me now, because, if people do not like using our app, if people do not like using our product, then I do not get paid. That is a deal-breaker for me, because, in addition to being lazy, I am also greedy. I want to be able to do things, like eat, like take my wife to the movies, occasionally. That is good motivation for me. But what about you, guys? I imagine that we are all in the same boat, right? I imagine all of our destinies are, in some way, tied to the success of a project or a group of projects. At some point, people actually have to use this stuff we are building. People have to use it and if they do not wind up using it, if they are not able to use it, that really can affect the overall success of your project and your future income. Now, a couple of obvious examples, right? If your sign-up process sucks, you could be losing 10 percent of your users without even realizing it. If your app is, generally, hard-to-use, if people do not like it, then you are going to lose customers, because of that. I just want to point out that, if these were development issues, if your user system was throwing away one out of every 10 users that sign up, that would be a huge freaking deal. That is the kind of bug that you stay late to fix, right? But there is not a bug in the code, there is not a problem with the code, there is a problem with the design. Design can have bugs. [silence] I am just going to let that bore itself into your eyeballs a little bit longer.
Audience: [laughs]
Starr: [laughs] Design can have bugs. The bugs can have real consequences just like code bugs can. For some reason, we don't really treat them as seriously. At least, that's my experience. We tend not to treat them as seriously as bugs in the code. Why is this? As an industry, as programmers, we're obsessed with quality. We want to do everything perfectly. I imagine most of you guys wouldn't dream of shipping code that you hadn't tested, that you didn't know worked, we ship designs all the time that haven't been tested. That not a single user has actually seen. I'm guilty of this as well. Why do we do this? It can't be because we don't care because people who don't care don't come to conferences. Why do we do this? I think it's just that we don't think there's a problem. I don't think we recognize that this is even an issue. "I can use my app. My app works fine. What's the big deal?" Things seems fine, so we don't hesitate to ship designs that later can actually be proven are broken, have problems, that make it hard for users to do things. This leads you down this crazy spiral of self-doubt. This morning is just the bummer. All the uplifting talks are tomorrow. This leads you down this really weird path where you start to wonder, "OK, is something wrong with me? I can see problems in my code. I can stop those before they ship. Why can't I stop these problems with my design before they ship? How come I have to find out about them a month later when I see that revenues have dropped?" I was talking to a friend of mine. I was at a barbecue at my friend's house a couple weeks ago and we were talking about this stuff. I mentioned that I was going to be giving this talk. A talk about usability for developers. She said, "Starr, if I was going to give that talk, I would just have one slide and on that slide it would say, 'You are not your users.'" I thought that was interesting, so I made an infographic for you guys.
Audience: [laughs]
Starr: You're not your users. You're programmers. Has anybody here ever tried using them? I don't care if you use it on a regular basis. Has anybody tried using it? Do you know any non-programmers who have tried using them, who even know what them is? It's a great text editor. People have to edit text right? The thing that makes us awesome and the thing that makes us really weird is that we're able to and willing to take on enormous amounts of technical pain if, at the end of it all, we think we're going to be able to make a quadricopter deliver us a martini.
Audience: [laughs]
Starr: Even when we can't do something that awesome, even if there's no obvious payout, we do this stuff because it's fun, because it's just what we do. Just like that's our greatest strength as programmers, it's also our greatest weakness. It's like "Superman". Actually, I don't know anything about "Superman", so never mind. We deal with this technical pain so often, we're so used to it, we get immune to it. We just get used to it. It becomes easier for us to ship interfaces that we think are fine, that look fine to us, but that to anybody else just seem like gibberish. I know some of you guys are probably thinking that this is a solved problem. Developers have this really bad reputation for making user interfaces, like Linux. We have this really bad reputation, but that's a known problem. It's solved. "We have designers, man. That's what we pay these guys for." Well, your designer isn't really your user either, are they? I'm just going to go off on a rant for a second. I hate the word "designer". I hate it, because it's one of those words that means so many things that it doesn't mean anything at all. You have floral designers. You have architectural designers. You have battleship designers. You have one word that means all of those things. The type of designer we most often deal with as developers is a graphic designer, a visual designer. These guys want to make stuff pretty, which is awesome because I like pretty things, but they don't know how your users are going to behave when they hit your application. Maybe they pretend they do. Maybe they had a class about it. Maybe they've got really good instincts, but unless they're actually doing testing, unless they're actually interviewing users, they really don't know. They're just kind of guessing just like you are, just like I am. Your client isn't your user. This is kind of hard for me to swallow personally, because I was a freelancer for a long time. One of the things I really prided myself on was being able to deliver a deliverable that made my client happy, that satisfied their requirements, was real, and was in the compounds of reality and all that. I'm here to tell you, you can make a product that makes your client 100 percent happy, but is just a wretched, ungodly, unusable mess. I should know, because I've done it. In case you're wondering, it usually involves making the logo bigger. I'm just going to pause on all my ugliest slides. Your boss isn't your user. Your ops guy isn't your user. Only your user is your user, but Twitter doesn't count because people are going to tell you all sorts of things. People are going to tell you that your kerning sucks, that your font sucks, that your design is great, that your design is awful, that you need to implement features X, Y and Z, and then they'll buy your product. They might be right, but there's a lot that people don't tell you. For example, people aren't going to tell you that form that they had to fill out made them feel dumb, because they didn't use the terminology, or they didn't understand the terminology that you used. People aren't going to tell you that they just left your site after five minutes, because they didn't know how the heck they were supposed to use it. These are the things that people won't tell you, but really these are the huge problems. This is the stuff we need to know the most. This is where usability testing comes in. This is how you actually find these huge design bugs. Once you find them, a lot of times they're not that hard to fix. In the next part of this talk, I'm going to go over some of the actual "how-to", how to do this stuff. You really don't need much. It's really incredibly simple. You don't need a big usability lab. You don't need microphones and video cameras, and two-way mirrors, and guys with black coats. You don't need any of that. Chances are most of you have the tools you need right now on you to get started on this. This is my setup. The participant sits in that chair. They use the computer. We record what they do. We record what they're doing on the screen and what they're saying. If the Dev-Team wants to listen in, we just use screen sharing software for that. They don't even have to be in the same building. Pretty awesome. I sit in the chair next to them with a clipboard, and ask questions occasionally. In terms of infrastructure and technology, this is really all you need. It's also pretty cool, because you don't even need to know that much. I love stuff like this that I can do without even having to really know that much. I'm sure you can go to Amazon. I'm sure you can buy hundreds of books that tell you hundreds of different ways to do usability testing, and all of them are super complicated. I'm sure they're very useful to some people, but we've gotten great results just using two types of tests. I'm going to go through them now. The first of these is called the "What Is It" test. This is where you're going to give your user a mock-up, an apped in sketch. Have them go to a website and have them look at it, and ask them what it is, what it does, does anything stand out? I'm sure nobody here has any sort of website or application that could benefit at all from this. You can tell your friends about this. I'm sure we've all come across examples of applications that could have perhaps used this. I think this is a chess app. I'm still kind of unsure. I don't know. Anyway, the second type of test. It's called the "Please Do This" test. These are my terms by the way. You won't find anything by Googling them. As you may suspect, this involves asking somebody to do something, but it's a little bit more complicated than that. What we're going to do is make a little experiment. The first step, we need to make a list of things that we like to know about our site. For example, a couple of weeks ago I was running one of these studies. We had made some changes to the credit card pages. I wanted to know if people could still navigate their way around, if they could just still add a credit card. I put on my list, "Can people still navigate around?" That's step one. Step two is you write what's called a "Scenario." A scenario is a task for the user, and it sounds a little bit like a Cucumber feature description, if you ever used Cucumber. My scenario for this question might be, "As an existing Honeybadger customer, you have billing information on file, but you know your credit card is about to expire. You need to go update that. Please log-in using this information and update your credit card." Then you give it to the user. You have it printed out. You read it to them. You give it to them, and just see what happens. I'm going to show you what this looks like. I'm going to show you some screenshots. I'm going to pretend to be the user in this case. In real life of course, you wouldn't be using screenshots. You'd be using the real app. I'm going to talk out loud, because that's what I'd ask the user to do. That's where you get your real insight from. People just saying what they're thinking at the moment. [impersonating the user] "I'm going to go change the credit card. Signing in, type, type, type, type. OK, this looks like my home page. There's the little icon guy over on the right. I'm going to click on that. OK, profile. OK, there's the billing tab. There we go. Wait a sec, I don't understand this. There's not a credit card. Oh, OK, it's over on the side." That's about it, right? What we found here is what I like to call a leak. It's a small problem, but when you're dealing with stuff like billing small problems can have big consequences. This is a pretty simple scenario. If you guys do this your scenarios will probably be a little bit more involved. You'll have three to five of them per user test. Each test will last about 40 minutes. I repeat the tests three to five times. You'll find that once you've done this and once you've gone back and looked at your screencasts, you'll have a really good idea of four or five areas that could use improvement. In fact, if you do more than five of these tests...I did seven one time. By the seventh user, I was so bored. I just wanted to point out. Like, "OK, you're just going to have problems. Just skip that." We've covered the technology, we've covered the methodology. Both of those are really simple. The only complicating factor here is that we're dealing with people and people are really weird. Your role as the moderator of one of these tests is also weird because you've got to, on the one hand, be impartial. You've got to be a blank slate, not give them any clues. If you give them clues, you could mess up your test results. On the other hand, you have to make sure they actually do the test. You have to make sure they do what you want them to do. Otherwise, what's the point if they go and check their email for an hour? What you'll find is that people will always be trying to get clues from you, even if they don't realize they're doing it. As humans, we have this really weird unconscious desire and ability to continuously give, receive, ask for validation to make sure that we're behaving correctly in the context that we are. As a moderator, you have to turn off that side of yourself. You've got to be completely impassive, like a psychotherapist almost. It's strange. That's on the one hand. Then on the other hand, people are going to do weird stuff. Right? When you write this scenario, you're building a road for them to go down. Usually, it's going to be a pretty short, simple, straight road but people are going to be veering all over the place. People are going to do things you didn't expect. Sometimes they can auto-correct. Sometimes, if you have them to delete a record that's right in front them and they wind up on your Terms of Service page you can say, "So, I see you're on the Terms of Service page..." Then you don't say anything and that awkward silence prompts them to be like, "Well, OK. I was looking for something but it's not here, so I'm just going to go back and do the thing you actually asked me to do." Then other times, people go off the road. They go off the cliff next to the road. They fall into the ACME explosives factory at the bottom of the cliff. There's no way they're getting back on track without an intervention from God, which you kind of are. Then you can just rewind it. You can say, "OK, let's just go back. Let's start over. I thought I wrote this in a clearer manner, but it seems like it's made you a bit confused. Let's just go back." That's kind of it. That's a little bit more complicated than the other stuff, but still it's pretty easy. Now you guys have pretty much all the knowledge you need to go out and do one of these tests. I hope I've also convinced you that it's worthwhile because for a long time I put this off. There's always something more important to do. There's always something more pressing. Now that I've made this a part of what I do and what our company does on a regular basis, I can't think of anything that I can talk to you about that has given us a greater return on investment. For a day's worth of work, you get invaluable information about how your app works in the real world. How actual users really use your app. That's really amazing. For a process that's so freaking simple, it's wonderful. If you'd like to learn more about this, I highly recommend you read this book. It's awesome. You can read it in an evening. It costs 20 bucks. It's made for non-usability professionals. Everybody should buy that. If you want to know more about what I'm up to, check out []( That's where I blog. I also tweet on the twitters. That's it. Thank you very much. If you have any questions, I'll be happy to answer them later. Just pull me aside and grab me. You guys can start clapping now, I'm sorry. [audience applaud]