Brainstorming: User Testing and Peer Response

Posted: November 4, 2007 in Brainstorm, Discount Peer Response, iteration, Usability

As the title of this post suggests, this is an informal brainstorm. I do not know what will come out of it, but hopefully it will be useful. It is my intention to try to pump out at the very least 2 pages of good text this weekend. My hope is that will include both a ‘disclaimer’ (that this is a sample from a larger work, written for a class, etc) and a solid introduction. I also want an outline, which is why I’m here.

What do User Testing and Peer Response have in common? The first thing that comes to mind is the learning curve. Jakob Nielson provides a graph of a usability testing curve that suggests that once five people have tested a product, they will find roughly 85% of the problems. The suggestion is to then do multiple iterations of five testers (with redesigns in between), eventually finding/solving virtually all of the problems. When you have more than five users, Nielson says, “you are wasting your time by observing the same findings repeatedly but not learning much new.”

This seems to me to be very similar to peer responding. If a student has two people look at their paper, the responders will find many errors/problems. Some of those will overlap, though not too many. Add a third reader and even more problems are found. More than three starts to get very redundant; a new draft has to be written to get further value. Once a new draft is written, having someone (or two or three people) look at the new draft will find even more mistakes/problems. Then a new draft, dealing with this new set of problems, is written, and the process repeats.

By Nielson’s calculations, the first time through finds (and presumably fixes) 85% of the problems. The second time fixes 85% of the still extant problems (98% of the original), and the third time through fixes 85% again (99.7% of the original). This doesn’t fix ALL problems, but is about as close as you can get and the last time you find a meaningful increase (a fourth time nets you 99.955%, I think).

I don’t know how much a single peer responder can find. A conservative estimate, I think, would be 50%. So two responders would find, say, 75%, and three would find 88%. So if each iteration had 3 responders, you’d only really need three drafts to get past the 99% mark, which theoretically should guarantee a good grade (unfortunately, it doesn’t, which is a difference I need to consider).

On the other hand, if there were only two responders the first time through (75%), and then one each time after that (88%, then 94%, then 97%, then 98.5%, then 99.25%) you’d need a total of 7 drafts (including the original) to get the same result.

What matters here is that there is a parallel, even if it’s only the fact that there is (probably) an upward limit to how many users/responders are useful for any single iteration.

What else do they have in common? In both cases, the tester will be most helpful/successful if she takes on the role of the end user. Granted, in a computer program that end user is a nebulous client base, and in peer responding the end user is the teacher/grader of the paper. Still, in both cases it is more helpful for the writer/programmer attempt to take on that role. So that is two things in common.

In my experience, a solid paper requires at least three examples, each stronger than the last, to make a convincing argument. Maybe this feeling is artificial, informed (as so much is) by Rene Descartes and his Meditations on First Philosophy (particularly the first one). Even if it is, that is what has always worked for me, and hence is what I will stick with unless/until someone shows me a more effective model. So. I need a third parallel.

There is the parallel of trying to perfect a product. This at first seems pretty weak, but maybe there is something there. If the goal is to create a product (paper in writing and program in programming), then this can be considered an ‘essential’ step towards that goal. I put essential in quotes because it is often ignored (both by student writers and, unfortunately, by some programmers –I saw a web designer’s site earlier today that was programmed in flash), but when it is used, it almost universally results in a better product.

I am a little bit hesitant about this idea because of the focus on product for writing. When writing a program, it is the product that matters. But process theory argues that this is not the case in writing. I could tackle that issue, but it seems WAY too big to be a side bar. Can I just say that while it is process that is important, the product is nonetheless the goal of that process? I think I can probably doing that without inviting a lynching.

So. Using the goal of a product in both cases, the parallel of ‘necessary step’ comes back up. This is certainly not the strongest argument/example I have come up with (I think both the ‘end user’ and the ‘upward tester limit’ arguments are stronger.)

This leaves me with my three arguments. First, and weakest, I have the Necessary Step (Descartes’s “My senses deceive me”). Next, quite a bit stronger, I think would be Upward Tester Limit (Descartes’s “I may be dreaming”). Finally I think the strongest argument is the End User argument (Descartes’s “Evil Demon”).

The question really is whether my second strongest argument will be more acceptable and more difficult to refute than my strongest, as is the case with Descartes (I apparently have him on the brain right now).

My constant reference to Descartes brings to mind another theory I’ve had: that Literary Theory will develop along similar lines to those of Philosophy, and English studies can get an idea of where they are going by studying the history of philosophy. I don’t know if this same parallel exists in Rhetoric. The reason I believe it in Literary Theory is that LT is so young and Philosophy is so old. But since Aristotle is called “The Father of Modern Rhetoric,” it’s very difficult to make the same parallel for Rhetoric (though by the time of Aristotle, Philosophy was already several hundred years old).

If the parallels between Philosophy and English Studies is not already on my list, it needs to be.

Enough digression. I think I’ve got enough to start working with.

  1. cbdilger says:

    One suggestion, one question:

    S: Move the language away from eliminating problems to improving the match between the system’s goals and its actualization of those goals. On both sides of the equation, it’s easy to see the task in the former manner. At least in writing studies, there’s good literature on why that can be reductive.

    ?: Are you suggesting we apply Nielsen’s “discount” ideas to peer review? That is, instead of One Big Formal Timeconsuming Peer Review we have five quicker ones? Interesting…

  2. cogitas says:

    I’m not sure if this response addresses exactly what you’re talking about, but let me try:

    S: When I started writing, I presented it as hopefully showing a connection between User Testing and Peer Response in order to suggest that each one can learn from the other how to improve methodology. I mention that this is part of a larger issue of the potential connection between programming and writing studies.

    Or do you mean when talking about the effectiveness of either one (85% for UT)?

    ?: I am suggesting that we have several smaller peer reviews yes. More drafts, more iterations. It’s how I do 180. I have 1 formal one (2 for the research paper) and give the students the option (and advice) of 3 additional peer reviews on their own for extra credit.

  3. cbdilger says:

    S: It’s a shift in language and focus. Good sources say, and rightly so, “Don’t see (peer review|user testing) as problem finding, but as improving the ability to meet goals.” Heck, that’s one of the connections you can talk about.

    ?: This is very much worth flushing out: “discount peer reviews” is definitely the reverse of other peer review scholarship I’ve seen; the tendency is to make them MORE formal in the hopes of improving quality. So here’s something WS could learn from UCD…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s