— 21 March 2003 —
A paper (abstract) on remote testing versus in-lab testing by Tom Tullis, et al. from Fidelity Investments in Boston. One of the interesting things on this is that the subjective ratings (preference questions) were scored lower in the remote testing. In other words, people were more critical when they didn’t take time out of their day to come to a nice lab, meet nice people, and be nice all day. Another thing was that between the two tests only 11 of the same usability issues were found. 26 were found in-lab, and 16 were found in the remote. So if 11 were the same and the preference data was more critical, is remote testing better? They did a second test that only found 7 similar usability issues.