Erika Hall, preach!!! The co-founder of Mule Design absolutely nails it here, pointing out the pitfalls of surveys, or rather, how they are often abused. Surveys are easy to create and provide you with structured data, lulling you into thinking that, by default, they always produce valid, useful information. But if you think this tool can define human behavior as quantitative data, you’re not doing yourself any favors.
Hall does a tremendous job of driving home that point. These two lines in particular say so much about how to approach virtually any form of user research, let alone surveys.
“And as I say again and again, and will never tire of repeating, never ask people what they like or don’t like. Liking is a reported mental state and that doesn’t necessarily correspond to any behavior.”
How often do you come across survey results that claim X uses Y 75% of the time, or that X relies on Y to make important decisions? Never trust a survey offering quantifiable results by way of human response.
While user perception is hugely important, perception-based answers should not be confused as a substitute for quantitative data like analytics. Can’t get answers with the analytics available to you? You may have to settle by applying deductive reasoning to the data you have, but don’t fall into the trap of using a survey to fill the gaps.
Why? Erika says it better than I can:
“Bad code will have bugs. A bad interface design will fail a usability test. It’s possible to tell whether you are having a bad user interview right away. Feedback from a bad survey can only come in the form of a second source of information contradicting your analysis of the survey results.”
Use surveys sparingly, and when you do, base your questions on measurable outcomes.