I just completed an online survey at the invitation of a company I’ve purchased from in the past.  It was obvious that the survey was an example of what the market research industry calls “D-I-Y” research.  If the quality of the questionnaire had not given this away, there was the “Powered by [name of enterprise feedback software vendor]” at the bottom of the screen.  I was asked to look at two different print ads for one of the products this company sells and answer a few questions that bore some slight resemblance to the questions you might find in an ad test conducted by one of the MR firms that specialize in that type of work.

One can only assume that the results of this survey are meant to drive a decision of which ad to run (there may be other candidates that I didn’t see).  If that’s true, then I think this may be a case where D-I-Y will turn out to be worse than no research at all.  The acid test for any market research is whether or not the decisions made on the basis of that research are “better” than the decision that would have been made without the research.

There were a number of flaws in this particular survey, some egregious and some merely annoying.  Among the most egregious:  presenting two different ads for the same product.  Once I’d seen the first ad, any measure of purchase interest (this survey used a typical 5-point PI scale) for the second ad was contaminated by my exposure to the first ad.  Most of the other survey questions are likely to produce ambiquous results at best.  To top it off, each set of ad-related questions ended with a list of attributes that I was asked to associate with the parent brand.  I guess the survey author thought seeing a second ad might actually change whatever parent brand associations I might have formed from the first ad.  These two questions were more annoying than anything else.  For one thing, the attributes on the list appeared to be the sort that are thrown in without much thoughtful consideration (or, even better, pilot testing).  I confess that I expressed my annoyance by checking the attribute “boring” in one of the lists, but did not bother in the second.  How will that be interpreted?

The user interface for the survey was not that great.  In particular, the images of the ads were small (at least on the screen of my MacBook) and it was hard to read all of the copy.  The company sells educational materials–primarily courses on DVD–and all that copy is probably important to the purchase decision.  I should probably disclose that I already owned the particular title featured in the two ads (I noted this in the open-ended question following the purchase intent measure for the first ad).

So, will this D-I-Y market research provide any value to the company?  I know which ad I think they should run.  One ad contained an illustration that communicated the topic of the course clearly; the other was pleasant to look at but the main visual was a picture of the DVD package.  I think there’s a good chance that the data generated by this survey could lead the company to run a less effective ad.  In that case, doing research might turn out to be worse than no research at all.

Do-it-yourself research has been around for a long time.  In the pre-Internet era, any company could send out a mail survey or, if they had people available, conduct their own telephone research.  The arrival of CATI systems and WATS long distance pricing favored outsourcing of telephone research, but the ease of conducting research using the Internet has prompted many companies to experiment with D-I-Y.

The Internet has had a profound effect on the practice of market research, in part because tools that were once concentrated in the hands of market research companies and marketing consultants are now widely available.  This has resulted in what my former colleague at Harris Interactive, Jon Siegel, has called the “de-crafting” of market research.  I think many firms will be poorly served by D-I-Y research but we can expect to see this trend continue.  It will be a shame if the poor quality of some D-I-Y research sours companies on the value of all market research.

Copyright 2010 by David G. Bakken.  All rights reserved.