Better survey design is one sure way to improve the ROI of marketing research.  However, despite advances in our understanding of the cognitive processes involved in answering surveys, market researchers continue to write poor survey questions that may introduce considerable measurement error. 

I think this is due in part to the fact that the processes involved in asking a question are fundamentally different from the processes involved in answering that same question.  Recent contributions to our understanding of the answering process have been integrated into a theory of survey response by Roger Tourangeau, Lance J. Rips, and Kenneth Rasinski (The Psychology of Survey Response, Cambridge University Press, 2000).  According to Tourangeau, et. al., answering a survey question involves four related processes:  comprehending the question; retrieving relevant information from memory, evaluating the retrieved information, and matching the internally generated answer to the available responses in the survey question.

“Think aloud” pretesting, sometimes known as “cognitive” pretesting or “concurrent protocol analysis” is an important tool for improving the quality of survey questions, and  well-designed think aloud pretests often have, been in my experience, the difference between research that impacts a firm’s business results and research that ends up on the shelf for lack of confidence in the findings.

Warning–what follows is blatant self-promotion of a sort.  ESOMAR is offering my workshop, “Think like a respondent:  A cognitive approach to designing and testing online questionnaires” as part of Congress 2011.  The workshop is scheduled for Sunday, September 18, 2011. This year’s Congress will be held in Amsterdam.  I’ve offered the workshop once before, at the ESOMAR Online Conference in Berlin last October.

Hope to see you in Amsterdam.

The options represented trade-offs, the simplest being that between cutting programs or raising revenues.  Someone has to suffer, and suffering was not evenly distributed across the options.  Nearly seven thousand Twitter users completed the puzzle, and Leonhardt has summarized the choices.  You might still be able to access the puzzle online at nytimes.com/weekinreview.

Leonhardt was able to group the solutions according to whether they seemed to consist mostly of spending cuts or a mix of spending cuts and tax increases.  He admits that the “sample” is not scientific and, given that it’s comprised of Twitter users, may skew young.  Unfortunately, no personal data was collected from those who completed the puzzle, so we’re left to speculate about the patterns of choices.  Perhaps a little data mining would shed some additional light on the clustering of responses. 

Even though this is not survey resarch in the way that we know it, there may be much value in using this type of puzzle to measure public opinion about the tough choices that the U.S. is facing.  The typical opinion survey might ask respondents whether they “favor” one course of action or another (“Do you favor spending cuts or tax increases for reducing the deficit?”).  The options presented in Leonhardt’s puzzle represent real policy choices, and the differences between them force you to consider the trade-offs you are willing to make.  While the choices were comprehensive, they were not contrived in the way that conjoint analysis structures choices; that might present a problem if we are trying to develop a model to predict or explain preferences.

There’s no reason this technique cannot be used with the same kinds of samples that we obtain for much online survey research.  Add a few demographic and political orientation questions and you have what I think could be a powerful way to capture the trade-offs that the public is willing to make.

Copyright 2011 by David G. Bakken.  All rights reserved.

Finding regional differences in these written expressions is interesting in its own right, but I’ve just finished reading the paper describing this research and there’s a lot more going on here than simply counting and comparing expressions across different geographic regions.  The paper is an excellent example of what market researchers might do to analyze social media.

The study authors–Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing–are affiliated with the School of Computer Science at Carnegie Mellon University (Eisenstein, who was interviewed for the ATC broadcast, is a postdoctoral fellow).  They set out to develop a latent variable model to predict an author’s geographic location from the characteristics of text messages.  As they point out, there work is unique in that they use raw text data (although “tokenized”) as input to the modeling.  They develop and compare a few different models, including a “geographic topic model” that incorporates the interaction between base topics (such as sports) and an author’s geographic location as well as additional latent variable models:  a “mixture of unigrams” (model assumes a single topic) and a “supervised linear Dirichlet allocation.”    If you have not yet figured it out, the models, as described, use statistical machine learning methods.  That means that some of the terminology may be unfamiliar to market researchers, but the description of the algorithm for the geographic topic model resembles the hierarchical Bayesian methods using the Gibb’s sampler that have come into fairly wide use in market research (especially for choice-based conjoint analysis).

This research is important for market research because it demonstrates a method for estimating characteristics of individual authors from the characteristics of their social media postings.  While we have not exhausted the potential of simpler methods (frequency and sentiment analyses, for example), this looks like the future of social media analysis for marketing.

Copyright 2011 by David G. Bakken.  All rights reserved.

Given that market researchers apply much of the same reasoning as scientists in deciding what’s an effect and what isn’t, the decline effect is a serious threat to creating customer knowledge and making evidence-based marketing decisions.

Lehrer suggests that the decline effect is a consequence of publication bias.  Only studies that demonstrate some noticeable effect (usually consistent with a prior hypothesis) make the cut for publication.  Our beliefs about randomness and the likelihood of large effects have lulled us into a false confidence about the validity or reliability of study results.  A researcher observes a large effect, that study gets published and, in many cases that is the end of it.  The researcher moves on to another topic.  Unless there are multiple attempts at replication, we cannot be certain (even with statistical “significance” testing) that the big effect was not a random outlier.

Before you dismiss this idea, consider two empirical examples.  First is Lehrer’s account of an experiment conducted in the late nineteen-nineties by John Crabbe, a neuroscientist at the Oregon Health and Science University.  Crabbe replicated a study on mouse behavior in three geographically dispersed labs, but he made sure that all other aspects of the experiment were “identical” across the three labs down to the smallest detail, such as the day on which the mice were shipped to the labs (and all from the same supplier and genetic strain, of course).  The study measured the effect of cocaine on rat movement.   In one lab, the rats moved, on average, 600 centimenters more than their baseline after being injected with cocaine.  In the second lab the moved an average of 701 additional centimeters.  In the third lab they moved an additional 5,000 centimeters!  Three experiments, as identical as possible, yielded effects that varied by more than eightfold.

My second example is closer to the core of market research.  With access to a sample of 2,000 online survey respondents, I conducted a quasi-bootstrap analysis to determine just how much sampling variation I might see with repeated random samples drawn from this larger “population.”  I created 20 separate random samples with 500 respondents in each sample and looked at the “sampling” distribution of a few key measures from the survey.  It’s a simple matter to calculate a 95% confidence interval for each of the measures.  Over the long run (many thousands of replications) we would expect that no more than 5% of the samples would fall outside the 95% confidence interval for any metric.  With only 20 samples, it’s reasonable to expect that 10% or 15% (2 or 3 of the samples) would fall outside that range.  Across the various measures, I found 30%, 45%, 25%, and 50% of the estimates fell outside that 95% confidence interval.  (By the way, I repeated this exercise with a different sample of online survey respondents and found 20% to 40% of the values lying outside the 95% confidence intervals.)

Many “evidence-based” marketing decisions are informed by facts and insights obtained from one-off samples of (at best) a few hundred “representative” consumers.  We calculate confidence intervals for point estimates or apply “appropriate” tests, such as Student’s T, for comparing subgroups on some measure of interest, and as long as the calculations inform us that a confidence interval is sufficiently narrow or the difference in the means of two groups is sufficiently large, we accept the result without much question.  Logically, these tests tell us something about the sample but may be wildly misleading with respect to inferences about a larger population of interest. 

Online samples–whether from “opt-in” panels or the wide wide river of the Internet–have raised (properly) several concerns about the quality of the evidence on which we hope to base marketing decisions.  As proplems of “respondent quality” have been addressed by digital fingerprinting and other verifiation solutions, attention has turned to measurement consistency between samples (The Grand Mean Project, is one example).  After all, if you cannot get the same answer when you measure the same thing in the same way on different occasions, how much faith can you have in online survey research?

A focus on consistency is important but needs to be combined with a better understanding of how much consistency we should expect in the first place.  Bootstrapping analysis is a good place to start.  And, once we have a better understanding of true sampling error, we can develop decision strategies that better reflect and incorporate our uncertainty about the “evidence.”

Copyright 2011 by David B. Bakken.  All rights reserved.

Schultz tried to communicate just what they were thinking in an online video, saying that the new design respects the brand’s hearitage but also looks toward a future for the firm that’s not just coffee.  And on the Starbucks website one of the company’s creative managers, calling the logo change “the project of a lifetime,” says:  “From the start, we wanted to recognize and honor the important equities of the iconic Starbucks logo.”

Firms do change logos without the flack that The Gap generated a couple of months ago, but that change (abandoned in the wake of criticism) and Starbucks new logo may offer important lessons about when and why to change logos, and what not to change.  As I’ve expressed in a previous post, I’m skeptical about neuromarketing, but logos are one area where I think companies might do well to invest in a little neuroscience before making big changes.  Logos provide a sort of cognitive shortcut, and we humans are cognitive misers.  The more “iconic” a logo is, the more effective the cognitive shortcut.  To be iconic, a logo needs to be distinctive (not easily confused with other symbols) and consistently associated with with a specific set of brand experiences.  Moreover, the iconic elements of a logo are usually basic properties like shape, color, typeface and spatial relationships within the logo.  The true test of an iconic element is the extent to which you can minimize or degrade the image (showing just a small part, for example) and evoke the complete logo or the associated brand.  For example, in the hit CBS drama NCIS the main characters drink coffee from cups that have some sort of green and black circular logo with white lettering–just enough to suggest Starbucks.  I think one problem with The Gap’s proposed logo was that it changed several iconic elements at once (background color, typeface, and spatial relationships within the logo).

Having been on the inside of a few re-positioning efforts–some involving new logos or package designs–I think that we often over-intellectualize the meaning of the elements in a logo.  Starbucks appears to believe that the mermaid (or siren) in the center of the logo is the symbol of the brand, but some of the comments posted to their website indicate that many customers never really noticed that the element in the center of the logo was a mermaid.

I don’t know whether Starbucks (or The Gap, for that matter) did any consumer research in the course of developing their new logos.  However, the typical research methods for this type of thing rely on verbal responses, while logos operate mostly at a nonverbal level.  Perceptual research methods are more appropriate for this type of problem than focus groups and surveys.

Many companies have modified their logos, of course, but those that succeed seem to recognize that it’s more important to understand the Gestalt of the logo than any potential symbolic meaning of the logo or individual elements.  You can tweak a typeface, for example, to look more modern, or play with the saturation and tint of the color scheme, but if several elements change at the same time, or the spatial relationships change, you can expect the kind of feedback that The Gap and Starbucks have experienced.

Starbucks may well stick by its new logo.  In that case, I suspect customers still will be able to find a store when they want, or recognize Starbucks coffee on their supermarket shelf.  My main issue with these logo changes is that they are superficial.  Consumers don’t buy “Starbucks” or “the siren”–they buy a particular constellation of experiences and solutions to problems.  If you get the latter right and you’re lucky enough to have an iconic logo, why abandon it?

Copyright 2011 by David G. Bakken.  All rights reserved.

The study, “Genetic underpinnings of survey response,” is to be published in the Journal of Organizational Behavior. A press release  from North Carolina State University quotes Dr. Foster:  ”We wanted to know whether people are genetically predisposed to ignore requests for survey participation.  We found that there is a pretty strong genetic predisposition to not reply to surveys.”

The researchers  sent a survey to more that 1,000 sets of twins, some identical (and possessing identical DNA) and some fraternal (no more genetically similar than any two siblings).  The study found that the it was possible to predict the propensity to respond for one identical twin from the response (or non-response) of the other twin, but there was no such relationship for the fraternal twins.  The researchers “used quantitative genetic techniques to estimate the genetic, shared environmental, and nonshared environmental effects on people’s compliance with the request for survey participation” according to the paper abstract.

Notwithstanding the power of the right statistical methods, it’s very difficult to rule out plausible rival hypotheses in single generation familial inheritance studies.  I spent one summer during graduate school analyzing data from an adoption study attempting to prove the heritability of schizophrenia.  In addition to the adoption paradigm (that is, looking for differential incidence rates  among the biological and adoptive relatives of the adopted, afflicted individual), we have two types of twin studies–those that compare identical twins reared apart and those that compare sets of identical twins with sets of fraternal twins, as in this case.  Twins reared apart studies got a bad rap as a result of Cyril Burt’s fraudulent data purporting to show the heritability of intelligence.  Comparisons of identical and fraternal twins run up against the fact that having an identical twin is a very different experience from haing a fraternal twin.

I see two potential problems with this study.  First, we can’t rule out differences in interaction between identical twins and fraternal twins as a possible explanation.

The second problem–all genes are expressed at a cellular level in the form of different proteins.  Survey non-response, in contrast, is a specific and high order (far removed from cellular activity) that, really, is unlikely to be governed by a a few small chemical differences.  I believe that anyone making a claim about the heritability of any behavior ought to suggest a plausible cellular mechanism.  It’s also desirable to have some plausible selective pressure that would favor such a genetic predisposition.  Given that survey taking is a relatively recent (in human history) activity, I’m not sure you can make a case for any selective advantage in refusing to participate in surveys.

Maybe–and it’s a big maybe–there’s a selective advantage in some cluster of behaviors–such as cooperation–that just happens to manifest itself in propensity to take surveys.  That might be plausible.  Perhaps the authors offer that explanation in the full paper.  We’ll have to see.

Copyright 2010 by David G. Bakken.  All rights reserved.

Perhaps because he has a consulting business devoted to applying his ideas, Bueno de Mesquita provides only a few details of his method.  We learn that it’s based on game theory in general and John Nash’s contribution to solving games, and that the method requires measuring or estimating the self-interest of each of the players.  Even without the underlying mechanics, we get a sense of the potential of his approach in compelling accounts of application of the model to problems of CEO succession, negotiating with North Korea, plea bargaining, the Israeli-Palestinian conflict, and the near future of Iraqi politics in the aftermath of US troop withdrawals.

One of the best things about this book is what it teaches us about developing and using predictive models.  One chapter is devoted to a kind of validity analysis in which Bueno de Mesquita revisits a few important historical events and lets his model loose on them.  Keeping in mind that an important element of game theory involves inferences about the intentions of others, his model suggests that if Great Britain had signaled its commitment to Serbia in the summer of 1914–perhaps by sending a warship or two into the Adriatic–WWI as we know it might have been avoided, with Austria-Germany accepting less than the surrender of Serbian sovereignty.  In another chapter, writing about the failure of the Clinton administration to bring about healthcare reform, the model is thrown off by an “external shock,”  leading Bueno de Mesquita to the realization that he may need to allow for the impact of random or unanticipated events.  In this specific case, the predicted outcome was dependent on the influential role of Daniel Rostenkowski, the Democratic congressman from Illinois and chairman of the House Ways and Means Committee.  Unfortunately for the Clinton healthcare plan, Rostenkowski was indicted for corruption, throwing a big, big wrench into the works.  The model had been based on the assumption that all of the key players/elements would be constant over the course of congressional bargaining for the legislation.  There was no mechanism for taking this “random shock” into account. 

Bueno de Mesquita is not afraid to stick his neck out.  Towards the end of the book he applies his method to predicting the course that Iraq will follow after US troups are withdrawn.  In particular, he compares a complete withdrawal to a draw down with about 50,000 US troups remaining in Iraq.  In particular, he is interested in whether or not Iraq moves closer to Iran.  Given that we now have committed to keeping about 30,000 troops (not a condition that was tested, or at least not reported), it will be interesting to see how things play out. 

As much as I enjoyed reading this book, I’m not sure how to apply this approach to the typical problems we encounter in marketing.  Game theory has been applied to price competition (and perhaps other interactions between suppliers) but not (or not often) to the interaction between sellers and consumers.  There’s no question that naked self-interest plays a role in these interactions.  However, most consumer purchases don’t involve much negotiation.  For those that do–Bueno do Mesquita offers his proven method for getting the lowest possible price on a new car.  Be sure to check out the first chapter if you’re in the market for a new automobile!

Copyright 2010 by David G. Bakken.  All rights reserved.

For my part, I am more interested in DIY as a symptom of deep and fundamental change in the market research industry.  When I began my career in MR (on the client side at first), most research companies were vertically integrated, owning their own data collection capabilities and developing their own CATI software, for example.  This made sense when the ability to coordinate and integrate the diverse activities required for a typical research project was a competitive strength.  Perhaps you remember the days when a strategic segmentation study might have three or four phases, take six to nine months to complete, and cost $500,000 ( in 1980 dollars!).  But vertically integrated industries tend to “de-integrate” over time.  Firms may spin off or outsource some of their capabilities, creating value chain specialists who are proficient at one link in the chain.  The emergence of WATS call-centers and off-the-shelf CATI software were early steps on the march towards de-integration for the MR industry.

Technological change (especially in the form of disruptive innovation) also provides opportunity for new entrants.  Sure, some of the face-to-face interviewing companies made the transition to telephone, and many telephone interviewing companies successfully converted from paper and pencil questionnaires to CATI, but each of these shifts provided a point of entry for new players.

The large, integrated firms have managed to hang on to a substantial share of industry profits, but there are three looming threats.  The first is (so-called) “commoditization”–the downward pressure on pricing.  While some supplier side researchers complain that clients are unwilling to pay for quality, this downward pressure is the result of basic competitive dynamics:  there are many competing firms, few barriers to entry, many substitutes (e.g., transactional datamining) and not that much difference in value propositions or business models across MR firms.

The second threat is do-it-yourself research.  At the moment, DIY appeals to the least demanding and most price sensitive customers.  DIY removes the access and affordability barriers, thereby democratizing survey research.  As Lucy and Richard’s research showed, customers like the low cost, speed and convenience of DIY, and I expect many will move up the learning curve quickly.  I hope so–many of the DIY surveys I’ve seen from even big companies have been pretty ghastly. 

The last threat to the traditional MR business model comes from the sheer deluge of data generate by both commercial and non-commercial online activity.  How much could Google tell any marketer about customer preferences based just on search data, for example?

At the end of the session in Athens I offered this analogy.  Imagine that you need a bedstead.  You could go to a furniture store and choose from a selection of attractive, well-constructed and expensive bedsteads.  Or you could go to the local home improvement store, purchase some plywood and paint or stain and with a few tools (which could be borrowed or renterd) and some minimal ability, construct a perfectly serviceable platform bed–at much lower cost.  This represents the difference between the full service integrated research firms at the top of the latter and what we’ve historically thought of as do-it-yourself market research.  The gap between the two has been sustained until now by a skill barrier and limited access to better, easier to use tools.  This is the gap that Ikea filled in the home furnishing market by creating a new business model based on attractive, customer-assembled furnishings. 

Unfortunately for the incumbent research firms, this kind of business model innovation does not often come from the current players in a market.  The incumbents have too much personal investment in the current business model.  Let’s face it–most of us are in market research because we like the high-touch, intellectual problem solving that’s involved.  It’s what we’ve trained to do.  Designing something like appealing flatpack furniture that customers take home and assemble themselves just does not fit our self-image.

The smarter, easier to use tools are here.  Who will be the first to package them into a new way to deliver market research?

Copyright 2010 by David G. Bakken.  All rights reserved.