Think like a respondent: improving MR ROI through better survey design

There’s no question that marketers are more focused than ever on the ROI of marketing research.  All too often, however, it seems that efforts to improve ROI aim to get more research per dollar spent rather than better research. 

Better survey design is one sure way to improve the ROI of marketing research.  However, despite advances in our understanding of the cognitive processes involved in answering surveys, market researchers continue to write poor survey questions that may introduce considerable measurement error. 

I think this is due in part to the fact that the processes involved in asking a question are fundamentally different from the processes involved in answering that same question.  Recent contributions to our understanding of the answering process have been integrated into a theory of survey response by Roger Tourangeau, Lance J. Rips, and Kenneth Rasinski (The Psychology of Survey Response, Cambridge University Press, 2000).  According to Tourangeau, et. al., answering a survey question involves four related processes:  comprehending the question; retrieving relevant information from memory, evaluating the retrieved information, and matching the internally generated answer to the available responses in the survey question.

“Think aloud” pretesting, sometimes known as “cognitive” pretesting or “concurrent protocol analysis” is an important tool for improving the quality of survey questions, and  well-designed think aloud pretests often have, been in my experience, the difference between research that impacts a firm’s business results and research that ends up on the shelf for lack of confidence in the findings.

Warning–what follows is blatant self-promotion of a sort.  ESOMAR is offering my workshop, “Think like a respondent:  A cognitive approach to designing and testing online questionnaires” as part of Congress 2011.  The workshop is scheduled for Sunday, September 18, 2011. This year’s Congress will be held in Amsterdam.  I’ve offered the workshop once before, at the ESOMAR Online Conference in Berlin last October.

Hope to see you in Amsterdam.

 

The “decline effect,” random variation, and evidence-based marketing

There’s an interesting article by Jonah Lehrer in the Dec. 13 issue of The New Yorker, “The Truth Wears Off:  Is there something wrong with the scientific method?” Lehrer reports that a growing number of scientists are concerned about what psychologist Joseph Banks Rhine termed the “decline effect.”  In a nutshell, the “decline effect” is an observed tendency for the size of an observed effect to decline over the course of studies attempting to replicate that effect.  Lehrer cites examples from studies of the clinical outcomes for a class of once-promising antipsychotic drugs as well as from more theoretical research.  This is a scary situation given the inferential nature of most scientific research.  Each set of observations represents an opportunity to disconfirm a hypothesis.  As long as subsequent observations don’t lead to disconfirmation, our confidence in the hypothesis grows.  The decline effect suggests that replication is more likely, over time, to disconfirm a hypothesis than not.  Under those circumstances, it’s hard to develop sound theory.

Given that market researchers apply much of the same reasoning as scientists in deciding what’s an effect and what isn’t, the decline effect is a serious threat to creating customer knowledge and making evidence-based marketing decisions.

 

Are market research agencies the “back room” to the back room?

An insightful new report from Boston Consulting Group reveals that “most companies have not yet unlocked the value of consumer insight.”  The report is based on a quantitative survey of more than 800 executives from 40 global companies with at least $1.5 billion in sales.  The survey was supplemented with around 200 qualitative interviews, and the participants included line managers as well as members of the consumer insight function in these companies.

The authors found that companies fall into one of four stages of consumer insight capability:

• traditional market research function
• business contribution team
• strategic insight organization
• strategic foresight organization.

The companies falling into the last two stages are getting the biggest return on their investments in consumer insight.  However, according to this report, only about 10% of the surveyed companies are in one of these two stages of insight capability.  In Stage 1 companies, the insight function is more or less an “order taker” relegated to “back room” status, and the focus is on tactical research.  Things are a little better in Stage 2 companies in that  sometimes projects are more strategic, but the insight function is still project-focused.

If the consumer insight function is relegated to back room status in the majority of companies, does that make research agencies a back room to the back room?

 

Seeing the Story in the Data

The New York Times is one of the more interesting innovators when it comes to using data visualization to tell a story or make a point.  In particular, the Business section employs a variety of chart forms to reveal what is happening in financial markets.  The Weather Report uses “small multiples” to show 10-day temperature trend for major U.S. Cities.

Even more interesting are the occasional illustrations that appear under the heading of “Op-Chart.”  For a few years now the Times periodically presents on the Op-Ed page a comparative table that tracks “progress” in Iraq on a number of measures such as electric power generation.

Another impressive chart appeared in “Sunday Opinion” on January 10, 2010.  Titled “A Year in Iraq and Afghanistan,” this full page illustration provides a detailed look at the 489 American and allied deaths that occurred in Afghanistan and the 141 deaths in Iraq.  At first glance, the chart resembles the Periodic Table of Elements.  Deaths in Iraq take up the top one-fourth or so of the chart (along with the legend); deaths in Afghanistan occupy the bulk of the illustration.

Each death is represented by a figure, and each figure appears in a box representing the date which the death occurred. One figure shape represents American forces, and a slightly different shape signifies a member of the coalition forces.  For coalition forces, the color of the figure indicates nationality.  A small symbol indicates the cause of each death (homemade bomb, mortar, hostile fire, bomb, suicide bomb, or non-combat related).  Multiple deaths from the same event or cause on a date occupy the same box.

Most dates have only a single death, but a few days standout as particularly tragic:  seven U.S. troops dying due to a non-combat related cause in Afghanistan on October 26; eight killed by hostile fire on October 3rd; seven killed by a homemade bomb on October 27; six Italians killed by a homemade bomb on September 17; five Americans killed by a suicide bomber in Mosul, Iraq, on April 10.

The deaths are linked to specific locations on maps of Iraq and Afghanistan.  Helmand Province was the deadliest place, with 79 of the 489 deaths in Afghanistan.  In Iraq, Baghdad was the most dangerous place, accounting for 42 of the 141 deaths in that country.  While Americans are the largest number, 112 of the dead in Afghanistan were British troops.

There is a wealth of information in this chart with four pieces of information on every death, but in some ways there is too much detail.  To get at the numbers I provided above, I had to manually count the pictures.  There are no summary statistics.  The picture grabs our attention, and immediately conveys the magnitude of the price the U.S. and our allies are paying in Afghanistan.   But if we want to act on data, we need a little more than just a very clever visual display.  Summaries of the numbers would help, here.  It’s useful to know, for example, that 65 of the 141 deaths in Iraq (46%) were due to non-combat related causes, compared to 48 (10%) of the deaths in Afghanistan.  Eighty percent of the fatalities in deadly Helmand province were due to hostile fire; 57% in other parts of Afghanistan were caused by homemade bombs (in Iraq there were 19 deaths, or 13% of the total, from homemade bombs).

Two of the creators of this chart, Adriana Lins de Albuquerque (a doctoral student in political science at Columbia) and Alicia Cheng of mgmt.design, produced a slightly different version of this chart summarizing the death toll in Iraq for 2007 (click here).  That earlier version did not have as much detail about each individual death (location information is not included, for example) but includes some additional causes, like torture and beheading that, thankfully, appear to have disappeared.

The advantage to displaying data in this fashion lies in the ability of our brains to form patterns quickly.  The use of color to designate coalition members makes the contributions of our allies apparent in a way that a simple tally might not.  Even without a year-to-year comparison, we can see that Iraq has become, at least for US troops and our allies, a much safer place than Afghanistan.  Additionally, this one chart presents data that, in other forms, might require several PowerPoint slides to communicate: deaths by date, deaths by city or province, deaths by nationality, causes of death, number killed per incident, and cause of death.

Any complex visual display of data requires making trade-offs.  In this case, for example, the creators arranged the deaths chronologically (oldest first) within each geographic block.  That means that patterns in other variables, such as cause of death or nationality of troops, may be harder to detect on first glance.  The chronological ordering has layout implications, since on some dates there were multiple casualties.

All in all, it’s a great piece of data visualization that to my mind would be even better with the addition of a few summary statistics.

A disclaimer–I counted twice to get each of the numbers I provide above, but I offer no guarantee that I am not off by one or two deaths in any of those numbers.

Copyright 2010 by David G. Bakken.  All rights reserved.

 

D-I-Y Market Research: Worse than no research at all?

I just completed an online survey at the invitation of a company I’ve purchased from in the past.  It was obvious that the survey was an example of what the market research industry calls “D-I-Y” research.  If the quality of the questionnaire had not given this away, there was the “Powered by [name of enterprise feedback software vendor]” at the bottom of the screen.  I was asked to look at two different print ads for one of the products this company sells and answer a few questions that bore some slight resemblance to the questions you might find in an ad test conducted by one of the MR firms that specialize in that type of work.

One can only assume that the results of this survey are meant to drive a decision of which ad to run (there may be other candidates that I didn’t see).  If that’s true, then I think this may be a case where D-I-Y will turn out to be worse than no research at all.  The acid test for any market research is whether or not the decisions made on the basis of that research are “better” than the decision that would have been made without the research.

 

Online survey research–how accurate?

The debate over the accuracy–and quality–of survey research conducted online is flaring at the moment, at least partly in response to a paper by Yeager, Krosnick, Chang, Javitz. Levendusky, Simpson and Wang: “Comparing the accuracy of RDD telephone surveys and Internet surveys conducted with probability and non-probability samples.”  Gary Langer, director of polling at ABC News, wrote about the paper in his blog “The Numbers” on September 1. In a nutshell, the paper compares survey results obtained via random-digit dialing (RDD) with those from an Internet panel where panelists were recruited originally by means of RDD and from a number of “opt-in” Internet panels where panelists were “sourced” in a variety of ways.   The results produced by the probability sampling methods are, according to the authors, more accurate than those obtained from the non-probability Internet samples.  You can find a response from Doug Rivers, CEO of YouGov/Polimetrix (and Professor of Political Science at Stanford) at “The Numbers,” as well as some other comments.

The analysis presented in the paper is based on surveys conducted in 2004/5.  In recent years the coverage of the RDD sampling frame has deteriorated as the number of cellphone-only users has increased (to 20% currently).  In response to concerns of several major advertisers about the quality of online panel data, the Advertising Research Foundation (ARF) established an Online Research Quality Council and just this past year conducted new research comparing online panels with RDD telephone samples.  Joel Rubinson, Chief Research Office of The ARF, has summarized some of the key findings in a blog post. According to Rubinson, this study reveals no clear pattern of greater accuracy for the RDD sample.  There are, of course, differences in the two studies, both in purpose and method, but it seems that we can no longer assume that RDD samples represent the best benchmark against which to compare all other samples.

 

Who knows what happiness lurks in the hearts of men? Facebook knows.

Have you heard about the Facebook Gross National Happiness Index?  On Monday, October 12, the Times ran an article (by Noam Cohen) reporting some of the findings based on analysis of two years’ worth of Facebook status updates from 100 million users in the U.S.  The index was created by Adam D. I. Kramer, a doctoral candidate in social psychology at the University of Oregon, and is based on counts of positive and negative words in status updates.  According to the article, classification of words as positive or negative is based on the Linguistic Inquiry and Word Count dictionary.

Among the researchers’ conclusions:  we’re happier on Fridays than on Mondays; holidays also make Americans happy.  The premature death of a celebrity may make us sad.  According to a post by Mr. Kramer on the Facebook blog, the two “saddest” days–days with the highest numbers of negative words–were the days on which actor Heath Ledger and pop icon Michael Jackson died.  Mr. Kramer points out that, coincidentally, Mr. Ledger died on the day of the Asian stock market crash, which might have contributed to the degree of negativity.

We’re going to see a lot more of this kind of thing as researchers delve into the rich trove of information generated by users of search engines and web-enabled social networking.  The happiness index, based as it is on simple frequency analysis of words, is the tip of the iceberg.  At the moment, “social media”–I’m not exactly sure what that label means–is getting incredible attention in the marketing and marketing research community.  The question that has yet to be posed, let alone answered, is, “what exactly do we learn from all this information?”

 

Recommended reading: The Psychology of Survey Response

The Psychology of Survey Response by Roger Tourangeau, Lance J. Rips, and Kenneth Raskinski (Cambridge University Press, 2000) will change the way you think about the “craft” of survey design.  While there are other, well-regarded books on survey question construction (such as Asking Questions by Norman Bradburn, Seymour Sudman, and Brian Wansink, Jossey-Bass, 2004) and tons of individual research papers and articles on various aspects of survey design, measurement scales, question construction and the like, this is the first book I’ve encountered that presents a practical conceptual framework for understanding the cognitive processes that produce a response to a given question.  Moreover, the authors review a lot of relevant research to support their framework.

 

Does Your Innovation Process Start with Your Customers?

How does your organization generate ideas for new products, services, or business processes?  Many of the companies I’ve worked with over the years rely largely on internal sources for new ideas.  If the ideas come from R&D, they are most likely responses to specific technical challenges or attempts to find applications for inventions.  If the ideas come from product marketing, they are often aimed at creating points of difference versus competitors.  Of course, some innovations result from the combination of technical solutions with product or service differentiation.

Looking at innovation from the technical side of the equation, it’s useful to think of a continuum that ranges from small, incremental improvements in existing solutions (applying a bit of adhesive to notepads to create Post-It Notes) to innovation that springs from discovery of a new phenomenon (such as nano-technology).  Most innovations that make it into the marketplace are near the incremental end of this continuum.  It can takes years or perhaps decades for the discovery of a new phenomenon to result in commercially viable products or services.

It’s a truism that most new products fail–estimates range from 80 to 90% of consumer products.  I once heard the chief marketing officer of a consumer products company cite this statistic and then go on to say that his company’s solution was a fourfold increase in the number of products they planned to introduce.  The products that succeed are those that do a job for customers that either is not being done by existing solutions or not done well.

The idea that customers “hire” products and services to do jobs for them is not new.  In an article in the December, 2005 issue of Harvard Business Review (“Marketing Malpractice:  The Cause and the Cure”) Clayton Christensen, Scott Cook and Taddy Hall remind us that Theodore Levitt would tell his students that customers “don’t want to buy a quarter-inch drill.  They want a quarter-inch hole!”

Some innovations do not change the job so much as the way the job is done.  Before FedEx, people sent documents and packages, but they seldom sent them overnight.  And now, many of the documents that were sent as FedEx overnight letters are transmitted within seconds or minutes as attachments to email.  These are examples of innovations that completely changed the way the job was done.

Whether the job is existing or emerging (more on emerging jobs in a moment), successful innovation depends on understanding the jobs that customers want to perform.  The challenge, form the customer knowledge perspective, is identifying and categorizing the jobs in a way that systematically informs innovation efforts.  Think of the job a customer wants to do as a demand creating condition. We can operationalize demand creating conditions as the concerns and interests that lead individuals to their everyday pursuits, and may lead to behavior in the marketplace, such as a search for a product that does a particular job.  Emerging jobs are those that result from underlying structural changes, such as an increase in the number of mothers of school age children working out of the home giving rise to a host of new “management” challenges for those moms, and opportunities for innovations like mobile telephones.

Geraldine Fennell is a consultant based in Ireland who has developed a framework for understanding consumer motivations. Along with Geraldine and her collaborator Greg Allenby, I designed a study to apply this framework to brand choices for automobiles.  Geraldine breaks consumer motivations into seven different categories, some of which are “sticks” (things we want to avoid) and some of which are “carrots” (things we want to approach).  The sticks include:  solving immediate problems; preventing potential problems; and maintaining the status quo.  The carrots include:  exploratory opportunities and sensory opportunities.  You’re thinking–that’s only five categories.  You’re right.  These five categories are independent of the focal activity.  They apply whether the job I’m doing involves drilling holes in a piece of wood or getting to and from the grocery store. Two additional categories reflect specific experiences in doing jobs:  dissatisfaction in use (with the product or service hired to do the job) and ineffective or frustrating outcomes (when no product or service exists that does the job well).

Companies often fail to identify emerging jobs and jobs that are not done well by existing products and services because they frame the question in terms of the current or existing marketplace.   As an example, the typical “needs-based” segmentation for automobiles will enumerate benefits and associated product features, such as “interior storage,” “rear seat legroom” and “fuel economy” and ask consumers how important or appealing each of these is when choosing a vehicle.  This approach leads to small, incremental improvements in features and benefits.  You won’t hit on an innovation like cupholders by asking consumers how important interior storage is.

A good starting point is qualitative research (and I recommend individual in-depth interviews rather than focus groups) organized around the stick and carrot categories I listed above.  Here are statements reflecting some of the “sticks” that might affect our automotive choices:

• I’m concerned about getting from point A to point B without getting injured (solving an immediate problem)
• I’m concerned about the impact I have on the environment (preventing future problems)
• Driving is no big deal for me (maintaining stable state)

Here are some “carrots:”

• I am easily bored when I drive (exploratory opportunity)
• It’s important for driving to be fun (sensory opportunity)

And here are examples for the last two categories:

• I’m concerned about the mechanical reliability of the car I drive (dissatisfaction in use)
• It’s difficult to find a vehicle that meets all my driving needs (ineffective/frustrating outcome)

You can use these categories to guide the discussion, even listing them for the interviewee.  A second step involves matching the concerns and interests to specific usage or consumption occasions.  In our case, we asked consumers whether each concern was an issue in usage occasions such as traveling to the store for groceries alone, with children, or with other adults (three separate occasions).

To briefly summarize the study results, we found that a segmentation based on these (and additional) concerns and issues helped explain preferences for luxury car makes.  As an example, preference for Mercedes was strongest in a segment that was concerned both about safety and operating costs.  Volvo also did reasonably well in this segment, and also in a segment of consumers who are concerned about their impact on the environment.  Both of these brands have advertised the safety of their vehicles.

This might seem like a “traditional” needs based segmentation but remember–we did not ask consumers how important it was that a vehicle have certain safety features.  We asked them whether they had any concerns about their safety in each of several usage occasions.

When it comes to innovation, this type of information can help a company focus resources on opportunities that align with consumer motivations.  Each one of these concern-occasion intersections is a potential job that a consumer is trying to do.

Practically every “breakthrough” or game-changing innovation I can think of either enabled the customer to do a new job, or did an existing job better than any existing solution.  The best way to increase the odds of successful innovation is to start with the customer.

Copyright 2009 by David G. Bakken.  All rights reserved.

 

The Path to Customer Knowledge

If we’re going to create a pathway to customer knowledge, we ought to start with some kind of definition, so we’ll know where we’re trying to go.  One might argue that anything we “know”–any facts that we have about customers–constitutes “knowledge.” At one level that’s true.  But, as I’ve suggested in an earlier post (“The Advantage in Customer Knowledge,” May 21, 2009) knowledge is more than just a collection of facts.  In our ESOMAR paper “Creating Customer Knowledge” (ESOMAR Consumer Insights, 2004), Mike Lotti and I defined customer knowledge as “the understanding of customer motivations, attitudes, perceptions, and experiences such that we can predict customer behavior.”  In other words, customer “knowledge” really represents our “theory of the customer.”

Almost all marketers have one or more implicit models of customer behavior that guides their decision-making.  In many cases, these models are developed on the basis of limited information, combined with specific psychological processes that we use to help us explain and predict the behavior of other people.  These processes are sometimes referred to as implicit psychology and they affect our perceptions of others in predictable and biased ways.  For a good introduction see Implicit Psychology:  An Introduction to Social Cognition by Daniel M. Wegner and Robin Vallacher (Oxford University Press, 1977, still in print).

To create our theory of the customer we need to mimic the explicit process represented in the scientific method we all learned in school.  Around the time Mike and I were preparing our paper, an article by Clay Christensen and Michael Raynor appeared in the Harvard Business Review (“Why Hard-nosed Executives Should Care About Management Theory,” September, 2003) describing this process as applied to management theory.  Theory building begins with observation and description of the phenomenon of interest.  This has been a major role of market research–to observe and describe the behavior of customers.  The second step is categorization of the observations and descriptions, perhaps revealing patterns of association or correlations in the data.  The third step is the development of predictions or hypotheses about causal relationships.

Despite all the data that are available about customers from transactional databases, surveys, focus groups, and so forth, many organizations are unable to capitalize on opportunities to create customer knowledge.  There are at least four reasons for this.

• Much data gathering is aimed at answering specific and urgent tactical questions.  Research may be conducted to populate marketing dashboards, to measure reactions to new product ideas or to gauge the impact of specific marketing actions (ad copy, price changes, etc.), but research is seldom conducted specifically to develop and test theories of customer behavior. That does happen in the academic community, but sometimes such work is too theoretical or focused on too narrow a problem.  At any rate, once the immediate question is answered, management’s attention quickly turns to the next tactical problem.
• Most data gathering is project-specific.  This is a consequence of the combined effects of the tactical and decisive nature of most research and the budgeting process in most firms.  I helped create business unit market research budgets when I worked for AT&T, and it was a process of estimating the number of concept tests we would do in a year, ad copy tests, customer satisfaction tracking studies, and so forth.  No one ever said “How much do we need to spend to understand our customers?”  This makes it difficult to integrate multiple studies into the kind of programmatic research–typical in the academic world–that is essential to the development and refinement of theory.  Granted, some firms are adopting knowledge management technology to facilitate retrieval of information that has been gathered across multiple projects, but that’s a far cry from systematically designing research to build on what’s known and to test specific propositions about customers.
• Market researchers and users of market research too often equate correlation with causation (there’s a related problem in that we sometimes see illusory correlations–associations that we expect to find based on our implicit models that are not actually present in the data.  If you don’t think this happens, take a look at How Doctors Think by Jerome Groopman).  As our analytic methods and our computers have become more powerful we’ve come to rely almost exclusively on number crunching to to develop “predictive models” of customer behavior.  
• The internal politics of the organization may derail even the best intentioned and executed knowledge creation efforts.  Different stakeholders have different motivations, and they often have different ways of gathering and interpreting data from customers (think of the way the sales force thinks about customers compared to the way a product marketing manager or brand manager thinks about customers).  In some organizations, much of the customer data is “owned” by the management information systems or information technology groups.  

Market segmentation research offers a good example of the ways in which companies can squander opportunities to create customer knowledge.  There are many varieties of market segmentation, but the original paper on the topic by Smith (“Product Differentiation and Market Segmentation as Alternative Marketing Strategies,” Journal of Marketing, 21, 3-8, 1956) described an orientation based on understanding the varying motivations that buyers bring to the marketplace.  In this view, segmentation requires an in-depth exploration of consumer motivations outside the marketplace.  This is a critical distinction. As Greg Allenby of The Ohio State University and his collaborator Geraldine Fennell point out, demand creating conditions–the problems that consumers are trying to solve to improve their well-being–are spawned upstream of the marketplace.  

Best-practice market segmentation requires a multi-stage research solution, beginning with broad qualitative research to understand the potential variability in consumer motivations, followed by more qualitative to refine the hypotheses generated in the first round, and finally, carefully designed quantitative research to put those hypotheses to the test.  What we’d really like to see is quantitative that allows us to test competing hypotheses about the drivers of consumer behavior, rather than confirmation or disconfirmation of a single hypothesis.  This tends to make best-practice market segmentation (where the goal is understanding variability in consumer motivations, rather than simply targeting consumers based on differential response to marketing efforts) complex and expensive, especially in today’s environment when firms need to understand their customers on a global basis.

In my recent experience, many firms are “doing” segmentation.  That is, they are commissioning their research supplier partners to conduct segmentation research.  All too often that involves a kitchen sink set of questions and whatever method the supplier recommends for clustering survey respondents based on the answers to those questions.  Good segmentation work is still being done, but there is a lot of poorly conceived segmentation work happening as well.

What’s missing in most segmentation research today is the systematic upfront development of theory to explain variability in consumer motivations.  Quantitative segmentation studies should be confirmatory, not exploratory.  If you don’t have a pretty good idea of the conceptual dimensions (or “constructs”) that will define the segments going in, you’re likely to be disappointed with the results of a quantitative segmentation study.

Look for more on best practices in market segmentation in future posts.

Copyright 2009 by David G. Bakken.  All rights reserved.