Why I’m running for ESOMAR Council

CareerCast.Com has released its list of “most underrated jobs” for 2012.  Number 5 on the list?–Market research analyst.  Ahead of MR on the list are computer systems analysts, civil engineers, veterinarians, and biologists.  These jobs make the list on the basis of projected growth in employment, relatively good compensation, and lower stress levels than more glamorous or otherwise high-profile occupations.  Here’s what CareerCast says about market research as a career:  “One of the fastest growing fields per the [Bureau of Labor Statistics], market research analyst makes a vital impact on the direction of business decisions by applying data of economic and technological trends.” (emphasis added)

ESOMAR Congress 2012 took place in Atlanta, Georgia during the second week in September.  This was the first time the Congress has been held in the United States, reflecting the extent to which ESOMAR has become the most important global organization of market and social researchers.  This also happens to be the 65th anniversary year for ESOMAR.

The current and past ESOMAR Councils, together with the Director General, Finn Rabin, and the rest of the ESOMAR team, have done a terrific job in building a robust and resilient organization and a public voice for market research.  My desire to serve ESOMAR as a council member does not reflect any dissatisfaction with the direction of the organization.  Rather, I believe that I can make a contribution to the continued growth and health of ESOMAR.

My involvement in ESOMAR has grown steadily since I attended my first ESOMAR-sponsored conference ten years ago (Automotive–where I had the privilege of presenting my first ESOMAR paper).  More papers and events followed (Consumer Insights, Asia Pacific, and Congress).  I was honored beyond imagination to have my 2010 Congress paper, “Riding the Value Shift in Market Research,” selected for the Excellence Award.  More recently I’ve served on the juries for the Effectiveness Award and for this year’s Excellence Award and I presented a workshop on the cognitive aspects of survey design (“Think Like a Respondent”) at the Online conference in 2010.

Outside of ESOMAR I’ve been involved with the American Marketing Association, CASRO, and the American Psychological Association for many years.  I served as president of my local AMA chapter, leading the successful turnaround of a struggling chapter.

As I expressed in my Candidate Statement, I have three particular areas of interest that support ESOMAR’s overall mission of encouraging, advancing, and elevating market research throughout the world.  These areas are:  professional development, collaboration with other related MR organizations, and finding new business models that will enable sustained growth for MR.

ESOMAR’s continued growth will depend on both capturing and reflecting the diversity of global market research.  The election rules insure a measure of geographical diversity.   It’s equally important to have diversity of experience and industry perspective.  I believe that I bring a unique point of view–as do the other nominees–that will help me contribute even more to ESOMAR’s future success if I am fortunate enough to serve as a council member.

Thank you!

dgb

2 October 2012

 

Think like a respondent: improving MR ROI through better survey design

There’s no question that marketers are more focused than ever on the ROI of marketing research.  All too often, however, it seems that efforts to improve ROI aim to get more research per dollar spent rather than better research. 

Better survey design is one sure way to improve the ROI of marketing research.  However, despite advances in our understanding of the cognitive processes involved in answering surveys, market researchers continue to write poor survey questions that may introduce considerable measurement error. 

I think this is due in part to the fact that the processes involved in asking a question are fundamentally different from the processes involved in answering that same question.  Recent contributions to our understanding of the answering process have been integrated into a theory of survey response by Roger Tourangeau, Lance J. Rips, and Kenneth Rasinski (The Psychology of Survey Response, Cambridge University Press, 2000).  According to Tourangeau, et. al., answering a survey question involves four related processes:  comprehending the question; retrieving relevant information from memory, evaluating the retrieved information, and matching the internally generated answer to the available responses in the survey question.

“Think aloud” pretesting, sometimes known as “cognitive” pretesting or “concurrent protocol analysis” is an important tool for improving the quality of survey questions, and  well-designed think aloud pretests often have, been in my experience, the difference between research that impacts a firm’s business results and research that ends up on the shelf for lack of confidence in the findings.

Warning–what follows is blatant self-promotion of a sort.  ESOMAR is offering my workshop, “Think like a respondent:  A cognitive approach to designing and testing online questionnaires” as part of Congress 2011.  The workshop is scheduled for Sunday, September 18, 2011. This year’s Congress will be held in Amsterdam.  I’ve offered the workshop once before, at the ESOMAR Online Conference in Berlin last October.

Hope to see you in Amsterdam.

 

David Leonhardt’s Interactive Deficit Puzzle: New Way to Measure Opinion?

In November of last year David Leonhardt, an economics writer for The New York Times, created an “interactive puzzle” that enabled readers to create a solution for reducing the federal deficit by $1.3 trillion (or therebouts) in 2030.  A variety of options involving either spending cuts or tax increases that reflected the recommendations of the deficit reduction commission were offered, along with the size of the reduction associated with each option.  Visitors to the puzzle simplyselected various options until they achieved the targeted reduction.

The options represented trade-offs, the simplest being that between cutting programs or raising revenues.  Someone has to suffer, and suffering was not evenly distributed across the options.  Nearly seven thousand Twitter users completed the puzzle, and Leonhardt has summarized the choices.  You might still be able to access the puzzle online at nytimes.com/weekinreview.

Leonhardt was able to group the solutions according to whether they seemed to consist mostly of spending cuts or a mix of spending cuts and tax increases.  He admits that the “sample” is not scientific and, given that it’s comprised of Twitter users, may skew young.  Unfortunately, no personal data was collected from those who completed the puzzle, so we’re left to speculate about the patterns of choices.  Perhaps a little data mining would shed some additional light on the clustering of responses. 

Even though this is not survey resarch in the way that we know it, there may be much value in using this type of puzzle to measure public opinion about the tough choices that the U.S. is facing.  The typical opinion survey might ask respondents whether they “favor” one course of action or another (“Do you favor spending cuts or tax increases for reducing the deficit?”).  The options presented in Leonhardt’s puzzle represent real policy choices, and the differences between them force you to consider the trade-offs you are willing to make.  While the choices were comprehensive, they were not contrived in the way that conjoint analysis structures choices; that might present a problem if we are trying to develop a model to predict or explain preferences.

There’s no reason this technique cannot be used with the same kinds of samples that we obtain for much online survey research.  Add a few demographic and political orientation questions and you have what I think could be a powerful way to capture the trade-offs that the public is willing to make.

Copyright 2011 by David G. Bakken.  All rights reserved.

 

What dialect do you Tweet?

If you were lisitening to NPR’s “All Things Considered” broadcast on January 18, you might have heard a brief report on research that reveals regional differences (“dialects”) in word usage, spellings, slang and abbreviations in Twitter postings.  For example, Northern and Southern California use spelling variants koo and coo to mean “cool.”

Finding regional differences in these written expressions is interesting in its own right, but I’ve just finished reading the paper describing this research and there’s a lot more going on here than simply counting and comparing expressions across different geographic regions.  The paper is an excellent example of what market researchers might do to analyze social media.

The study authors–Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing–are affiliated with the School of Computer Science at Carnegie Mellon University (Eisenstein, who was interviewed for the ATC broadcast, is a postdoctoral fellow).  They set out to develop a latent variable model to predict an author’s geographic location from the characteristics of text messages.  As they point out, there work is unique in that they use raw text data (although “tokenized”) as input to the modeling.  They develop and compare a few different models, including a “geographic topic model” that incorporates the interaction between base topics (such as sports) and an author’s geographic location as well as additional latent variable models:  a “mixture of unigrams” (model assumes a single topic) and a “supervised linear Dirichlet allocation.”    If you have not yet figured it out, the models, as described, use statistical machine learning methods.  That means that some of the terminology may be unfamiliar to market researchers, but the description of the algorithm for the geographic topic model resembles the hierarchical Bayesian methods using the Gibb’s sampler that have come into fairly wide use in market research (especially for choice-based conjoint analysis).

This research is important for market research because it demonstrates a method for estimating characteristics of individual authors from the characteristics of their social media postings.  While we have not exhausted the potential of simpler methods (frequency and sentiment analyses, for example), this looks like the future of social media analysis for marketing.

Copyright 2011 by David G. Bakken.  All rights reserved.

 

The “decline effect,” random variation, and evidence-based marketing

There’s an interesting article by Jonah Lehrer in the Dec. 13 issue of The New Yorker, “The Truth Wears Off:  Is there something wrong with the scientific method?” Lehrer reports that a growing number of scientists are concerned about what psychologist Joseph Banks Rhine termed the “decline effect.”  In a nutshell, the “decline effect” is an observed tendency for the size of an observed effect to decline over the course of studies attempting to replicate that effect.  Lehrer cites examples from studies of the clinical outcomes for a class of once-promising antipsychotic drugs as well as from more theoretical research.  This is a scary situation given the inferential nature of most scientific research.  Each set of observations represents an opportunity to disconfirm a hypothesis.  As long as subsequent observations don’t lead to disconfirmation, our confidence in the hypothesis grows.  The decline effect suggests that replication is more likely, over time, to disconfirm a hypothesis than not.  Under those circumstances, it’s hard to develop sound theory.

Given that market researchers apply much of the same reasoning as scientists in deciding what’s an effect and what isn’t, the decline effect is a serious threat to creating customer knowledge and making evidence-based marketing decisions.

 

Starbucks new logo

Last week Starbucks CEO Howard Schultz unveiled a new logo for the brand.  By now we should know that any attempt to mess with a popular brand (formula, name or logo) is going to elicit at least a few expressions along the lines of “what are they thinking?”

Schultz tried to communicate just what they were thinking in an online video, saying that the new design respects the brand’s hearitage but also looks toward a future for the firm that’s not just coffee.  And on the Starbucks website one of the company’s creative managers, calling the logo change “the project of a lifetime,” says:  “From the start, we wanted to recognize and honor the important equities of the iconic Starbucks logo.”

Firms do change logos without the flack that The Gap generated a couple of months ago, but that change (abandoned in the wake of criticism) and Starbucks new logo may offer important lessons about when and why to change logos, and what not to change.  As I’ve expressed in a previous post, I’m skeptical about neuromarketing, but logos are one area where I think companies might do well to invest in a little neuroscience before making big changes.  Logos provide a sort of cognitive shortcut, and we humans are cognitive misers.  The more “iconic” a logo is, the more effective the cognitive shortcut.  To be iconic, a logo needs to be distinctive (not easily confused with other symbols) and consistently associated with with a specific set of brand experiences.  Moreover, the iconic elements of a logo are usually basic properties like shape, color, typeface and spatial relationships within the logo.  The true test of an iconic element is the extent to which you can minimize or degrade the image (showing just a small part, for example) and evoke the complete logo or the associated brand.  For example, in the hit CBS drama NCIS the main characters drink coffee from cups that have some sort of green and black circular logo with white lettering–just enough to suggest Starbucks.  I think one problem with The Gap’s proposed logo was that it changed several iconic elements at once (background color, typeface, and spatial relationships within the logo).

Having been on the inside of a few re-positioning efforts–some involving new logos or package designs–I think that we often over-intellectualize the meaning of the elements in a logo.  Starbucks appears to believe that the mermaid (or siren) in the center of the logo is the symbol of the brand, but some of the comments posted to their website indicate that many customers never really noticed that the element in the center of the logo was a mermaid.

I don’t know whether Starbucks (or The Gap, for that matter) did any consumer research in the course of developing their new logos.  However, the typical research methods for this type of thing rely on verbal responses, while logos operate mostly at a nonverbal level.  Perceptual research methods are more appropriate for this type of problem than focus groups and surveys.

Many companies have modified their logos, of course, but those that succeed seem to recognize that it’s more important to understand the Gestalt of the logo than any potential symbolic meaning of the logo or individual elements.  You can tweak a typeface, for example, to look more modern, or play with the saturation and tint of the color scheme, but if several elements change at the same time, or the spatial relationships change, you can expect the kind of feedback that The Gap and Starbucks have experienced.

Starbucks may well stick by its new logo.  In that case, I suspect customers still will be able to find a store when they want, or recognize Starbucks coffee on their supermarket shelf.  My main issue with these logo changes is that they are superficial.  Consumers don’t buy “Starbucks” or “the siren”–they buy a particular constellation of experiences and solutions to problems.  If you get the latter right and you’re lucky enough to have an iconic logo, why abandon it?

Copyright 2011 by David G. Bakken.  All rights reserved.

 

A Survey Gene?

Declining response rates have a been a problem in survey research for a long time.  Now a study by Lori Foster Thompson of North Carolina State University, Zhen Zhang of Arizona State University, and Richard D. Arvey of National University of Singapore, there may be a genetic predisposition to decline to participate in surveys.  Or maybe not.

The study, “Genetic underpinnings of survey response,” is to be published in the Journal of Organizational Behavior. A press release  from North Carolina State University quotes Dr. Foster:  “We wanted to know whether people are genetically predisposed to ignore requests for survey participation.  We found that there is a pretty strong genetic predisposition to not reply to surveys.”

The researchers  sent a survey to more that 1,000 sets of twins, some identical (and possessing identical DNA) and some fraternal (no more genetically similar than any two siblings).  The study found that the it was possible to predict the propensity to respond for one identical twin from the response (or non-response) of the other twin, but there was no such relationship for the fraternal twins.  The researchers “used quantitative genetic techniques to estimate the genetic, shared environmental, and nonshared environmental effects on people’s compliance with the request for survey participation” according to the paper abstract.

Notwithstanding the power of the right statistical methods, it’s very difficult to rule out plausible rival hypotheses in single generation familial inheritance studies.  I spent one summer during graduate school analyzing data from an adoption study attempting to prove the heritability of schizophrenia.  In addition to the adoption paradigm (that is, looking for differential incidence rates  among the biological and adoptive relatives of the adopted, afflicted individual), we have two types of twin studies–those that compare identical twins reared apart and those that compare sets of identical twins with sets of fraternal twins, as in this case.  Twins reared apart studies got a bad rap as a result of Cyril Burt’s fraudulent data purporting to show the heritability of intelligence.  Comparisons of identical and fraternal twins run up against the fact that having an identical twin is a very different experience from haing a fraternal twin.

I see two potential problems with this study.  First, we can’t rule out differences in interaction between identical twins and fraternal twins as a possible explanation.

The second problem–all genes are expressed at a cellular level in the form of different proteins.  Survey non-response, in contrast, is a specific and high order (far removed from cellular activity) that, really, is unlikely to be governed by a a few small chemical differences.  I believe that anyone making a claim about the heritability of any behavior ought to suggest a plausible cellular mechanism.  It’s also desirable to have some plausible selective pressure that would favor such a genetic predisposition.  Given that survey taking is a relatively recent (in human history) activity, I’m not sure you can make a case for any selective advantage in refusing to participate in surveys.

Maybe–and it’s a big maybe–there’s a selective advantage in some cluster of behaviors–such as cooperation–that just happens to manifest itself in propensity to take surveys.  That might be plausible.  Perhaps the authors offer that explanation in the full paper.  We’ll have to see.

Copyright 2010 by David G. Bakken.  All rights reserved.

 

Follow up to “Recommended read–The Predictioneer’s Game”

After I posted my brief review of The Predictioneer’s Game by Bruce Bueno de Mesquita, I discovered that you can explore the “game” in more detail at Bueno de Mesquita’s website, http://www.predictioneersgame.com/game.

 

Recommended read: The Predictioneer’s Game

The Preditioneer’s Game:  Using the Logic of Brazen Self-Interest to See and Shape the Future by Bruce Bueno de Mesquita makes a pretty strong case for using models to make critical decisions, whether in business or international policy.  To anyone involved in prediction science, Bueno de Mesquita’s claim of 90% accuracy (“According to a declassified CIA assessment…”) might seem an exaggeration.  But the author has two things in his favor.  He limits his efforts at prediction to a specific type of problem, and he’s predicting outcomes for which there is usually a limited set of possibilities (for example, whether or not a bank will issue a fraudulent financial report in a given year). 

 

Who will be the Ikea of market research?

I had the pleasure of participating in a lively discussion on the impact and future of “DIY” (do-it-yourself) research a few weeks ago at the recent ESOMAR Congress in Athens, Greece.  In a 90-minute “discussion space” session I shared a few thoughts about the future of the market research industry.  The other half of the program was presented by Lucy Davison of marketing consultancy Keen as Mustard and Richard Thornton of CINT.  They shared the results of some research on DIY research that they conducted among consumers of market research (i.e. “clients”).  Bottom line, many clients are favorable to DIY for a number of reasons.

For my part, I am more interested in DIY as a symptom of deep and fundamental change in the market research industry.  When I began my career in MR (on the client side at first), most research companies were vertically integrated, owning their own data collection capabilities and developing their own CATI software, for example.  This made sense when the ability to coordinate and integrate the diverse activities required for a typical research project was a competitive strength.  Perhaps you remember the days when a strategic segmentation study might have three or four phases, take six to nine months to complete, and cost $500,000 ( in 1980 dollars!).  But vertically integrated industries tend to “de-integrate” over time.  Firms may spin off or outsource some of their capabilities, creating value chain specialists who are proficient at one link in the chain.  The emergence of WATS call-centers and off-the-shelf CATI software were early steps on the march towards de-integration for the MR industry.

Technological change (especially in the form of disruptive innovation) also provides opportunity for new entrants.  Sure, some of the face-to-face interviewing companies made the transition to telephone, and many telephone interviewing companies successfully converted from paper and pencil questionnaires to CATI, but each of these shifts provided a point of entry for new players.

The large, integrated firms have managed to hang on to a substantial share of industry profits, but there are three looming threats.  The first is (so-called) “commoditization”–the downward pressure on pricing.  While some supplier side researchers complain that clients are unwilling to pay for quality, this downward pressure is the result of basic competitive dynamics:  there are many competing firms, few barriers to entry, many substitutes (e.g., transactional datamining) and not that much difference in value propositions or business models across MR firms.

The second threat is do-it-yourself research.  At the moment, DIY appeals to the least demanding and most price sensitive customers.  DIY removes the access and affordability barriers, thereby democratizing survey research.  As Lucy and Richard’s research showed, customers like the low cost, speed and convenience of DIY, and I expect many will move up the learning curve quickly.  I hope so–many of the DIY surveys I’ve seen from even big companies have been pretty ghastly. 

The last threat to the traditional MR business model comes from the sheer deluge of data generate by both commercial and non-commercial online activity.  How much could Google tell any marketer about customer preferences based just on search data, for example?

At the end of the session in Athens I offered this analogy.  Imagine that you need a bedstead.  You could go to a furniture store and choose from a selection of attractive, well-constructed and expensive bedsteads.  Or you could go to the local home improvement store, purchase some plywood and paint or stain and with a few tools (which could be borrowed or renterd) and some minimal ability, construct a perfectly serviceable platform bed–at much lower cost.  This represents the difference between the full service integrated research firms at the top of the latter and what we’ve historically thought of as do-it-yourself market research.  The gap between the two has been sustained until now by a skill barrier and limited access to better, easier to use tools.  This is the gap that Ikea filled in the home furnishing market by creating a new business model based on attractive, customer-assembled furnishings. 

Unfortunately for the incumbent research firms, this kind of business model innovation does not often come from the current players in a market.  The incumbents have too much personal investment in the current business model.  Let’s face it–most of us are in market research because we like the high-touch, intellectual problem solving that’s involved.  It’s what we’ve trained to do.  Designing something like appealing flatpack furniture that customers take home and assemble themselves just does not fit our self-image.

The smarter, easier to use tools are here.  Who will be the first to package them into a new way to deliver market research?

Copyright 2010 by David G. Bakken.  All rights reserved.