My friend and mentor Bob Klein introduced me to the marketing “retrospectograph” some years ago.  As I recall, we had been reviewing the analysis of data from a consumer study that was part of a simulated test market for a new product.  Something about a particular result was counterintuitive, and I was developing–out loud–a plausible after the fact cause for this finding when Bob said something like, “Well, if you turn on the marketing retrospectograph, you can explain anything.”

This was an important lesson and reminder of the potential fallacy of post hoc explanation (from post hoc ergo propter hoc–“after this, therefore because of this”).  All too often in the course of creating customer knowledge, I’ve seen marketers fashion a theory of the customer from a few facts–the results of a single survey or a few focus groups, for example–and that theory of the customer becomes the basis for action without any further testing.

Nassim Nicholas Talib explores this phenomenon in his book, The Black Swan, using the term “retrospective determinism.”   Taleb describes this mechanism–working backwards to a satisficing causal explanation–in his discussion of “silent evidence,” by which he means all the occurrences of something that are not observed.  Who is to say that the next set of survey results will provide the same facts on which our now codified theory of the customer is based?  His main point is that failing to take into account the possibility of different outcomes or input and outcome pairings not yet observed makes things seem much more deterministic (as opposed to random) than they really are.

Once a theory of the customer emerges, confirmation bias kicks in.  Our natural tendency is to give more attention and weight to events or observations that confirm our causal models than to those observations that are disconfirming.

The smaller the set of observations on which the theory is based, relative to the total number of possible observations, the more likely Taleb’s “silent evidence” is hiding disconfirming, if inconvenient, facts.  Of course, in the hypothetical situation where you have a perfectly identified population of these observations from which you can draw a large enough random sample, it’s possible to make inferences about the relationship between inputs and outcomes without worrying too much about “silent evidence” since we can make explicit statements about the likelihood of unobserved events.  However, this favorable circumstance rarely, if ever, occurs in the world of marketing research and customer knowledge creation.

There’s a lot of evidence coming from cognitive science indicating that we are more or less hard-wired to make erroneous causal inferences.  As one manifestation of this tendency, most fictional crime-solving, from Sherlock Holmes to CSI, relies on Taleb’s mechanism of retrospective determination, coupled with a strong dose of confirmation bias.  Of course, dramatic tension in these stories often comes from the discovery of disconfirming silent evidence that is too compelling to ignore.

The scientific method is supposed to protect us from these biases.  The main problem in marketing is that we do not–or cannot–take the time to execute the sort of programmatic research that treats each plausible explanation or theory of the customer as one of many possible theories that we must continually try to disconfirm.  Unfortunately, it’s all to easy to turn on the marketing “retrospectograph” and convince ourselves that we understand our customers.

Copyright 2009 by David G. Bakken.  All rights reserved.