Data scientists need psychological insights to tune customer analytics

Big Data Evangelist, IBM

Marketing comes down to demand generation, and that requires at least a rough idea of what makes people tick. In other words, it demands at least a passing familiarity with human psychology (though most of the pop psychology that pervades marketing is laughably shallow and, more often than not, largely unsupported by solid academic research).

But psychology has become unfashionable in marketing in the era of big data, according to a recent article from the University of Pennsylvania. The author claims marketing professionals are more likely than ever to leverage big data and statistical models to power real-world experiments and A/B tests involving different products and messages. It is far less likely than ever for market researchers to employ upfront focus groups and other traditional tools for gaining insight into customers' heads and hearts.

It's important to note that most marketing analytics are built on statistical models of phenomena that are fundamentally psychological, including experience, awareness, sentiment, propensity, loyalty, intent, influence, cognition, interests, customer journeys, relationships, natural language processing and decision making. Measuring, optimizing and shaping any of these require at least a passing familiarity with the social sciences.

In the cited article, marketing professors say psychological domain expertise is needed to keep spurious big data-driven predictive insights from running campaigns off the rails.

According to Peter Fader, co-director of the Wharton Customer Analytics Initiative at the University of Pennsylvania, what happens more frequently is that marketers and the data scientists who work with them “say ‘Let’s just try stuff and see what works.’ It’s a widely held belief that it has become much easier to test things in market [i.e., by saying,] ‘Let’s put out our concepts and see what gets clicked on the most.’ That can help you determine a winner, but it doesn’t help you design what would have been the best. By doing careful research and determining the underlying drivers that cause people to click, we can develop better products and services.”

The initiative's other co-director, Eric T. Bradlow, emphasizes that psychology is not necessarily an alternative to big data and analytics, but a necessary complement and reality check. “Can you possibly predict what people are going to do?" he asks. "Yes, you can. However, the science of psychology—why people are doing what they are doing—in traditional marketing research provides a great complement to what can be measured.”

Of course, psychology, though an important counterweight to statistical models, is far from infallible. Psychological analyses are notoriously inconsistent, subjective biases often run amok and causal analyses of psychological drivers of customer behavior often become exercises in 20-20 hindsight. They tend to grow shaky when the analyst attempts to use psychology to predict future behaviors under various scenarios. Psychological analyses are likely to depend on myriad factors, such as broader social influences or demographic trends that the analysts overlooked, underweighted or distorted in their limited theoretical frameworks.

That's why I had to cough when the article quotes Fader waxing nostalgic about some supposed long-ago heyday for psychology in the marketing arts: “It is a sad state of affairs," he moans. "Market researchers were the ones holding up the light so we could see an otherwise dark world. They used to be leading the way, from the 1950s, 1960s and 1970s on, at giving us a window into customer preference. For resulting managerial decisions, it really was a vital function for so many companies. Today it is in the back seat, if companies are doing it at all.”

That's actually not the case in today's marketing world. It would be overstating the situation to say that today's marketing data scientists have turned a deaf ear to psychology or any other social science.

In fact, the opposite is increasingly true: psychology-oriented specialties are a part the core repertoire for some of today's leading data scientists. Several months ago I blogged about the fact that the data science world is recruiting more people from the humanities for the super-hot specialty of customer experience modeling. In addition, I noted the growth of other specialties with an emphasis on psychology, sociology or both.

For example, the data visualizer is adept at finding innovative ways to present data-driven insights within instinctual graphics, the contextual analyst focuses on interpreting and describing quantitative insights within the larger business narrative context and the neuro-analyst takes a cognition orientation, focusing on how humans can best interact with data-driven analytics to impel comprehension, exploration and insights.

Are focus groups dead? Far from it, though it seems that the traditional ethnographic approach (for example: observing a group of people in a room through a two-way mirror) is shifting toward a more completely online focus, often using Skype or other desktop videoconferencing tools. Video, image, speech and gesture analytics tools will probably become more important tools for data scientists who try to parse these streams for psychological insights. For more information, see my blog post on how these technologies are being deployed in retail point-of-sale environments.

Actually, what these technologies do is bring 3-dimensionality to the notion of a "720-degree customer view," which is fundamental to experience optimization. Psychology is at the very heart of the 720-degree view, which is all about building an ever-deeper portrait of the internal journey of experiences, propensities, sentiments and attitudes that drive customers.

To do 720-degree view correctly, you need a correlated wealth of profile, attitudinal, social, transactional, survey and other data, of course. You also need a nuanced set of validated models that do justice to each customer's experience, desires and journeys through life. A strong grounding in the behavioral and social sciences will help you model how your customers might react and act within various real-world scenarios. This is important if you want to understand customers thoroughly so you can target them with offers and provide them with experiences that strengthen the relationship.

By doing machine-learning analytics on video, speech and other 3-D feeds from live shopping settings, these new technologies enable the customer's inner journey to be fleshed out with data on up-close interactions. Potentially, these new tools could blow the concept of focus groups wide open. They could enable data scientists to build powerful inner journey models from continuous video streamed from retail channels. Data scientists would be able to refine their journey models with fresh feeds on every person (customer or otherwise) who walks into a shopping environment. And to the extent that a retailer incorporates video, speech and image analytics in their e-commerce initiatives, their data scientists could gather even more useful customer data over online channels.

Psychological insights could be brought to bear at each step in a live retail environment. Regardless of the channels from which the retailer acquires the 3-D intelligence, it could be used to drive one-to-one predictive personalization of interactions throughout the buying cycle. To the shopper browsing your aisles, the kinds of recommendations being presented in real-time on in-store video kiosks would feel like there's an actual human attending to your needs.

You can realize that vision if you bring real psychological expertise into your data science centers of excellence.