A big data architect's guide to building real-time contextual marketing systems

Lead Architect & Head of Products, Knowesis Pte Ltd

Let us start by synchronizing our understanding of this word context first. Context here refers to the best representation of all the knowledge about an entity of interest. In a contextual marketing scenario, for example, the entity of interest is the customer and the context is the most accurate representation of the customer’s needs. Now to the phrase real time. There is no argument about the effectiveness of being real time in any business scenario that involves reaching out to customers. Say, if you are a shooter, would you pull the trigger five days after the target has left and hope for a miracle?

The challenge does not end with being contextual and shooting in real time. The important question is, would you choose a silver bullet? Or are you would just spray as many bullets as possible? What would be the right strategy?

Well, the right answer is: it depends. I know, a wise statement if you’ve ever heard one. Let me explain this using the marketing operations in the telecommunication industry. Traditionally, when an operator decides to market a product, he would spend a lot of time and effort in mining through the data warehouse to get the right target segment. This exercise is extremely expensive when you take into consideration the inevitable mistakes that will be made. Therefore, a lot of time needs to be spent to get it right. This is the silver bullet approach.

But the trouble is, the market is too dynamic to predict; subscribers’ preferences change rapidly and the competition offers more accelerator to this mix, so it is very hard to get the right segment. In the end, the campaign does not produce the anticipated returns and, often, it is even challenging to measure the effectiveness of the campaigns, but that is altogether a different topic. Therefore the only option left for the operator is to quickly tweak the segmentation criteria (using a very agile tool set), hit the market and experiment on what really works. This typically has to undergo multiple iterations and in an agile manner. This is the spray and pray approach: sloppily discharge bullets until a sufficient number of targets are hit.

That is not the end of it all. With that automatic weapon in the armory, the operator goes happily spraying until he realizes that his customers have quickly developed serious campaign fatigue. Everything goes on a downward spiral from that point onwards: now the operator no longer has the luxury to continue experimenting and he now needs to resort to the silver bullets. As you can see, this is a classic Hegelian dialectic. The operator slides between the thesis and antithesis of silver bullet versus the spray gun. The synthesis would be what is right for the market at any point in time.

If you are a tool-builder, you need to make sure your solution can cater to both the silver bullet and the automatic weapon approaches. Having a tool that rigidly sticks to only one of the approaches will bring a big loss and frustration to customers. In Knowesis, we have built such a tool (Sift) using fantastic components including IBM Infosphere Streams and IBM Unica Interact. Stay tuned to my upcoming posts, where I will attempt to narrate my experience in building such a system, the component choices we had to make and the architectural decisions involved.