Torture data until it talks and then challenge all that it confesses

A global client recently asked us to make them an orbuculum (or crystal ball), used for ‘scrying’ whereby images seen in it, were interpreted as meaningful information and used to make important decisions.. We’re often asked to look into the future – it’s what we do. Typically this means mining evidence from historical data to build out propensity models that suggest what is going to happen; what a customer will best respond to, how much people will pay, who will churn or renew. Now this brief required us to predict changes in behaviour across several million customers who would be offered a significant new proposition enhancement - a proposition with no comparable precedent. The model needed to show the impact on the business over a 5 year period. So you see why they wanted a crystal ball.

We had one chance to get this right. The outcome would be used to determine a go/no-go decision for investment in the development and launch of the global proposition.

 “If you torture the data long enough, it will confess”, warned Nobel Prize winning economist, Ronald H Coarse spelling out a key risk in this project. We knew that small variations in the input and any unfounded assumptions whilst analysing the decade of transactional data could result in drastically different outcomes. So, with Coarse’s mantra ringing in our ears we designed a methodology that interwove multiple inputs to produce a robust, grounded output. Here are some of those strands as a ‘how to’ list:

  1. Segmentation - Create a series of theoretical customer behaviours that would result from the introduction of the new proposition based on an understanding of the sector and of consumer motivational traits.
  2. Proxies - Identify ‘proxies’ and assumptions from a range of sources across the business for other companies / sectors and weight these proxies according to how relevant and applicable they were to this project. Apply these as qualification criteria to determine which customers fit into which group. For example, only customer who are X% of spend away from qualifying will change behaviour. Then model the revenue profiles through each of the customer groups over time, before qualification (the time when they ‘reach’) and after qualification (the time they keep the same spend or ‘dip’ behaviour).
  3. Project Management - Consult regularly with a tight, expert client steering group that buy into or reject the assumptions at every step. They are, after all, the ones who know their business the best. A consultant should present informed opinions and options, challenge established norms and empower clients to make decisions with robust thinking. And so, we learned that some of the outcomes from distant but related project / initiatives provide key inputs to create guidance on assumptions – for example “we have never seen behavioural change of more than X% for that customer group” was a powerful validator to ensure we didn’t use Y% as the assumption.
  4. Research - Commission new customer research that asks specific behavioural questions based on things that customer has actually done. This provides additional inputs to behavioural change. We don’t believe that asking people about how they might respond to something intangible gives a reliable answer.
  5. Network - Leverage your network of senior contacts in parallel industries and sectors who have relevant experience. This is powerful – it provides breadth, credibility and another input
  6. Common Sense - Sprinkle intuition and ‘hunch’ across all of the data analysis – a rabbit hole that produced an outcome that doesn’t feel right, usually isn’t. Based on a consolidation of all these inputs, re-crunch the data to run sensitivity checks that illustrate how variations in assumptions and proxies affect outcomes. With Coarse’s ‘tortured data’ warning in mind, this was a critical step, as relatively small changes could change the business case from a ‘no-brainer’ to a ‘no-way’ –

The whole process carried risk based on the potential size of the resulting investment decision, combined with the lack of hard factual input. We kept the data analytics anchored to the proxies and anecdotes from other sectors and from the expert ‘intuition’ from across the client business.

There is never a 100% right answer to a question like this – the best conclusion is one that all of the stakeholders are comfortable with. Comfort comes from collaborative development of the approach, co-validation of assumptions and sense checking with the ‘intuition’.

Data and intuition working together. A ‘crystal ball’. Nice.

About Iris Concise

At iris Concise we work with across a broad range of sectors to turn business and customer insights into powerful and practical strategies that empower people to take action.

We help clients answer questions such as:

-        How do we get more value from our marketing budget?

-        We need to get closer to our customers so we can evolve our proposition?

-        Is offering X working as hard as it could, how can we improve it?

-       We want to launch a new service offering. Should we?

  • news tags

MORE NEWS, IDEAS & OPINION