Welcome to the second half of our warts-and-all guide to Econometrics. If you haven’t read Part 1 click here. Otherwise, read on.
6. What does econometric modelling do badly?
Lots of things. We do spend a lot of time telling people they don’t need a model. Econometrics is very bad at predicting what will happen if you do something radically different to what was done before. One publisher wanted a model to tell them what would happen if they changed their publication from paid to free. No chance.
Similarly:
- Is there a price barrier at £3? Econometric models use curves and although it can be done it’s not really getting to the heart of the issue of consumer readiness to pay a certain price.
- What impact will my new product have? No idea, is it any good?
- We used 10 pieces of digital copy, spending around £10k on each – which worked best? Good luck answering that with econometrics.
In short, for all its strengths, there are many things econometrics does not do well.
7. What data do we need?
You might think that the modelling is the most time consuming, detailed and difficult part. In fact, the data collection usually is. A model needs a fully completed grid of (usually) weekly data for all the variables you want to test. And at least 2 years’ worth. And it has to be consistent. So, if you changed your data supplier half way through tracking study, or you have gaps in your data, you are in trouble. You should also always start with thinking about what drivers are relevant and try to source the data rather than with the data most readily to hand.
On the plus side, once you have done this you may find it quite illuminating to be able to graph all the drivers against sales. It’s often rare that management get to see all of their data in one place. The simple, but laborious, act of unifying the data often provides valuable insights in itself.
How data collection feels to every econometrician
8. Who needs to buy in to the model?
“Congratulations on your purchase of your new Econometric model. It will provide you with many years of reliable service.”
We have our model. We have our results. Now what do we do?
The bad news is that it may already be too late if you didn’t get your colleagues to buy in at the start. Some will say that the input data is wrong, others that the analysis is flawed, still others that there are things missing from the model. Unless, of course, it says that everything they did was excellent, in which case it’s just conceivably possible that they will be eerily silent. But how likely is that? Also, there may be winners and losers – brands and activities we should spend more and less on. Whilst we may applaud in principle the worthy goal of shining the brilliant light of Truth into the dark recesses of our business, some people don’t like what that shows them.
So, you need to plan ahead. Get key people from each function involved from the start. Yes, including finance. Talk to senior stakeholders about how they need a single version of the truth. Which they do. And try to set the ground work for establishing a learning culture. Now it really does sound like the modelling part is the easy bit, doesn’t it?
In our experience this is often the part that people tend to elide over, perhaps fearing that if they must get their colleagues on-side before doing the study, it might put the kibosh on the whole thing. But colleagues are always going to need persuading, and it’s better to do it at the start than delaying it to the end when it will be a hundred times more difficult. And painful.
Buy-in critical. Fist-bump optional
9. Getting from ‘did’ to ‘do’ – optimisation
One thing we can do now is to use the model to show us what our optimum marketing plan is.
Anyone who thinks this is as easy as it sounds hasn’t, I suspect, ever done it. Optimisation is a subtle and complex process which combines the model results, activity inputs, costs and – most importantly – a whole raft of business constraints. How much are we going to let the model spend, and where? Are we trying to maximise volume or revenue or profit? (No, you can’t have all three.) Are we prepared to spend 50% more on our key brand, if it means gutting the budgets for the rest of our brands?
The good thing is that a proper optimisation process does two things. Firstly, it brings the model to life through the process of running iterations:
“Oh, that made hardly any difference.”
“No well it wasn’t going to based on what was shown in the presentation.”
“Ah… I get it now.”
Secondly, it helps the organisation think about what it is, and isn’t, prepared to allow to happen and what its goals are.
In our experience, as much of the benefit of ‘doing an optimisation’ comes from forcing ourselves to answer these questions categorically as from the actual answers themselves. It follows that you should do it properly and go through the pain of asking – and answering – these difficult questions, and resist asking your consultancy to ‘pop that in your model and tell our CEO what it says.’ That never ends well.
10. A living, breathing thing
Models thrive on testing hypotheses. As we refresh the model, plan to test different activities at an appropriate scale to see an effect.
Do this as scientifically as possible. When you first build a model, you are handed what has been historically deployed as a fait accompli, which is unlikely to have been laid down just to help the model. Activities will have been run at the same time. Or have been always on. “What would happen if we turned that off?” No idea. But on the next iteration we can test it and see. We can misalign activities, structure a plan.
We all like to think we’re scientists, including this guy who is obviously a model
Think about the decisions your company can practically make and think about how you could test to see which is better. In this way, your model can become a ‘live’ tool that assists you with future planning, rather than just measuring the past.
So, there you have it. Our essential guide or Top 10 of econometric modelling. If we’ve not managed to address your questions, please feel free to get in touch with us.
Tom Lloyd