Mathematical modeling is helping companies across the globe forecast more accurately, optimize supply chains, assess risk, and keep customers from churning to competitors. However, recent market conditions (i.e. credit crisis) have shown that while models can provide an “air of certainty”, solely relying on them for complex decision making can be very costly. Under what circumstances can mathematical models be trusted?
Data, by itself is of little value. The real value lies in the capturing, cleansing, management and analysis of data thereby making it more useful for decision making.
One of the ways companies analyze data is to build or “program” models to simulate, test, learn and predict outcomes. Indeed, models built upon statistical techniques are helping companies identify fraud, predict next best offers, determine customer churn, and assess credit risk among other valuable applications.
But modeling isn’t a panacea, and a recent New York Times article, “How Wall Street Lied to Its Computers“, shows us how companies can get it wrong.
Writer Saul Hansell notes that Wall Street traders have long had very sophisticated models on market behavior–devised by quantitative analysts–that were supposed to help them hedge their positions and allow them to essentially manage their risks (or enable them to take bigger risks).
However a key challenge emerged when, “The people who ran the financial firms chose to program their risk management systems with overly optimistic assumptions and feed them oversimplified data.”
Even worse, many of the products (read: derivatives and derivatives of derivatives) weren’t understood by the creators of the products and thus it was near impossible to accurately assess the risk of these products with a mathematical model.
Modeling isn’t just for risk management, and can be a very valuable tool for companies assessing future scenarios, determining cause and effect, and allocating scarce resources. However the New York Times article highlights a great case study of pitfalls and key challenges when attempting to model a system, phenomenon or behavior.
First, understand that a model will only be as good as your assumptions. For example, in many risk management systems, models are designed to assume rational decision makers, a stable and relatively volatile-free marketplace, and that outliers generally have a limited effect on the entire population. Anyone who’s invested in a 401K and tracked their stock portfolio recently knows the futility of these assumptions (Nassim Nicholas Taleb explains why here).
Second, mathematical modeling is only as good as your data. Hansell’s article points out that it was in the best interest of traders to ensure the models didn’t warn them of impending danger, so they took efforts to smooth the data and manipulate the amount of historical data their risk management systems could analyze so as to take more aggressive trading positions.
Third, modeling is only as good as the design and designer of the model. “There was a willful designing of the systems to measure the risks in a certain way that would not necessarily pick up all the right risks,” says Gregg Berman of software company Risk Metrics. The design of an model should be checked for accuracy–not only of the accuracy statistical concepts used, but also that the model is not “tweaked” to produce desired results.
Last point: a model might be based on fair and accurate assumptions, sourcing clean and legitimate data, and designed properly–however it is of little use of politics stands in the way of recognizing and acting upon the output. All the analytical systems in the world are of little use if corporate politics dictate an outcome that is different than what the model prescribes.
No mathematical model is perfect–a model is just that–a model and not a silver bullet. Also such models are support tools that should be combined with good judgment, experience, and the input of others to effectively drive decisions.
That said, the time, energy, and investment dollars spent on mathematical modeling is close to worthless when poor assumptions, faulty/dirty data, bad design and corporate politics get in the way of good decision making.
Questions:
* Mathematical models are used by companies for customer segmentation, risk management, propensity to buy, loyalty management etc. Are you using models to help you make better decisions? If so, how?
* Do you think we often try to model things that are too complex–things that can’t be modeled effectively? What might be the ramifications when we get it wrong?
* In the business world, do you think the use of mathematics sometimes overrides “common sense”?
* Mathematical modeling–done right–can be a powerful business tool. How can we teach future generations of business leaders to use these tools ethically?
Tags: analysis, analytics, decision making, mathematical modeling, models, Nassim Taleb, risk management
“Do you think we often try to model things that are too complex–things that can’t be modeled effectively? What might be the ramifications when we get it wrong?”
Um… climate change, anyone?
Great post Paul. It’s a cliche but “garbage in, garbage out.” In modeling we have to be smart and honest about the data we enter. If anything, we should err on the side of caution and be concervative minded regarding both data in and data out.
I am a huge proponent of multi-channel marketing using inbound channgels to collect, analyze and predict. But the data cannot be a standalone decision maker. We also need to add the human element by understanding our customer’s emotional wants and needs by spending as much face time as possible listening, observing and conversing.
Ike, thanks for taking the time to comment. Accurate weather modeling is very difficult as there are so many variables that affect the weather (even many chaos theorists doubt it can be done properly). That said, with Moore’s law and compute power doubling every two years, maybe we’ll get better at it over time?
Lewis, solid advice as usual. One of the key challenges with modeling as you pointed out is the data used to feed the model. And while GIGO is a key factor, I’m discovering the assumptions upon which the model is based is probably the number #1 factor in whether mathematical modeling can be trusted…
It’s a cliche but “garbage in, garbage out.” I agree with Lewis, and in fact, I’ve stated this on another post of yours, Paul. The other thing I would say is this: to avoid even the appearance of faulty, politically motivated design flaws or omissions, it’s probably a good idea to get input and a sign-off from a third party for a new modeling program. That party should be above ethical reproach and not tied to the business that is developing new mathematical modeling programs in the least. Since these programs have the potential to impact the customer in a negative if not disastrous way, the damage that can be done should be a serious concern to executive management. Transparency and honesty would go far to gaining customer trust. Too much of our trust has been violated lately. There is plenty of opportunity for companies that demonstrate they can be trusted to fare well, slow economy or not. The cream always rises to the top, doesn’t it?
Claire, what a novel idea about gaining sign-off from neutral third parties. This might not be exactly what you had in mind, but there is probably much to be gained from having another department or division with little stake in the matter take a look at the model, review assumptions, check the data and in essence provide another opinion on the validity of the model’s output.
However, this approach would require humility and an openness to being challenged. “The way we’ve always done it” would also be up for review. As business executives, are we ready for such things, and isn’t easier to just go with the status quo?
Paul,
Data and analysis create a wonderful B&W picture of the situation. To add some color, have someone with a touch of gray in their hair look at it:-)
I highly recommend Malcolm Gladwell’s Blink to round out the subject.
Paul, I’m quite versed in chaos/complexity, and don’t think we’ll ever get the long-range weather forecasting nailed down.
That said, climate is a different issue. There are many long-term swings and factors that we simply have taken for granted or outright ignore in the modeling. And I mention it because I see it as a perfect analog to your example about market predictions. Climatologists are just as prone to bias their models by seeding for results as money-market managers.
The difference is that the money-market moguls might cost us a couple of trillion dollars in the short-term; reshaping a global economy to meet a future climate that might be forecast fiction might cost us hundreds of trillions (for no discerning difference.)
Ron, I love what Thomas Davenport says about intuition, that its just “built in analytics”.
Also, I agree with you that we need more “grey-hairs” in the workplace (I have some greys myself!) Perhaps that will be less of an issue as unfortunately many have had to put off retirement due to the declining values in the equities market…
Ike, thanks for making a terrific point regarding models used for climate change. About six months ago, I was at Scripps Oceanic Institute in my hometown of San Diego. They have very interesting pictorial showing how their climate model predicts absolute disaster in the coming years. It’s pretty alarmist thinking.
What’s interesting about the representation is that there is no discussion of the actual model used to create the output, and that people take the model as gospel since it was created by scientific minds.
My point is not to debate the effects, or purported effects of climate change. Simply that in business and life we tend to trust models and the output of models without challenging the assumptions, data, output, and “motivations to build” behind some of these models.
There is much value in a “questioning mind”…
Another topic near and dear to my heart, Paul.
We’ve helped companies for a long time use modeling to make better marketing decisions and we’ve seen them used for both good and evil.
On the good side, companies use them to make adjustments to a marketing plan before a real world new product launch, determine the most effective and efficient media plan given a limited budget, and more. Unfortunately, like any marketing research tool, there is the potential for abuse. For instance, marketers sometimes use models to “justify” their decision rather than “identify” the right one. Like the Wall Street traders, they feed in assumptions to ensure they get their answer back. In other situations, marketers don’t adhere to the assumptions of the model in the real world, yet are shocked when the outcome isn’t as predicted. They turn around and point fingers at the model.
The more we talk about the terrific results–higher sales, stronger brands, better companies, a promotion and big fat raise–of using models as well as the thoroughly unpleasant results of abusing them, the more we’ll drive home the message to future generations that there’s a right way and a wrong way to use them.
On your question about the use of mathematics sometimes overriding common sense in the business world, this situation is definitely more the exception than the rule. We’ve found many more instances of a model giving a no-go recommendation only to have the marketer go ahead anyway because he or she “feels” it’s the right thing to do. We fight the urge to say “we told you so” when they come back a few months later and admit the model had it right.
Kevin, I appreciate that you’ve taken the time to comment on this post.
You aptly point out the challenges of “gut thinking” vs analytics in the marketing world. Even if we choose reasonable assumptions, properly program the model, and use accurate data, there are plenty of times when we ignore all that and go with our gut.
Don’t get me wrong, there are definitely times to “override” an analytical output with gut decision making. However, if intuition is winning the battle vs. analytics for “the final decision” more often than not, it begs the question, why did we spend all the time developing the model in the first place?
Thanks for adding your valuable real world experience to this post!
You know what I’m going to say, right, Paul? Before any models are constructed, two things ought to happen if those models are going to gain trust and confidence among employees and customers. First, a clearly defined strategy needs to be put in place–and articulated in print for all to see. Second, the design of these models is crucial. By testing and having a third party seal of approval (as someone else suggested) that the strategy has been met and the model delivers the kind of information that is meaningful, important and reliable, then something positive and significant will have been achieved.
Ted, thank you for taking time from your busy day to comment!
Regarding a “seal of approval”, it’s pretty hard to have third parties give a seal of approval on these models, since most are developed in-house. That might really slow down the test, learn and improve approach. I think the best we can ask for is to have another org in the company review our assumptions and validate the model or to your point, if we’re working with an outsourced or other marketing consultancy, we also may want them to review/validate the model.
On your first question, I was thinking about circumstances where mathematical models can and are used, but perhaps shouldn’t be.
The problem is that most mathematical models need to be based on past behavioral data. But behavioral data as tangible and predictive as it might seem to be, don’t always tell the whole story. I’m thinking specicifically of market segmentation.
You can segment certain markets with a mathematical model based on past shopping and usage behavior, but by excluding attitudinal measures, unsatisfied needs, future presdipositions towards certain brands or behaviors, and exposure to new product concepts and positionings, you can be neglecting a large part of what can make for a truly effective market segmentation strategy.
The fault is not in the modeling, per se, but in the data available.
So instead of restricting our evaluation of models to confidence intervals and parameter estimations, we should always take a step back and think about the breadth of our analytical goals compared to the nature of the data that could populate the model.
My point here is that models are almost by definition backwards looking and when it comes to market segmentation, we need to be looking forward too.
Steve, thank you for contributing to the discussion. As you know, models need to be “fed” data – and usually the more historical data the better. However, as you point out, even the most advanced statistical techniques can only project forward based on historical data. The assumption being of course, that the past is a reliable predictor of the future, which as we know from the events of this 2008, is a very dangerous assumption.
Thanks for reminding us that modeling based on historical data provides a view, but not necessarily the complete picture.
Excellent article. I agree many of the mathematical models today suffer from the problem of bad assumptions and bad data. I think nobody has taken them seriously enough to build realistic models with them. But I think that will change in the near future especially after the present crisis.
Ivs, thank you for taking the time to comment!
While there are myriad issues/reasons for the financial crisis of 2008, one of prominence was the usage of models built upon faulty assumptions. Case in point the Black Scholes model did not and could not predict the volatility we had in 2008, since according to many of the models based on B/S, such events couldn’t have possibly occurred in a hundred million years. We’re talking 5-22 standard deviations here!
I’m not convinced even after 2008 we’ll be able to build realistic and highly accurate models of very complex things like markets and weather. Maybe Mandelbrot’s fractals will be able to provide some insights…
That said, until we get it closer to “right”, models shouldn’t be junked. Just proceed with caution!