Jonah Lehrer, in the December 13, 2010 issue of the New Yorker, authors a stunning article on the increasing ineffectiveness of therapeutic drugs. Medications tested eight to ten years ago on large populations seem to work only half as well in similar trials today. So then, many tests for medications in randomized, double blind clinical trials (considered the gold standard in science) are not repeatable—not even close! Lehrer points out the conundrum: “The test of replicability … is the foundation of modern research. Replicability is how the (scientific) community enforces itself.”

Lehrer mentions this phenomenon isn’t simply relegated to pharmaceuticals. In fact, in vitamin and cardiac stent tests, what was once “known” and proven is now mostly without validation if tested again. “It’s as if our facts were losing their truth,” Lehrer says. “Claims that have been enshrined in textbooks are suddenly unprovable.”

Observation, testing, experimentation, and replication are foundations of scientific process. The issue in a nutshell is explained by Lehrer: “If replication is what separates the rigor of science from the squishiness of pseudo-science, where do we put all these rigorously validated findings that can no longer be proved?”

Some may argue that a simple case of changing populations, or even improvements in psychology account for changes in testing results. Or even that perhaps there was miscalculation, statistical fluke, or other anomaly in previous trials. These are entirely possible explanations, but it is unlikely that hundreds, if not thousands of studies, would be facing the same challenges today. And even so, there might be some statistical difference, but variances of 30-50% are common in repeated experiments, especially those conducted over a period of years. There must be something to this drastic decline.

The “decline effect” is keeping scientists across the globe quiet about whether their experiments can be replicated. One scientist was told by his mentor that his “real mistake was trying to replicate his work” and that he would only set himself up for “disappointment.”

Adding insult to injury, it appears that scientific journals don’t want to publish findings regarding the decline effect. Biologist Leigh Simmons from the University of Western Australia tried to submit results to various publications detailing his difficulty in duplicating his experiments, but “the journals only wanted confirming data,” he says. And that’s ultimately a shame because there is much to learn not just from success, but failure.

This then, may explain why much scientific “evidence” is often contradictory. One year, we learn that coffee is good for health, and then two years later, it’s bad. We find out that hormone replacement therapy for a menopausal woman is recommended one year, only to find out the next that it never should have been recommended in the first place. “The situation is even worse when the subject is fashionable,” Lehrer says.

All told, there is bias, conflict of interest, faulty design, selective publishing, psychology progress and certainly other developments at work. However, the decline effect is a much more significant finding as it “reminds us how difficult it is to prove anything.” Because what is true today may not be true tomorrow, the best result when dealing with “proven” scientific facts is a healthy dose of skepticism and a question everything mindset.

Questions:

• If yesterday’s rigorous and validated findings can no longer be proved, what does this mean for those practicing empirical, fact based decisioning?
• Statistically speaking, the decline effect shouldn’t be happening, but it does. What other reasons explain why facts can lose their “truthiness” over time?

In May 2010’s issue of Wired Magazine, author Clive Thompson laments the poor mathematical literacy of his fellow citizens. For example, he cites people laughing at the concept of global warming as they face some of the harsher winters on record, or the extra-vocal debate on vaccines and possible links to autism. Mr. Thompson would tell us that it’s the trend lines that matter, and we too often look at the trees and miss the forest.

The problem, he says, is that “statistics is hard” and an overall understanding of this important discipline is severely lacking. He says, “If you don’t understand statistics, you don’t know what’s going on, and you can’t tell when you’re being lied to.”

Thompson is correct that statistics are difficult for most of us, and that thinking by the numbers takes training and much effort. It’s also true that one must understand statistical concepts, especially when percentages, populations, and probabilities are bandied about in business and technical press. However, broader acceptance of the power of statistics should be tempered with limitations of this mathematical science.

Before accepting any statistic, study or experiment as gospel, the following should be considered (there may be more…):

1. Assumptions: What are the assumptions underpinning the research? As seen from recent debate on CBO numbers for the U.S. health reform package, assumptions matter tremendously.
2. History: How much historical data was used in the study? What was the time scale? As seen from the 2008 financial crisis, the models used by Wall Street mavens often only took into account 10 years of data in judging the volatility and probability of failure of complex financial instruments.
3. Samples: Are the samples selected randomly? From what populations? Is there enough data for statistical significance?
4. Data Quality: The output is only going to be as good as the quality of data feeding the analysis. Garbage in, garbage out.
5. Survivorship Bias: Author Nassim Taleb points out that “losers are often not in the sample.” Does the analysis include a population of survivors and those who also failed?
6. Falsification and Omission: Yes, in an era of IPCC’s Climate Gate, one needs to ascertain if data are hidden, missing or outliers ignored.
7. Association equals causation fallacy: Correlation does not equate to causation (a common mistake made by marketing and finance executives alike).
8. Proper Application of Statistics: The effective use of statistics by insurance actuaries, scientists, and even casino managers is well-documented. However, real danger results when mathematical concepts are used to denote certainty indecision-making and divining behavior of markets.

Now, please don’t get me wrong. Statistical analysis is very important for many industries (e.g., health care, transportation, and manufacturing). Statistics, however, can give us an illusion of control in a world that’s much more complex than our models suggest. Nassim Taleb, author of the Black Swan likes to remind us that “(real) life isn’t a casino.”

Statistical analysis is definitely a powerful gadget in the business manager’s decision-making toolkit. But one needs to understand the limitations of this science.

After all, Taleb points out that many of today’s statistical models work as though we have “full knowledge of the probability of future outcomes.” And this just isn’t so, especially when it comes to fat tails, or the “ten sigma” event. Indeed, sometimes those rare events have extremely large impacts. Were he alive today, the former captain of the Titanic, E.J. Smith would wholeheartedly agree.

Questions:
• Clive Thompson calls statistics “the language of data.” How important is it for marketers to understand and apply statistical concepts?
• “Lies, damned lies, and statistics” is a phrase popularized by Mark Twain in the context of using statistics to unduly persuade, obfuscate or even swindle. Can statistics get its reputation back? If so, how?

Similarly in the marketing discipline, while practitioners often debate whether marketing is more “art than science”–a trend towards analytics is afoot.

Tradition and convention are certainly hallmarks of Major League Baseball. And for many years, the status quo reigned–especially in the processes used to construct a baseball team.

Using knowledge, intuition and experience to evaluate talent, field managers and scouts would scour high schools, practice fields and colleges looking for the missing pieces that could potentially elevate them to a championship. Gut decision making ruled–until Billy Beane and the Moneyball analytics revolution started.

An ESPN Magazine article shows how based on geographical location, Oakland was forced to compete in a smaller market with revenues far lower than teams like Boston or New York. Attempting to level the playing field, Billy Beane took a different approach to baseball resourcing. Instead of trying to sign big name players with the best batting average, Beane used statistical analysis to discover indicators that he believed would have a better correlation with offensive success.

Michael Lewis, author of Moneyball–a book on Billy Beane’s methods writes,

“By analyzing baseball statistics you could see through a lot of baseball nonsense. For instance, when baseball managers talked about scoring runs, they tended to focus on team batting average, but if you ran the analysis you could see that the number of runs a team scored bore little relation to that team’s batting average. It correlated much more exactly with a team’s on-base and slugging percentage.”

And for awhile, Moneyball worked. In the early years of Moneyball, the Oakland A’s were competitive with payrolls in the $50 million range whereas larger market teams were spending $100 million plus. It wasn’t that Oakland was choosing to pocket the $50 million annual difference–they simply didn’t have that kind of money to spend. Oakland needed a way to compete and they chose analytics.

Unfortunately for Billy Beane, his competitive advantage didn’t last very long. Other baseball teams adopted statistical analysis and General Managers like Boston’s Theo Epstein quickly combined analytical prowess with the advantage of a major revenue market to assemble a perennial powerhouse. Like it or not (and some GMs still don’t), the adoption of analytics drastically changed baseball and now the use of analytics to help build a ball club is a standard process.

Similar to the adoption of Moneyball, marketing is in the throes of an analytical revolution.

Specifically, practitioners of marketing know they need fresh and accurate data for advanced marketing functions such as better segmentation, devising more effective campaigns and offers, and creating relevant interactions with the customer across multiple touch points. This data must be clean, modeled and managed–a large undertaking that involves marketers working closely with IT.

Marketers also are realizing that some understanding of analytical applications and business intelligence know-how is necessary to help analyze and translate data into actionable information that can be used to create better customer experiences. Hundreds of case studies in business publications and books have emerged over the past five to seven years as a testimony to these trends.

Analytics helped a small market team like the Oakland A’s compete with clubs that had much larger budgets. Indeed, Oakland enjoyed a period of success before larger teams “caught on” to Beane’s analytical approach.
In the same vein, the window of opportunity for marketers to adopt business analytics–before their competitors–is closing rapidly.

• With the early success of Moneyball, Billy Beane parlayed himself an ownership stake in the Oakland A’s. For marketers, how valuable will analytical skills be in the near future?
• Are you competing with companies that have much larger budgets and personnel resources? If so, what strategies are you using to win?
• Critics of Moneyball say that one cannot run a major league baseball team with a computer. Going forward–in marketing–will knowledge and intuition win out over analytics?

Are you prepared for the “Moneyball-itzation” of marketing?

Almost every consultant hates the phrase “paradigm shift.” And in effect, because of the recent global financial crisis, it is easy to see how consumer and business sentiments have changed quite radically. At least for now, the days of freewheeling risk taking, unabashed materialism and wanton spending have been replaced with frugality, caution and spending cutbacks.

PIMCO bond king Bill Gross, would agree that “thrift” is a new mainstay. In an Atlantic article, Mr. Gross suggests that as a result of the global financial crisis there’s something different regarding investor outlook:

“Risk taking went over the edge. We are inventing something new. We’re very afraid. We know from the Depression that people who lived through it didn’t change their mentality for the rest of their lives. They were sewing their socks. My sense is that it will take 10-20 years to find that kind of risk taking in people again.”

A recent survey conducted by Money Magazine validates Mr. Gross’ positions. Polling over 1,200 Americans, the survey discovered:

* 54% report they are worse off now than a year ago
* 89% say they’ve changed how they manage money
* The top three new habits are: eating at home more often, looking for discounts and cutting back on luxury purchases
* New attitudes are emerging with 88% surveyed saying they will be more frugal, 81% playing it safer with investments and 74% ignoring advice from Wall Street
* Men seem more pessimistic about the economy than women
* 73% said in the future they will play it safer with money and focus less on materialistic gain

To be sure, the results of the survey–at least for Americans–present a new prototypical consumer who is less trusting, a bit more conscious of his or her finances, and one that is getting “back to basics.”

Whether we are permanently in a new paradigm–or not–these statistics paint a new reality that marketers must take into account. Most companies have accepted this new reality and are baking marketing strategies accordingly. But many company executives anxiously sit on the sideline, hoping that market conditions get “back to normal” so they can raise prices, increase capacity and worry less about operational efficiencies.

In statistics, reversion to the mean indicates there are driving forces towards the average, and that outliers will eventually join the “normal”. Perhaps however these changes in consumer and business sentiments are permanent–and in effect, the mean has moved.

Questions:
* Longer term (in the next four years) will you be better off than today? What’s your outlook–optimistic or pessimistic?
* Target commercials show your home patio deck as the new vacation spot, a “Slip and Slide” as the new water park, and playing with a Wii as the new dance club. What do you think of these commercials? Does Target have their messaging right?
* Do new attitudes of frugality and safety have staying power? Once the recession ends–and it will–is it back to the past, or is this the new reality for consumers and businesses?
Related post: Advice on Altering Your Message, Product, Brand

Forecasting sales and inventory levels is probably one of the most difficult jobs of a product and/or supply chain manager as companies need to marry demand signals with supply. Adding more complexity to the mix is global supply chains that span weeks, multiple countries and sometimes oceans. Lots of hand-offs, tons of data to track, and lots of points for things to go wrong.

For many product managers (and the marketing/brand managers that support them) inventory management is a critical task. By not carrying enough inventory, companies can not only lose out on sales but also suffer reputation damage by not meeting customer needs.

Nonetheless, with companies hoarding cash–it seems the last thing companies need is to be stuck with unsold finished goods or piecemeal parts.

Apple’s Chief Operating Officer Tim Cook agrees. In a recent Fortune article, Cook says inventory is “fundamentally evil.” And Cook should know, as he’s in the very fickle consumer electronics business. “You kind of want to manage it like you are in the dairy business,” he says. “If it gets past its fresh date you have a problem.”

Here’s the rub, however. Forces of globalization and tight coupling are magnifying the complexity, impact and frequency of events. Once steady suppliers are going bankrupt, some suppliers cannot get loans in the credit crunch, and disruptions in the supply chain are becoming more commonplace. Your product launch date doesn’t matter much if your suppliers cannot deliver.

But can’t analytical modeling save us? After all, most companies are using advanced planning applications to predict future trends and behaviors, right?

While statistical forecasting techniques can help extrapolate future trends, these methods rely on building models based on historical data. And some executives say in volatile times, historical data can no longer be trusted to accurately model and predict the future.

So what’s the solution? Should we build more redundancy into our supply chains to better manage the risk of suppliers, or stay the course with the trend towards information management and just-in time supply chains that are well optimized and thin?

Better communication is a potential answer says Camille Schuster, President of Global Collaborations. What is needed, she says is, “Proactive contact with suppliers on a regular basis to determine how supplies are doing, what issues are coming up, whether any shortages are foreseen, whether there is any softness in any product area, what changes and/or rumors are floating about.”

For many companies, effective inventory management is a critical component of financial health. With “cash” at a premium in this global financial pandemic, inventory decisions can literally make or break your company.

When it comes to inventory, what level of risk are you comfortable with?

Questions for DailyFix readers:

* Is a little inventory cushion warranted as risks (environmental, political, criminal, financial, reputation, terrorism etc) seem to be increasing in intensity, complexity and frequency?
* In volatile times, should forecasting and inventory management be more focused on “gut” decision making rather than mathematical models?
* Stockouts leave “money on the table” and ultimately reduce customer satisfaction. What is your marketing advice to supply chain, operations and/or engineering executives in these volatile times? Hedge their bets with a little more inventory, or continue to operate “thin”?
* Is inventory still evil? Should it be avoided at all costs?

Like it or not, important marketing decisions regarding forecasting, budgeting, hiring and resource allocation must be made for the coming year and beyond.

Marketing executives who sit on the sidelines and watch/wait could be missing some valuable opportunities to stake a claim in new markets, build market share, or capitalize on competitor weakness. Then again, sitting on the sidelines may be the smarter approach.

A common form of decision making is using probability to determine potential outcomes. In a casino–with games of chance–it’s pretty easy to figure out the “odds” of beating the house. Outside of the casino, life gets a little messier. We can however, use statistical analysis based on historical data to help us divine the probability of certain events happening (assuming a normal distribution and independence).

In an Edge essay, Dr. Nicholas Nassim Taleb tells us, “Statistical and applied probabilistic knowledge is the core of knowledge. Statistics is what tells you if something is true, false or merely anecdotal. It is the logic of science, and the instrument of risk taking.”

“You cannot be a modern intellectual and not think probabilistically,” Dr. Taleb declares. But he also counsels us that there are many instances where “statistics don’t work–where stats are unreliable, where your knowledge is no longer valid.” And judging from the volatility of events in 2008, we just might be in new and uncharted territory where the usual tools and methods just plain don’t work.

If you believe 2008 has ushered in a new paradigm where the old rules no longer apply, one is essentially left with two choices: Pattern recognition of a different kind–(intuition) and thereby making decisions “by the gut”, or inaction–doing nothing, at least for now.

Is gut decisioning the best approach to plan for 2009? Gary Klein, author of “The Power of Intuition” says, “Analysis doesn’t work well in challenging situations where information is scarce, time is short, and stakes are high.”

While it is hard to argue that information is scarce, time is definitely of the essence and for many companies; the stakes (i.e. survival) have never been higher.

Lastly, there’s always inaction as a completely valid alternative. Many companies are maintaining the status quo, hoarding cash, letting the bodies pile up, and waiting for a better day before committing to major investments.

Every company is different, and every industry has their own challenges right now. But it has been said that challenge is only one side of the coin. Turn it over, and “opportunity” might be staring back at you.

Key questions:
* Given the events of 2008, are we in a new paradigm? If so, do the old rules no longer apply?
* Consumer confidence is at levels unseen since the early 1990s. Is the marketplace, (both B2C and B2B) overly pessimistic?
* In your forecasting models (or mental processes), are you “weighting the events of 2008″ more heavily than previous years?
* How closely are you monitoring daily/weekly events to determine whether to ramp up/down your spending plans for next year? What “key event” are you seeking that will be a major variable in your decision making?

Chris Anderson, editor of Wired Magazine and author of “The Long Tail” penned a provocative article in the July 2008 issue titled, “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete.”

Mr. Anderson makes the claim that in “the Petabyte Age”, it’s more important for companies (and the marketers within them) to identify and act upon correlation first, and worry about context later. For instance he writes, “Google’s founding philosophy is that we don’t know why this page is better than that one: if the statistics of incoming links say it is, that’s good enough. No semantic or causal analysis is required.”

And dismissing many of the sciences that attempt to bring us understanding of the world around us, Mr. Anderson notes, “Who knows why people do what they do? The point is that they do it and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.”

As companies collect more data about their customers, competitors and macro/micro environments, Mr. Anderson makes the claim that our approach to science (hypothesize, model, test) is becoming obsolete.

Science looks for causation. Scientists hypothesize as to why something works/reacts/behaves like it does and then attempt to build a model to represent reality. The goal is to use the model to test and learn, thereby gaining an understanding of a particular phenomenon.

Modeling is not only confined to the realm of physicists and quants. In the field of marketing, managers often work alongside statisticians, to build models that help predict customer proclivities such as items they might like to buy, or identifying customers who might churn to a competitor. Models can be tested and refined and with enough tweaking, models generally get more accurate over time. The output of those models can be used to piece together more complete customer profiles.

However, in the Petabyte Age, Mr. Anderson claims, “There is now a better way. Petabytes allow us to say correlation is enough. We can stop looking at the models.”

Let’s suppose Mr. Anderson is on to something. Mr. Anderson makes the claim that since modeling is often a poor representation of reality and tends to over-simplify things, we’re much better off with less explanation and more action based on the identification of correlation.
In a sense, he says, in an age of massive data, we’re better off with fewer discoveries of knowledge and understanding.

I’m not sure I agree.

Peter Atkins, author of “Galileo’s Finger“, says it much better than I can: “With the rise of the computer and its ability to handle huge numerical calculations of the greatest intensity, we are seeing a shift from analysis to numerical computation. (This is dangerous because) resorting to numerical solution can distance us from understanding.”

It is true that building models is an imperfect science, and cannot in all instances be 100% accurate–it is after all just a model!

That said, models help us test our assumptions and verify our forecasts. They help us test what we think we know, with the ultimate goal of improving our decision making in real-life situations where marketing budgets and return on investment are on the line. Modeling helps us piece the world together, and look beyond the patterns towards discovering “why” things happen as they do. And models help us transform reams of raw data into intelligence thereby helping us predict outcomes with greater accuracy.

While focusing on correlation can help us make better decisions to a degree, philosophically I am concerned with marketers potentially losing our interest in truly understanding what makes our customers, prospects and partners tick. For me at least, I want to know more than “my customers do this, or they do that”; I want to know “why“!

In all fairness to Mr. Anderson, every coin has two sides. There is a fine line between too much pontification and too little action. Indeed, there are many instances where it doesn’t make sense to dig deeper in understanding–where it doesn’t matter “why”, only that a given solution produces results.

I’d like to open this discussion up to the MPDailyFix community.

* As a marketer, do you want to understand and be able to explain the causes of why customers/companies/competitors do what they do, or is observing and acting on the behaviors good enough?
* Building accurate models requires tons of data collection, the proper analytical technologies and applications and of course the know-how. It’s not easy work. Do you agree with Mr. Anderson that for most decision making opportunities in the Petabyte age, correlation is enough?
* In a complex and busy world, where a marketer’s time is at a premium, will brainstorming, learning, and piecing the world together, become passé?