The global economy may be a wreck, but data volumes keep advancing. In fact, there is so much information competing for our limited attention, companies are increasingly turning to compute power and algorithms to make sense of the madness.

The human brain has its own methods for dealing with information overload. For example, think about millions of daily input the human eye receives and how it transmits and coordinates information with our brain. A task as simple as stepping a shallow flight of stairs takes incredible information processing. Of course, not all received data points are relevant to the task of walking a stairwell, and thus the brain must decide which data to process and which to ignore. And with our visual systems bombarded with sensory input from the time we wake until we sleep, it’s amazing the brain can do it all.

But the brain can’t do it all—especially not with the onslaught of data and information exploding at exponential rates. We need what author Rick Bookstaber calls “artificial filters,” computers and algorithms to help sort through mountains of data and present the best options. These algorithms are programmed with decision logic to find needles in haystacks, ultimately presenting us with more relevant choices in an ocean of data abundance.

Algorithms are at work all around us. Google’s PageRank presents us relevant results—in real time—captured from web server farms across the globe. Match.com sorts through millions of profiles, seeking compatible profiles for subscribers. And Facebook shows us friends we should “like.”

But algorithmic programming can go too far. As humans are more and more inundated with information, there’s a danger in turning over too much “pre-cognitive” work to algorithms. When we have computers sort friends we would “like”, pick the most relevant advertisements or best travel deals, and choose ideal dating partners for us, there’s a danger in missing the completely unexpected discovery, or the most unlikely correlation of negative one. And even as algorithms “watch” and process our online behavior and learn what makes us tick, there’s still a high possibility that results presented will be far and away from what we might consider “the best choice.”

With a data flood approaching, there’s a temptation to let algorithms do more and more of our pre-processing cognitive work. And if we continue to let algorithms “sort and choose” for us – we should be extremely careful to understand who’s designing these algorithms and how they decide. Perhaps it’s cynical to suggest otherwise, but in regards to algorithms we should always ask ourselves, are we really getting the best choice, or getting the choice that someone or some company has ultimately designed for us?

Question:
Rick Bookstaber makes the case that personalized filters may ultimately reduce human freedom. He says, “If filtering is part of thinking, then taking over the filtering also takes over how we think.” Are there dangers in too much personalized filtering?

An algorithm is simply a step-by-step approach for solving a problem. Think of an algorithm like a formula; it can be complex, or relatively simple in design. Now add compute power from today’s super fast computers coupled with the know-how to design, build, and maintain these formulae and you have a winning combination! Companies across the globe use algorithms to make recommendations (think: If you like this product, you’ll probably also like this), choose optimum delivery routes for packages, and even route calls to agents that can best diagnose a particular problem.

How can an algorithm confer competitive advantage? Depending on the type of business you’re in, it’s easy to see how algorithms can reduce all available choices into the very best options. Take for instance, Google. In the February 22, 2010 issue of Wired Magazine writer Stephen Levy points out, “For years, (Google) has used its mysterious, seemingly omniscient algorithm to, as its mission statement puts it, “organize the world’s information.” Google’s algorithm is constantly tweaked, honed, tested, and improved to better interpret searchers’ requests, no matter how awkward or misspelled, says Levy. And this competitive advantage in its search algorithm has (so far) confirmed a 65% share of the search market for Google.

In a sea of data, algorithms can also help reduce choice overload. Online dating sites often use proprietary algorithms to divine appropriate partner matches based on user inputs such as preferences for race, religion, eye or hair color, and more. eHarmony’s algorithm for example, helps select potential partners based on a 258 question personality test. eHarmony’s algorithm takes too much choice (sea of available singles) and distills/simplifies millions of choices into much more manageable options.

And while companies like eHarmony rely on data input by a user, a new recommendation engine called Wings mines your social media “bread crumbs” left on various websites (including Facebook, Netflix, Twitter, Foursquare and others) to feed into its algorithm to pick a suitable dating partner. A MIT Technology review article on Wings says, “The idea is that the computer’s analysis of your behavior provides a richer analysis than you’d say about yourself.”

More data has been created in past three years than in past 40,000 years, says Teradata CTO Stephen Brobst. Indeed, today and into the near future, companies that can sort through, analyze and utilize this rich trove of data treasure faster (in some cases with blinding speed) than competitors will dominate over those enterprises slow to comprehend this critical transition.

Related: “Social Network Analysis: Hype or Help?” and “The Zero Latency Future is Now”

Questions:

• Are recommendation engines becoming more or less reliable? Think of a website you often use that uses recommendation algorithms. How “close to home” are its choices for you?
• Do you think a computer can discern your tastes in romance better than you can?

Land line and mobile operators–by nature–capture a significant amount of transactional data (call detail records, web visits/transactions, and GPS data just to name a few). To extract value from this data, companies employ data mining techniques.

Data mining is the process of discovering hidden patterns from large data sets. Using sophisticated algorithms, companies in all industries are “mining” mammoth data warehouses to identify useful information (relationships, rules, and sequences) that can help them customize and personalize offers, and optimize business processes.

However, according to a Technology Review article titled “Reality Mining,” MIT professor Sandy Pentland thinks mobile operators are poised to take data mining to a whole new level. Dubbed, “reality mining”, Dr. Pentland thinks mobile operators have an opportunity to record more than just where you’ve been or who you have recently called.

The article notes that Pentland “would like to see phones collect even more information about their users, recording everything from their physical activity to their conversational cadences.”

In the near future, Dr. Pentland suggests the following can be “learned” from studying data captured from your cell phone:

* Your cadence may reflect your state of mind that day …. are you happy, sad, depressed?
* Through capture of location based data, it will be possible to “predict” places you are more likely to visit
* Your calling patterns can help map your social network
* Your physical activity (or lack thereof) could be monitored by health professionals via your mobile device. Pedometer anyone?

The article continues, “Within the next few years, Pentland predicts, reality mining will become more common, thanks in part to the proliferation and increasing sophistication of cell phones. Many handheld devices now have the processing power of low-end desktop computers, and they can also collect more varied data, thanks to devices such as GPS chips that track location. And researchers such as Pentland are getting better at making sense of all that information.”

Of course, there are strong privacy considerations with the advent of these services. How does one opt in/or opt out? What information is shared and how much is shared and with whom?

Arguably, on the marketing side, more detailed information (including location based data), collected and analyzed by your wireless carrier could help them tailor and personalize specific offers–raising marketing effectiveness. And mapping your social network could help you share information more easily (think: favorite five plans–on steroids).
But there is a fine line between “benefit” and “big brother”.

• What do you think of the concept of “reality mining”?
• Would you be willing to opt-in to potential benefits of reality mining?
• Where would you “draw the line”?