Comments on: Is There Something Rotten in CGM Research?

By: Recent Reads in China Digital and IWOM/Social Media , April 12, 2010 | China IWOM Blog- Making Sense of the Buzz

Wed, 14 Apr 2010 02:32:36 +0000

[...] Is There Something Rotten in CGM Research?; Marketing Profs [...]

By: David Crane

David Crane — Fri, 04 Aug 2006 08:03:35 +0000

Bill,
Thanks for all your input. It’s very interesting to hear criticisms of this research technique from an expert.
You said–
“Science, and the scientific method are the foundations of all acceptable marketing research. Without the discipline of a scientific foundation, all data, and the information gleaned from that data, represent information that is unreliable.”
Whilst CGM data may be unreliable from a scientific standpoint it is seen as highly reliable by people making purchasing decisions.
We trust our friends’ opinions more than advertising messages. Bloggers or other forum members are step-friends, we know them from the community in which they write or from what they’ve written before. Even if this is the first time we’ve come across them, other clues tell us whether they can be relied upon.
Their influence, whether as individuals or en-masse, is why they should be studied …. even if we might not have precisely the right methodology to extract exact information just yet. I imagine traditional market research didn’t start off knowing just how to phrase questions or run focus groups either.
You also said–
“Some who have had problems are very vocal about their supposedly negative experiences.”
That’s true and some very motivated individuals can have a disproportionate effect, at least temporarily. But if the product itself is good then its supporters will make themselves known.
The internet is a meritocracy; if sites like http://www.IHateFordTrucks.com are easily found it’s because other people think they’re worth linking to. If a substantial amount of disagreement exists then that will be easily found too. Look in one place and you might only find one view. Look around and you’ll get an idea of the prevailing view.
Having said that, I believe posters err on the site of disappointment and we should find ways to account for this bias.
Finally–
“Would you and your organization have made managerial decisions based on the number postings addressing the Y2K problem?”
I happen to think that the Y2K problem was something many computer professionals knew wasn’t a big issue but couldn’t get on air to say so because of the news media’s inbuilt bias toward stories of impending doom.
Had the blogosphere been more advanced we’d probably have found thousands of posts saying “don’t panic”. Since many of these posts would have come from people who appeared to know their stuff they would have compensated for the overriding message of “run for the hills” and thus given decision makers a more rounded view of the situation.

By: Tim Grace

Tim Grace — Fri, 23 Jun 2006 16:45:37 +0000

While I won’t argue with Mr. Neal that the CGM research methods discussed here don’t come close to meeting the standards of scientific rigor acsribed to the best tradtional market research methods, I find this to be a curious comment:
“There is no “structure” to the inquiry, so any respondent can say anything they want, and it is up to the researcher to interpret whatever those words mean. It’s akin to asking people to take a survey, but each can make up their own. There is absolutely no control of what is asked and what is responded to.”
This is quite true, but I’m unsure why this is mentioned perjoratively. Many compelling consumer insights that have subsequently changed the fortunes of brands have come from observational, qualitative/ethnographic research that recognized that structuring what we ask consumers often limits their ability to tell us what we really need to know. Uncovering these nuggets as a critical part of the insights generation process. While I agree that the representativeness of these insights uncovered via CGM research or ethnogrpahy should be testing against a wider population using traditional methods, dismissing CGM research seems like an overly purist position and not one most brand stewards (yes, this includes researchers) would ascribe to.

By: Peter Kim

Peter Kim — Mon, 19 Jun 2006 17:19:19 +0000

I hope it’s not lost on anyone that the comments here are CGM. So:
Should I ignore the discussion and not form an opinion? Maybe draft a survey and recruit a focus group of marketing executives to find out if CGM is good or bad?
Or should I read and analyse the opinions, follow links and check backgrounds for credibility, and then form an opinion on whether CGM research is rotten or not? Perhaps discounting the opinions of vendors on both sides, who invariably have a stake in the outcome?

By: Bill Neal

Bill Neal — Mon, 19 Jun 2006 11:56:09 +0000

Folks,
Please permit me to briefly respond.
First of all, I don’t know where this “Godfather” of marketing research stuff started, but I’m not it …. Alfred Politz, Art Nielsen, Sr., and George Gallup are the real godfathers of this profession. I’ve just spent the last 34 years working in it and trying to make it better.
As for Max Kaleoff’s comments, I’ve got a thick skin, so not problem with the replies at all.
I would just like to make a few additional observations:
1. Science, and the scientific method are the foundations of all acceptable marketing research. Without the discipline of a scientific foundation, all data, and the information gleaned from that data, represent information that is unreliable …. and I am using the classic scientific concept of reliability …. the ability to replicate an experiment and get the same results within acceptable bounds.
2. At its very best, CGM research is purely descriptive, and purely qualitative. It is not predictive and cannot ever meet the conditions for predictive research unless there is a well-defined population and a well-validated sampling frame from which that population can be sampled and tested. I strongly suspect that if someone tried to define the sampling frame for current CGM research efforts, they would see the immediate problems.
3. Therefore, one cannot (should not) ever consider using the results of CGM research in its present state as a basis for market segmentation, product/service positioning, product/service optimization, brand value analysis, or consumer/customer acceptability testing. These research-based protocols are the foundation of marketing strategy and tactics.
4. So what is left? A piece of qualitative, descriptive “research” based on a sampling from an incomplete and undefined population. There is no “structure” to the inquiry, so any respondent can say anything they want, and it is up to the researcher to interpret whatever those words mean. It’s akin to asking people to take a survey, but each can make up their own. There is absolutely no control of what is asked and what is responded to.
5. The bottom line is that CGM research cannot be relied upon for much of anything. So what if it includes some “super consumers”, “early adapters”, and “extreme loyalists and detractors?” There is no telling which are which. Who else is in that mix of responders? Just who do those to initiate or respond to blogs represent? We don’t know. So where is the science in that?
So, pardon me. Yes, I may be a traditionalist, and not hip to the new world of alternative media. But, science is science, and there is no science here.

By: Max Kalehoff

Max Kalehoff — Sun, 18 Jun 2006 13:09:36 +0000

Hi Toby,
Thanks for bringing Bill Neal into the fold — because he is well respected, as well as someone who needs to be inducted into the world of CGM. I just wish he learned more about the subject of CGM before making so many broad, INCORRECT assertions. I believe he owed that to your readers and his followers (which is a hefty bunch, to be sure!).
In fact, I was so moved by how off base I believe he was, I dedicated my last weekly MediaPost column to it — along with concurrent blog posting. I would like to invite you, Bill and others to a virtual Skypecast discussion to debate the issues of CGM research — an educational, good spirited discussion. You should hear from a few of us shortly. In the meantime, here are key points from my post:
http://attentionmax.com/blog/2006/06/is_there_really_something_wron.html
1. CGM Is Not Representative
“In many ways it [CGM] combines the worst elements of non-scientific research – self selection and advocacy – both positive and negative. That is, those out there in the Internet world who are generating their own media are self-motivated to do so and are not representative of any defined population of buyers. And, given the fact that they have taken a public position on a particular product or service, it means that they more often than not have exceptional or non-typical attitudes about those products and services–”
CGM is biased by self-selection and advocacy, and that’s precisely why it’s so valuable! CGM creators are, by definition, a highly defined population of hand-raisers … the ones so engaged with a category or product that they talk about it without prompting. These are the most elusive individuals, though highly valuable. They are your super consumers, your early adopters, your extreme loyalists and detractors, who have disproportionate influence on broader discussion and ideation. The argument that CGM is weak because it happens to be effective at identifying powerful stakeholders outside of the average is without merit. The problem with marketing today is that it too often constricts itself to the average, where findings are diluted, insightful nuances ignored and early indicators forgotten.
2. CGM Is Neither Credible Nor Reliable
“The information they [consumers] generate may be true, or not true – there is no way to discern which. Therefore, the information generated by those folks is neither credible nor reliable.”
Readers, participants and researchers in social media determine CGM credibility, reliability and influence according to a huge range of factors. For example: Am I familiar with the person or his affiliation? Is that person known to be trustworthy? How often does that person speak, and how often do others in the community respond? How often do others seek out someone precisely because that person is perceived as credible? How often is that person linked or referred? Are there norms and benchmarks, and historical indicators of trust?
Moreover, truthfulness often is irrelevant. Why? Because CGM acts just like media! It’s become one of the most prolific sources of online content. It’s attractive to search engines because of its robust text and numerous links, thereby enabling passionate information seekers and speakers to find one another. CGM tends to be compelling, refreshing and believable, and it is successfully competing for attention against all the other messages marketers continue to throw against the wall. Perceived credibility is why e-commerce sales increase when products are juxtaposed with consumer reviews.
3. CGM Includes Viral Discussion Which Marketers May Not Like
“But marketers must keep in mind that a few influencers can generate a great number of product mentions if they decide to feature a particular product or service in their blogs. And these things can get out of hand very quickly, signaling a problem that’s really not a problem to the vast majority of customers.”
What matters is not whether things get out of hand, but that there are influential consumers creating CGM, who spark ongoing, viral dialogue around products or services. This is CGM acting not only as media, but as influential news media.
4. I Don’t Know A Lot About CGM Research Firms, But I Do Know The Problems
“I don’t know a lot about them [CGM research firms] and have not used them in my consulting practice–But to the best of my understanding, they are primarily counting product/service mentions and, in some cases identifying the major sources of those mentions–I’ve already talked about the problems with simply counting the number of brand ‘hits’ and how that can be so misleading.”
CGM research firms are analyzing billions of public discussions to deliver a wide range of quantitative and qualitative research applicable in nearly every marketing situation. These measurements go far beyond product mentions and brand hits, and do dive deep into sophisticated consumer segmentation and insights, social influence mapping, and media measurements.

By: Laurent Flores

Laurent Flores — Sat, 17 Jun 2006 07:37:08 +0000

Dear Toby – to Jim’s point and in agreement with him, we discussed with Jim and in fact intially directed him towards your interesting post, I will be happy to share with you WOM/CGM traditional research data and build the case and comparison of the two data sources…not only for comparison purposes but also for the complementary nature of each data source that help each other to discover, shape and validate insights.
Feel free to get in touch, I will shoot you a note as well.
Hope we can circle back with Bill as well.

By: Toby

Toby — Sat, 17 Jun 2006 03:14:30 +0000

Hi Jim -
Thanks for adding your voice to this very interesting and I think important conversation.
Sorry I didn’t include Cymfony in the post; the mentions of the other companies were meant to be a tease/example. By no means was it intended to be a “list” of the firms that are doing work in the research/CGM space.
I would love to see some of your findings that compare CGM research against traditional research, as well as of course learning more about the work Cymfony is doing. If you’re peeking back in .. tobyb1@gmail.com
Ann, this would be an interesting topic for one of MarketingProf’s audio seminars..especially if both sides of the issue are presented by industry experts

By: Jim Nail

Jim Nail — Fri, 16 Jun 2006 10:16:35 +0000

A lot of valid points, but I think the bottom line is CGM is a new data source that brings additional insights, but is not a substitute for other research.
I agree with many of the points above, especially:
CGM is a medium from which you derive research data: at Cymfony we use a number of techniques to pull out the data.
No manager should make major changes based on CGM alone. Bill analogy to not acting on focus groups alone is dead on, as well as his point on trending data being more informative than point-in-time snapshot.
CGM is not representative of the population at large. True, but as Fred Reicheld has shown, advocacy drives sales and profits. So shouldn’t you care more about what promoters and detractors say than some Joe off the street who doesn’t particularly care one way or the other about the category?
Spontaneity is key to the value of CGM-derived data — every study I have ever been involved in there is a discussion of what bias we introduced by how the questions were worded, how the moderator handled the session, etc.
But I also have to ask Bill this question: with participation rates for surveys commonly in the 10 – 15% range, are those surveys statistically representative?
I think we are all in agreement that no single research technique or data source is enough. I advise our clients to test their CGM findings against their other data first, and if there is disagreement, consider further studies to validate or refute it. And a lot of questions clients are interested in just aren’t discussed (we have a client know asking us to look for discussions of what information people look for on packaging — but people don’t talk about this).
PS: Toby, I’m, of course, disappointed you didn’t include Cymfony in the list above. I’d like to update you on some new stuff we are doing.

By: Laurent Flores

Laurent Flores — Thu, 15 Jun 2006 21:30:36 +0000

interesting but I will say that the world is changing and the world of research needs to change as well…I’ve been in research for over 15 years now, working in the US, and Europe, and Asia, and now able to drive new insights to clients by LISTENING better to consumers…CGM offers wonderful abilities to tap into insights in a more natural way, a more conversational one, a true one…to this end, I guess that most researchers will gain more to be open to new ways of learning from consumers…of course, traditional quant research is relevant but still the coming years will see entries of new players in th research world, some players that will provide better insights…thse players will have a sound research background, leveraging technology and consumer intimacy for the better…
So, I do not agree, and because I listen to a changing world, I also think that the world of research will be a better one with better openess to the world…
No?

By: w. D. Neal

w. D. Neal — Tue, 13 Jun 2006 12:07:53 +0000

To all – good points on the postings. I don’t have any particular additional points to make in reply, EXCEPT to remind everyone of the great Y2K debacle. Do you remember how the world was facing a potential disaster because of the “very real” problem in so many legacy computer programs? Remember all of the articles, editorials, and commentaries on the enormity of the problem and its likely negative impact on commerce and daily life? As it turned out, it was a non-problem. Consider how this issue would have been handled in today’s blogosphere …. I would suggest that the traffic would have been very high, the opinions varied, but mostly trending to the disaster scenario. Now, ask yourself …. would that information have been valid and reliable? Would you and your organization have made managerial decisions based on the number postings addressing the Y2K problem?
The bottom line is this …. in some product/service categories researching CGM is probably a good idea as one of many tools for uncovering and identifying potential product/service problems and issues early on. But that’s it. You need other more scientific and quantitative research to determine if those problems are real, or not. …. Bill Neal

By: Paul Fabretti

Paul Fabretti — Tue, 13 Jun 2006 10:10:39 +0000

Toby
I couldn’t agree more. The worst thing that could happen is if metrics got in the way of free-flowing consumer opinion.
You have opened up a can of worms for me. I had not really considered the duality of CGM as both research findings AND ad medium before and it is beginning to hurt my head!!

By: Toby

Toby — Mon, 12 Jun 2006 22:49:50 +0000

Thanks for your comments on a discussion that will, no doubt,continue to take place with many marketers and researchers.
Paul, I stand corrected. I agree with you CGM is not a research technique but source of rich, raw information. What excites me is the technology allows, for the first time, time the ability for marketers “listen in” to unfiltered consumer-talk. It will be companies like Andy’s that will develop techniques that will help us analyze the data in a credible fashion.

By: Paul Fabretti

Paul Fabretti — Mon, 12 Jun 2006 20:29:19 +0000

You ask, “where and how does CMG “fit” as a marketing research technique?”
Why would you even consider it a reasearch technique in the first place? Surely CGM is the MEDIUM for customer’s voices and NOT the method of measurement of satisfaction?
In my opinion, CGM is about providing a medium for the consumer vocie to allow them to air their opinions about a brand/service/product.
Now, you might argue that in itself, the fact that they have aired any opinion (video/blog etc.) is simply the publicly-aired element of research, but I would also argue that with so many different CGM media around, there should be a seperate reasearch element to anaylse overall opinion over all of these methods.
With podcasting, wiki’s, blogs etc. being used by so many different types and ages of people (and with different levels of production skill in each) it would be foolish to consider any one medium as an overall opinion of my product.
I believe that you can only get a valid “market research” opinion after having assessed and reviewed all the methods you argue are the research themselves.

By: Andy Beal

Andy Beal — Mon, 12 Jun 2006 18:28:56 +0000

Hi Toby, thanks for mentioning Fortune Interactive. Some great info shared by Bill. At Fortune Interactive, we agree that tracking simple “mentions” is not enough. That is why we also track influence, audience reach and sentiment. We want our clients to understand which “ripple” caused the “tidal wave”.

By: David Armano

David Armano — Mon, 12 Jun 2006 15:42:17 +0000

“The information they generate may be true, or not true …. there is now way to discern which. Therefore, the information generated by those folks is neither credible nor reliable.”
That’s a pretty B&W statement for a grey topic. While it prabably isn’t wise for marketers to put all their eggs in any one basket when it comes to research–the opposite is also true.
To me CGM research seems like a no-brainer to at least consider. Yes many people have agendas–but it’s also a place for open, honest conversations which rearely happen in things like focus groups etc.
Point in case. What if Apple chose to ignore the digital conversations that were happening around the iPod battery issue? They would have risked alientating the same people who help evangilize the brand.
Really interesting discussion Toby. Thanks for documenting the conversation you had with Bill.