Comments on: Experiment or Go Home!

By: The boss is always right | garygilliland:

The boss is always right | garygilliland: — Wed, 03 Mar 2010 23:56:30 +0000

[...] From Experiment or Go Home [...]

By: Mike

Mike — Sun, 20 Dec 2009 12:07:46 +0000

Great article! It really got the wheels spinning on the various ways a tool like that could be used. Thanks!

By: Larry Epstein

Larry Epstein — Fri, 07 Sep 2007 02:07:54 +0000

Your description of the HiPPO scenario is right on the mark. We see it so often — either the HiPPO drives decisions or, worse, you have complete organizational paralysis because there’s no way to settle arguments.
But most of our customers would disagree with the characterization that inserting “a few lines of javascript code” is going to instantly remove IT and, for that matter, Web designers and coders from the process. It’s just not that simple.
Obviously, it’s in the interest of the JavaScript-based vendors to underplay the fact that unless your content is in old-fashioned circa 1995 static HTML, the page elements you wish to test are going to require reworking. And if your web content is generated by a back-end database (as is the case with virtually any ecommerce site), substantial work on the programming side is going to be required before your first test can be run.
I’d advise your readers to take a long, skeptical breadth when they hear the words “a few lines of javascript code”. Ask some hard questions, particularly about testing dynamic and database-driven content. Then speak to a reference account and ask the simple question, how long did it take to prepare your first test? The answer will surprise you.

By: Avinash Kaushik

Avinash Kaushik — Wed, 05 Sep 2007 16:14:26 +0000

Cam : You can totally test entire user workflows, though it requires a bit of technical creativity. A little while back I had coined a term for it “Experience Testing”.
(It is covered more in the book – and the blog – but I left it out here becuase I think I had already exceeded MP Daily Fix words in a post by 100% already!)
Couple ways to do it:
1) If your ecommerce platform supports it, like say ATG, then you simply create the different experiences and the platform with not a lot of work can take care of the splits.
2) You can have A/B/C versions of a page and each funnels people into a new flow (this is a bit more work but doable).
3) With the multivariate option you can still do #2 and depending on your IT complexity sometimes it might be easier using the tools mentioned in the post (becuase they do drive large amounts of IT independence and “set you free” to be a Marketer!).
In summary: It is possible. And I like the way you think!!
Emerson : I agree that the world would be a better place if we only listened to Analysts.
On a serious note, I encourage Marketers to consider different kinds of inputs. Wisdom from Analysts is helpful. So is the beauty of not having to do a three month RFP to find out what you need. Just install a free tool. Become smart by actually doing. Ditto for calling a couple of industry players and getting them to give you a free two month trial.
Learn by doing.
Hmm… Maybe I am a marketer after all, I can come up with slogans!
Thanks everyone.
-Avinash.

By: Emerson Hartley

Emerson Hartley — Wed, 05 Sep 2007 15:08:48 +0000

Have you seen the Jupiter Research Report? It stated that of all the multivariate testing players out there, Memetrics is ranked number one for a variety of reasons. Avinash, I encourage you to purchase and read this report from Jupiter Research.

By: Cam Beck

Cam Beck — Wed, 05 Sep 2007 14:25:43 +0000

I wonder how easy it would be to test not just the page layout, but the entire user workflow in this manner.
I suppose one could just link to unique pages in each instance, but it might be difficult (and costly) to do the accounting.
I left a comment on your three questions post, too. I can see why you call them “golden.”
Excellent work.

By: Lewis Green

Lewis Green — Wed, 05 Sep 2007 13:22:01 +0000

Avinash,
Good post full of useful information. I particularly like the three questions. Thank you!