The third in a series of webinars on Maximizing Magento – Optimizing Conversions with Website Testing. Google and the Magento team are joining forces in presenting Magento’s native implementation for Google Website Optimizer.
Full Webinar Transcript
Just going to switch to the next slide here.
This is our largest webinar to date, with over 1200 companies registered for the event. With us today is a diverse audience, representing a large spectrum of the e-commerce world, from the leading e-commerce merchants on the web, systems integrators and consultancies, to smaller businesses that are just starting out. We are excited to see such a huge turnout for the event with online merchants from across the world.
Before we get started, if you’re a Varien support customer and have any questions about the functionality you see today, we encourage you to open a support ticket. If you would like information about Varien’s support offerings for Magento, please visit magentocommerce.com/support.
As always, we invite you to join the conversation and the Magento community. The Magento blog is updated frequently with the latest Magento information. Visit the blog and subscribe to the Magento blog RSS feed today, follow Magento on Twitter, and join our Facebook and LinkedIn user groups. Interaction with the Magento community is an important part of our efforts to spread the word about the product, and we’d love for you to take part.
We’ll begin the webinar with Trevor Claiborne, product marketing manager for the Google Website Optimizer team – power tips and tricks that you won’t want to miss.
We’ll wrap up the webinar by answering questions submitted by you, the audience. Yoav Kutner, Varien’s CTO, will join us for the final segment of the webinar.
Throughout the webinar, please feel free to ask questions using the questions and answers module to the right of your screen. We’ll pick the best questions, and we’ll go through that, as I said, in the final segment of the webinar.
Without further ado, Trevor, the stage is yours.
Trevor Claiborne: Thank you, Roy. If you can just give me… There I go. Hi, everyone. Just one second here while I get everything showing. You probably see a large penguin right now. And I should see some slides.
OK. So, I just want to bring up a slide of the plan for today. So, I’m going to start off with a little bit of background. Then, I’m going to go into what is website testing? What is Website Optimizer? Then, we’re going to talk about a few best practices. I’m going to walk you through how Magento has done this integration, and then going to do a couple live landing-page critiques for Magento customers. And then, as Roy said, we’re going to have time for questions.
So, to start off, who am I? I know, on the webinar, you don’t see who you’re talking to, so I’ll include a little picture. So, that’s me and my Google swag.
I’m a product marketing manager at Google. I’ve been at Google for about two years. Other products that I’ve worked on at Google include AdWords, Google Trends, Trends for Websites, Insights for Search, and AdPlanner. These are all really metrics-driven products, and I was drawn to Website Optimizer because it really allows you to take control of your data and make some really actual insights and really move your business forward.
So, without further ado, what is website testing?
Website testing is a practice. Instead of showing them one page, you show them different variations of a single web page. In this case, we’re showing three different variations of a bike website, and we’re deriving traffic from 33 percent of the visitors to each page.
And then, we measure what happens. We see what page is performing best. What page am I getting the most conversions from? In this case, the top version, our original version – 14 percent, eight percent, and then 25 percent. We see what page has the best effect on our visitors and then we call that our winning combination.
From there, we could implement this variation as our original page, and then we could see that higher conversion rate across all traffic. And this is what website testing is all about. It’s about trying new content, deciding what works best, and then increasing your conversion rates and increasing your dollars.
So, what is Website Optimizer? A free testing product from Google. In November 2006, in beta, as part of AdWords. And then, in April 2007, we released a standalone version, and we also dropped the beta label. If you’re familiar with a lot of Google products, you’ll know that dropping the beta label is kind of a big deal. So, we’re pretty proud of that. And we’ve been continuing to add new features as it goes, and we’re having a lot of fun with it, and we’re seeing a lot of people increasing their conversions. So, overall, very good.
Website Optimizer can handle two kinds of testing: A/B testing and multivariate testing. A/B testing is testing one page against another. Now, you can also test one page against two other pages, A/B/C testing, A/B/C/D/E. You can test, I think, up to 18 pages against each other. But, A/B testing is one page against another page.
Multivariate testing is when you break down your page into sections and you test different combinations of the sections, say, for example, you might have two headlines, and you can run all four of those combinations and Website Optimizer will tell you which one’s working the best.
So, here’s a graphical depiction of A/B testing. You have A versus B. Pretty simple. And these pages could be very similar to each other, or they could be radically different. You could try a different layout. You could try different images. You could really do whatever you wanted. Two entirely different pages.
Multivariate testing – This is kind of a wire frame of your page. So, you’ve got a headline up top here. You’ve got your image, your “buy now” button, and your product description. So, maybe you have a few ideas on how you can improve this. And so, you run your better headline, your “buy today” instead of “buy now,” and you try shortening your product description. And so, you try all these different variations and you see what works best. And Website Optimizer will serve all these different variations for you and then report back what is converting the best.
Really, it comes down to conversion and money. If you are converting, you’re making more money. If you’re making more conversions, you’re making more money – all very good things.
And you also pay a lot of money to drive traffic to your page. Whether you’re using banner ads or affiliate programs, or if you’re using AdWords, a lot of the traffic that comes to your page is driven to your page by money. And when people come to your page and they don’t do what you want, you’re paying money for traffic that isn’t helping. So, if you’re not testing, you’re letting money kind of fall of the table.
Every person that you pay to come to your page should convert, hopefully. But, the average conversion rate is about two percent. So, what are you doing with that other 98 percent? With testing, you can create that conversion rate and move up.
Another reason to test is you can learn to be wrong quickly. Here, probably, a website redesign. They’re trying to try new variations, build a new website. They’re spending a lot of time and money on building a new website.
What if that page isn’t as good as you hoped? What if that page, instead of improving your business, actually hurts your business?
So, with website testing, you can learn to be wrong quickly. Because you’re testing your page and you’re keeping track of your conversions, especially against your original page, you will know if things are going wrong. So you may be wrong quickly, but you’ll only be wrong temporarily.
I want to show you an example of this. Here we have a stair-supply store. And this example’s provided to us by a company that works with us called LunaMetrics. They’re one of the Website Optimizer authorized consultants. They have this nice stair page, and they built this out, and they thought, “Well, let’s see what we can do to improve this.”
So, they built out this new page. You can see here, they’ve added this new navigation. I’m going to jump back and forth here. On the left-hand side, you have this navigation. They’ve added the live contacts here, so you can get in touch with someone. And they’ve done a few other additions as well.
Now, they thought, “Oh, here we’ve got this page. It’s going to do great. We’re going to improve our conversions.” These changes cut their sales in half. And imagine if you launched your redesign and suddenly your sales were half of what they were. Your boss would be furious. If you are the boss, you would just be scrambling to figure out what went wrong.
With website testing, you can avoid all this, and you can hopefully find changes that improve your conversion rate. But, in the case that they don’t, you’ll know when you’re wrong and you can fix it quickly.
I want to show you another example. This is an example from Google. This is a Picasa download page. Picasa is a product from Google that lets you organize and manage your photos – a very good product, if you want to try it out. But, this is our original download page.
Well, we had conversion rates of around, I think, 15 to 20 percent. And a conversion for us is downloading the software. You can see there’re a few different sections here. We have our headline. We have our download link. We also have some text and a product image.
So, we decided to try these few different areas and see what we could do about improving our conversion rate by adjusting some of this.
So, we had some theories. First, we thought that it would be better to describe user benefits rather than product features. So, we could try a different headline. Secondly, we thought that links would receive more clicks than buttons. We also thought that a free offer was better than a trial offer. You think “free” is free forever. “Trial,” I’ve got 14 days, 24 days, 31 days. Who knows? So, we thought “free” might be better than the “trial.”
Also, we are proud of our product. We thought it had a pretty good user interface, so we wanted to show off the product so people could see it.
So, we tried a few different things. So, these are all of our originals, on this left-hand side, and this is our new version. So, this is an A/B test. And so we tried a new headline, new button, and we tried removing the picture.
And this simple test, we saw a 30-percent increase in downloads – small changes, really, and not rocket science by any means. Try a button, set up a link, try a different text, got a 30-percent increase.
So, just by running this – this test took about two weeks to run – saw a 30-percent increase. And we didn’t pay anything for it. It’s pretty amazing to see the level of increase.
So, one thing you might be asking yourself is…
Oh, sorry. A little more left to this slide.
So, we found that user benefits were better than product features. In this case, product features worked out for the better. Our theory about links? Not so true. Buttons worked better. Free offer versus the trial offer? Now, this isn’t specifically a trial offer, but the idea of “Try Picasa now,” maybe it’s a little bit easier, a little less committal. And then, attractive product flaunted? Apparently people don’t like seeing the product. They just like to download it and try it out.
So, you might ask yourself: “How do I know what’s happening? How do I keep track of all these conversions? So, Website Optimizer offers you these reports.
And so, I’m going to run through this report. And basically, all these are combinations or different variations put together. So, combination five might be headline two, image three, and description two.
And all along here, it’s showing me… Up top is the original. And this gray bar here represents my original conversion rate. It’s got some variance to it. And then here are my variations. It’s just how they’re performing relative to the original. If they’re in gray, it means they’re either performing a little bit better or a little bit worse. We don’t have any of those in this image. If they’re in red, they’re performing much worse. And if they’re in green, we’ve got some possible winners here.
If you look on the right here, you see the “chance to beat original.” And this is the chance that that combination will beat your original combination. And then, here we have our observed improvement. So, here we’re seeing a 16-percent increase, a 16.4-percent increase, and then for this really poor one, a 30-percent decrease.
Now, what’s great is, when I see something like this happening, I don’t need to hold on to it. I can just click this box and I can get rid of that variation and don’t worry about it anymore. We call that pruning.
But, this is the way I can track exactly how my different variations are doing and what kind of improvements I’m seeing.
We also have another report called the Page Sections report. Now, this report breaks down the different sections into parts and shows you which sections are having the most impact. If you’ll look here, we’ve got a body, a testimonial, a case study, a title, all the action buttons, and another button.
This is for the Website Optimizer home page. We’re trying different variations to see what would have the most effect on sign-up. And as we tried different things, we found that, really, the most impactful thing was the text that we used in the body for the description of the product.
The addition of the testimonial didn’t really have any impact: it gets a zero out of five. The different headlines had almost no effect. We can see there was some movement in terms of conversion rates, but when it came down to it, the body text was really what moved things forward.
And we’re able to see, in this section – so, if you take this to light with your own pages, you can think, “Oh, is my product image the deciding factor? Is my headline the deciding factor?” And you can see which changes have the most impact. You can put your time into deciding, “Oh, what headlines do they respond to? Oh, well, because the headline’s been this impactful, I should spend some more time figuring that out.”
I’m going to go into a few different best practices here. The first idea is you should make a small number of big changes. And when I say a small number, I mean maybe two to three. When I say big, I mean notable. You have generally less than five seconds to make an impact on your customer or your visitor, and if they can’t see a difference in a few seconds, and if you can’t see a difference in a few seconds, neither will visitors.
So, a good test to see if a variation’s worth trying is to show something to a coworker and like, “Hey, do you see what the change is I made to this page?” And if it takes them longer than a few seconds to realize it, you may not have a big enough change.
And also, if the changes aren’t very big, they won’t make much difference in your conversions. And so, let’s say you have two pages that are very similar. If they don’t perform much differently, you’re not going to get results. So you should try and make some big changes first, making drastic changes. And when you have a good direction from those big changes, then is the time to fine-tune and irk out those last few drops of conversions.
You should test that. So, you’ve been seeing this along the bottom of your screen here. We put it on a t-shirt. We think it’s pretty important. You should test that.
What does that mean? It means that website testing is not a one-time deal. A lot of people, they like to run one test and they say, “I’ve run my test. I’m done.” Not the case here. Website testing is an iterative process. You want to try many different things. You want to iterate and keep on testing.
So, you run a headline against another headline and you see a 15-percent increase in conversions. You’re like, “Wow!” You go back and you say, “Well, how can I beat that?” And it’s a matter of constantly moving forward because, until your conversion rate’s 100 percent, you can’t say that you’re completely successful.
It also means you should test, really, any idea. And you should be methodical in your approaches. So, you want to try lots of different ideas. You have like, “Huh. I wonder what would happen if we did this.” Just test it. You’ve got a testing platform here that’s extremely easy to use, especially with the Magento integration. You can set up your test in minutes.
And later in the webinar, I’m going to show you how to set up a test using Magento. It’s very quick. And if you have ideas, test them and see if they really work. And that’s really what you need to know about that. And just, the idea is: start now. There’s nothing holding you back. And there’s a huge opportunity cost to not testing, which I want to talk about.
People have a lot of reasons for not testing a website. There’s lots of work to be done, especially if you’re a small company. There’s just so many things going on that, “Oh, I’ll get that website tested later.” If you increase your conversion rate, it has a lasting impact on your business. If you increase your conversion rate by 10 percent, that might mean another $25,000 in the bank over the course of a year. And every day that you don’t test, you’re missing all those days of higher conversions. And it’s just a huge opportunity cost.
And we’ve seen this time and again. People will run that first test, they’ll see a small increase, and they’ll stick there. And they’re missing out on this huge opportunity to really improve their website. They don’t need to drive any more traffic to it. They just need to make the traffic they have work better.
The power of compound testing. So, if you remember compound interest – I’m going to try and do a little drawing here. It’s kind of a back-of-the-napkin kind of sketch.
So, let’s say this is your sales, and this is time. So, it’s flat, and so are your sales.
Well, let’s say you run a little test – little test – and you see like, let’s say, a five-percent improvement. That’s cool. And then you stop. You get all this, and you get to hold on to it.
Well, let’s say you kept going with your tests. Let’s say you saw a five-percent increase and then you ran another test. And then, you ran another test. And then, you ran another test. You didn’t really notice any impact from that test. Then, you ran another one. Now, you’re seeing this increase. And this is compound testing.
So, you may make some small changes, and you may see some small increases. You may see some big increases. We’ve seen tests where people see 100-percent increases in conversion rate, doubling their conversion rate.
But, maybe you only see a five-percent increase. Five percent has a lasting effect. And then, if you go again, there’s another 10 percent increase, another 12 percent increase. As that happens, your conversion rate is going to move much higher than your previous base line. And that’s going to mean a large amount of sales and more money over time.
Also, another idea I want to share with you is that winners are not always intuitive. Oftentimes, we don’t expect the winners. We have an idea of what we think will work best, and then we put it into practice and it turns out not to do very well.
Just from our own samples, we had heard that videos were very effective in getting people to sign up for products. So, on the Website Optimizer home page, we decided to put up videos in a test. We had two different videos: we had a product tour, and we had an A/B demo, which, basically, was our product walk-through. And then we tested against our bullet points.
And we ran the test, and we saw that videos dropped 30 percent. And we were like, “Wow. Videos did not work for us.” We were expecting them to work, and they just did not work at all. Well, it’s a good thing we know.
Likewise, winners are not always intuitive. What’s important about winners not always being intuitive is that it’s worth testing any idea. This goes back to “you should test that.” If you have an idea, it’s worth testing it. At the very least, you’ll find out that it didn’t work at all, or it performs the same as the original. But, you might find out that it does a lot better than your original idea.
Also, it’s important to keep track of the results of your tests, especially the surprising ones. If you have something that’s unintuitive and it’s a winner, you should keep track of that. You should pass that around and tell people at your office what’s going on: “Hey, I ran this different headline, and we thought it was going to suck, but it actually increased our conversion rates 20 percent.” People will get excited about that, and they will get involved, and you’ll all kind of work around trying to improve this website.
Also, let the data speak for itself. Website Optimizer does a lot of statistics, and when it predicts a winner, it’s usually, I’d say, 99.9 percent accurate on that. We use a lot of stats to make sure that when we tell you you have a winner, it’s actually going to be a winner. So, if the data says something’s good, you can run with it.
And you can use that data to prove your point. Let’s say you have a boss that’s like, “Oh, we’re not going to do it.” It’s like, “Hey, look. The data says we have a 15-percent increase in conversions from this.” That’s going to be powerful data.
Lastly, beware of the HIPPO. You may have heard the term “HIPPO.” It’s a term that Avinash Kaushik uses. Avinash is Google Analytics’ evangelist, but also just a really savvy guy in understanding a website. A HIPPO is the highest-paid person’s opinion. This could be your boss. This could be you, if you are the boss, possibly. But, this is the person who says, “We’re going to run this,” and they basically dictate what’s going to be on the page.
The HIPPO is your enemy here, because, as we said, the winner is not always intuitive and experience is not always the best teacher here. With website testing, you’re letting your visitors tell you what works best. So, if you have a HIPPO, testing is your way to work around it. You can come back to them with data that shows: “These versions work better.”
So, just a few more resources from Website Optimizer, and then we’re going to jump to the walk-through.
So, google.com/websiteoptimizer is where you can find the page and the product. And we’ve got some best practices and case studies there. We also have a team of authorized consultants. And these are consultants who can help you with your testing. They can help you with implementation, with ideas. They can do anything there.
We also have a blog: websiteoptimizer.blogspot.com. You can also find some forums on landing paging. You might want to check out one of these two books here. The first is “Landing Page Optimization” by Tim Ash. He is also one of our authorized consultants. The second one is by Bryan Eisenberg: “Always Be Testing.” Bryan is another one of our consultants, but he’s also just kind of a guru in the field of conversion optimization, and he’s been doing this for over a decade.
So, I’m going to jump into the Magento walk-through here. So, let me close this out and switch to my Chrome.
See if this is the right one. Nope.
There we go.
All right. So, we are in our Magento page here. And the page that we’re going to optimize is going to be this Microsoft Natural Ergonomic Keyboard 4000. We can see what we might want to optimize.
So, we’ve got this product image here, which looks fine. I don’t see any issues with that. We’ve got the Magento slider. Cart button. OK. We’ve got a description. We have a headline. Another longer description.
OK. Well, this looks good. Let’s see what we can do.
So, if we want to do an experiment, first thing we want to do is go into Website Optimizer. So, I’m already signed in here, and I’m just going to click “Create another experiment.” For this, we’re going to run a multivariate experiment, in which we’re going to try different sections.
OK. I’ve read all this. I’ve completed the steps above. Create.
So, you want to give your experiment a name. And I’m just going to call it “MS Natural Keyboard.”
And test page URL. So, this is going to be your product page. Let’s just copy and paste.
Now, for the conversion page, I’m going to want to go into Magento for this. So, here I am in the Microsoft Natural Ergonomic Keyboard. And I’ve clicked over to product-view optimization. Now, the page I want to use as my conversion page – they list all these for you – is the order-success page.
One-page checkout. OK. And that’s the URL I need. Let me copy that. Having little issues here with the copy and paste, which is a little embarrassing, to be honest. But, this is what you get when you do a live demo.
This is really annoying. I’m sorry.
All right. I got it.
So, I’m going to paste that in there, and add that T back there that got cut off. And I’ll just continue.
So, what Magento did is it took all the scripts that Google provided and it popped them here. And I’m going to explain what these scripts do real quick.
So, you’ve got your control script. And the control script, what it does is it is basically the scientist in your experiment. Basically, every time your page loads, it says, “OK. This is the person’s new experiment. What page should I show them? Should I show them this variation or that variation?” And it handles all that.
Then, we’ve got the tracking script. Tracking script keeps track of what people do. Did they visit this variation, or did they visit that variation? This counts if people converted or not.
The next thing we want to do is we want to select our attributes. So, these are our sections. So, for these sections… Let’s see. I want to work with a name. You know what? I’ll just select them all. And I’m just using control-click to select these. All right. I’m going to hit save and continue to edit.
So, now that I’ve done that, I go back into Website Optimizer. So, when I hit save, Magento installed the scripts onto my page. So now, I just click “Check status.”
Oh, “Installation not started yet.” So, what we’ve run into here is not a bug. What the issue is is that your order-success page, when someone orders, the only reason they should see an order-success page is if they’ve actually got it. Well, we can create a little fake conversion page and get Website Optimizer to figure this out. So, here’s how we’re going to do that.
Take this conversion script – and this is one of the secret tricks I’m teaching you right now. Then, you can open up a Google document, or a Notepad file. And I’m just going to go into “Edit HTML.” And I’m just going to paste that there. Update. See if that worked out.
Alternatively, you can do an off-line validation, which also works.
All right. And then, I’m just going to share and publish this as a web page.
So, what I’m going to do is… So, Website Optimizer doesn’t need to find the conversion script on your conversion page, as long as the conversion script is on the page that you want it to convert off.
Just make sure that went through. And just in case I’m having some issues with Docs here, I’m going to do this in Notepad as well.
So, if you also want to do this in Notepad, you can just create a very simple HTML document. And then, you can just save this as…
I’m sorry. My computer is having a few issues here.
As you can see, Notepad has stopped responding, which is awesome. There it goes.
So, we’ll save this as “fake.html.” And then, it’s all set.
So, what we can do is we can go back into Website Optimizer.
Just make sure this one worked here. I apologize for the issues with the live demo here.
I’m going to reload this page. Apologies.
OK. Let’s publish that page. There we go. Publish document.
So, I’ve got this URL here. What I can do is I can go back into Website Optimizer, and for my conversion page, I can just give it that dummy file.
And you can just continue. “Web team will install.” Check status.
“Installation not started yet.” Oh, dear.
All right. Well, for now, what I’m going to do – I’ve run into this issue before and it’s worked out fine, and Yoav can talk about this later, but we’ll just use the shopping cart here as our conversion instead. Let’s go back and redo this real quick.
Ah ta doo… I am going to reinstall the scripts, because I changed the page. And say “continue.” And Magento is saving. All right. And test.
I am running into an issue here, so I am going to move on with this. I apologize for this. The demo worked very well in practice, and I have done this experiment using Magento and it works very well.
I am going to skip ahead, here. I do show you what would happen here.
When you do get into Magento, into a soft match, with your variations, you have the option of creating new variations.
This is an experiment that I set up on my category page. Here, for example, are the descriptions. I can save the filler description here. It is all under variation. So you have a better description.
I just added here. And now, I can edit this freely and change whatever I like. We sell the finest wood furniture in the State of Maryland.
Trevor Claiborne: Let me just jump in here. There seems to be a lag between the slides coming up and the actual talking. So, if you could just take it slower.
Roy: Oh. Thank you for addressing that. The slides may be the issue.
Trevor: Also, one thing. Questions. If you have any questions throughout the webinar, please feel free to submit those. We are looking for great questions, on the topic of course, and you can do that using the Q&A module on the right of your screen. Thanks.
Roy: I go to this other variation here. I apologize for the demo not working as well as it should. I can preview these combinations. What I am going to do is… Actually, we will preview. That will work better. In preview, what is going to happen is that it will show the different variations.
Here is my page, here. I have my very nice image, OK? Let’s see what my other worksheet looks like, with the better description.
Here I see my better description preview with all the variations, the two variations. I take those. You can do an integrated page optimization at three different levels, in fact. We can do it at the product level, at the category level, and also you can do it at the page level in your CMS.
All of these set-ups work very well, despite the earlier issue that I had today. Really, all of this… As you take your conversion page, you paste the URL that was the issue and Magento will install the screens for you.
With that done, I am going to move onto a couple of mini page critiques. Then, we are going to move onto questions.
Let me bring up my slides, again, here. Let’s see if it goes. There we go. I pull it up live. This is not so much a critique in the traditional sense. I am not looking for what’s wrong. In some cases, there might be things that can definitely be improved. But, we are looking for opportunities for what we can test.
We have a feeling that our page is pretty good as it stands. But, there are always possibilities for us to improve our conversion rate.
The page we are going to the look at is for Choe. What do they do? They make really good chocolate. I am speaking from experience there. It is delicious. And their site is choue.com. You may have seen them. They are a Magento case study. I have them right here. Actually, that is NerdyShirts. Here is Choe.
This is Choe’s home page. They asked me to take a look at their store. Let me pop that open. This is their store. This is where you might end up if you happen to click on an ad that is up here.
Let’s take a look at what we have here. My account log-in, my heart search store… OK. Different bars. We have their flavor wheel, which are their branding elements. They have different kinds of chocolate here. All wrapped. OK.
Then, they have some tasting packs and some gifts. It is kind of a long page. The first thing I would consider testing is putting the categories up front here. You have them here. And then, maybe, trying a left navigation, which might be more effective. I totally missed the different categories here.
The long page… And you have different pieces here. Also, the bars that they are leading with – I have citrus, nutty, fruity, chocolaty, and then “discover your chocolate.” I might start with “discover your chocolate.” You have these unique flavors here. You may want to push those forward. And then, maybe, leave with chocolaty, which is a very more approachable flavor, I think, versus fruity.
Depending on the kinds of traffic you are driving here, you may or may not find that it works better to have a more approachable flavor or one that is more out there. You can test that and see what works out.
You have these tasting packs which look pretty good, and this gigantic pack. There are opportunities here. You could try changing images. You have the 32 pack bars. Maybe even try that one unwrapped.
I have chocolate… Showing a big mountain of chocolate might be really appetizing for people. You have gifts. Looking at the gift collections here, you don’t have any prices listed. This could be an opportunity to list individual products, rather than just having gifts as one big group.
You have your treats section. I have a zoomed in view of the macadamias and the mango pieces. Then, here, you have a bundle. This seems appropriate, though you might want to leave…
You have these three zoomed in views, and you have the one bundled view. You might move this one, the bundled view, down and have the three bits center-stage, since they are more consistent. If they nips and we have some confusions.
That is a two minute look at this page. There are a lot of things you can test here. You can try many different things, and see what works better. That is the whole idea of testing.
We are going to move onto our second one here. This is NerdyShirts. What do they do? They sell nerdy T-shirts. They are cool. They have things like this one. Let it Bii. This is one of my favorites. If you are a Beatles fan, you can see that it is the “Let It Be” artwork with Wii avatars. Yeah, I like it a lot too. But, I am biased.
We have the NerdyShirts page. Here I am on their NerdyShirts page. I see that they have this promotion running. This promotion, here, when you buy two shirts, you get a free gift. Three shirts, 10 percent and a free gift; five shirts… OK.
Let’s look at the promotion. Promotions are a great place to test. You can see what promotion really works. Here, they have three different offers. It is a cascading offer. Maybe they should try just running one offer – buy three shirts, get 10 percent off and their free gift – rather than including the two shirts and the five shirts.
Also, they mention the free gift, but they don’t actually tell you what it is. They could try experimenting with different free gifts, say a free belt buckle, and then see which makes the most purchases.
They could do away with all of this; say, 10 percent off of your order and have a very simple promotion. Also this promotion is a bit big, just size-wise.
If you look at it, it moves a lot of the shirts below the fold. You really only have four shirts here at the top. And these are all Nintendo and Mario-based. I am not looking for Nintendo shirts. So, I might say: “Oh, there is nothing here for me.”
But, you really have quite a few shirts coming down here. There is the Let It Bii shirt. You have Jack Bauer. In fact, they have a lot of shirts. And they are all on one page.
Maybe, one possibility for them would be to categorize some of these shirts. You have some Nintendo shirts here, some political shirts, a little Obama. You have another Nintendo shirt. One Nintendo shirt on… You have TV shows, Nakamura from “Heroes.” I would try to make some categorization.
I am going to jump into a shirt here, and take a look at the product page. For this product page, here, they have a few different images. I have a nice looking model to show off the shirt, this “buy it” button, and actually, I want to close…
NerdyShirts just completed their first test last week. They tried this buy button versus this buy button, and they saw a nine percent increase in conversion. Congrats to them on their first test. I am hoping that they will go through many more. With that, I am going to close. Thank you, and let’s move onto questions.
Trevor: I’m Trevor. Let me switch back here to my screen.
Roy: Please do.
Yoav Kutner: Hi, everyone. This is Yoav Kutner, Varien’s CTO. I want to talk about this issue with the Check-up Success page and the My Account page, as well. We are actually going to have a solution built into Magento to smooth out this process and to enable you to use this a bit more freely and easily than what we just saw. That, we need to do. With that remark, I think we can jump into the questions. This will fulfill some of the information that one of the questions asks about. So, we can just start.
Roy: OK. I am still working on changing the screen back here to myself. OK. Sorry folks, we are trying to switch back here.
OK. Why don’t we start with the questions, and we will try to bring this up throughout the next couple of minutes. Oh, there it is. I think I see it now. OK.
The first question – and Trevor and Yoav jump in whenever you feel appropriate – is by Simon S.
“Can you explain the purpose with fake conversion page, again? Do you use the URL of Google documents for that and why?”
Trevor: Absolutely. With a fake conversion page, you can use that if your regular conversion page is not accessible. Yoav mentioned that a fix is… You won’t have to worry about this anymore. But, what you can do is if you have a conversion page, which isn’t accessible, for example, it is behind a secure page, you can use a fake conversion page to validate the experiment. As long as Websoft Magic sees a conversion script on that fake page, everything should validate. And then, when you run your actual test, when the user can actually see your conversion page, everything should work fine. The fake conversion page is just for the purpose of validating an experiment and getting everything set up.
Yoav: OK. We’ll jump over to the second question, here, by Robin S. “If you have multiple landing pages for the home page, how would you configure Magento to do this and try out AB testing?” [crosstalk]
Yoav: Go ahead, yes. Please. Please go ahead, yes.
Roy: I think, in this case, this is an AB testing engraved on basically a static page or that might have some dynamic elements in, but that is the classical AB testing. So, rather than using the home page, you would actually present a different page as an alternative to the home page. There is no multivariate testing on the home page at this time.
Yoav: That makes sense. Do you have anything else to add, Trevor?
Trevor: Just saying that you can set that up pretty easily in the match CMS area. So, you could just have the AB test running there.
Yoav: OK. Marco is asking where are all these variations stored? Another Magento page, or Website Optimizer creates those?
Trevor: The variations are stored on Google’s servers. Basically, when you create the variations in Website Optimizer, you are saving them as part of your account on Website Optimizer. When they get served, they are getting served from Google servers.
Roy: Kerry wants to know if this is integrated into Google Analytics?
Trevor: Natively, it does not integrate directly with Google Analytics. However, one of our authorized consultants has put together a modification to Google Analytics tracking script. And what this will let you do, it will let you track the different multivariate variations in your Analytics reports. I can get that link to Roy, and then he can get that link out with the Webinar as well. It’s a pretty simple switch, you just swap out your analytics tracking code for a slightly modified version, and then it all works very well.
Roy: Great. We’ll jump over to the next question. Do returning customers get the same variations, cookies or IP, or is it assigned upon entry?
Trevor: Returning customers do get the same variation. What happens is when a visitor visits an experiment page, they get an extra cookie, and that cookie http://break. So, if that experiment is running for the length of two years, they will see that same variation. And every time they visit, that cookie is reset. So, the visitor will see the same page every time they come back, unless, of course, they delete their cookies.
Roy: OK. We’ll jump over to the next question about variations. Does Magento auto-generate the page version on the go for each testing?
Trevor: This is actually what Websoft Magic does. The scripts that Magento installs from Website Optimizer allow Google Websoft Magic to swap out the different combinations. It’s really one page, it’s just these little sections, there, the headline or the image. That little bit of code, there, is being swapped out for the variations that you created. It’s the same page, just little bits are being swapped out.
Roy: OK. How do we install Google Website Optimizer within Magento?
Yoav: It should already be there. There’s three places where you can find it. Under your catalog, under managed categories, you can do category view optimization. On products, you can do a product view optimization, and in your Managed CMS, you can do a page view optimization. Those are different areas where you can use Websoft Magic [indecipherable 51:50].
Roy: OK. And again, this is available from Magento 1.17 and up, so if you’re not running the latest Magento release, we definitely recommend that you take care of upgrading as soon as you can so you can take advantage of this great functionality. “Is the feature, in Magento, for Website Optimizer an extra component, or is it in the core system?”
I’ll jump on this one. It’s part of the core system; as of Magento 1.17 and up.
Third question that we had before, let’s just skip this one. We had to answer this a few minutes ago about A/B testing and the Magento home page.
The next on is, “Are optimizations done on a per product basis? Can I test changes on all product description pages in the same experiment?”
I have the [indecipherable 52:35] on this, but you should go with it.
Trevor: So, the variations are sort on a per product basis. So, for the multivariate testing, the different alternatives – text, image, name, etc. – are actually on a per product level. You cannot generate a description for all your products at the same time. If that’s the question. If you’re looking on different layouts of the page; that’s actually something that we’re thinking of adding a bit later, and there’s another question about that, so I’ll save this answer. But, again, to answer this, it is on a per product at this time.
Roy: We’re repeating ourselves here, Magento 1.17 and up, in terms of the version that supports it. Singer has a question about…, and I’ll just summarize this. Trevor, at what point does it make sense for online merchants to start testing? What kind of traffic should they expect before it becomes relevant to their business?
Trevor: The rule of thumb that we have is you need about 100 conversions per variation for the length of the test. So, if you wanted to run an A/B test you would want to have about 200 conversions. Depending on your visits that might take you a week to get; it might take you two months to get. But if you get around that much traffic then you should be OK. Again, about 100 conversions per variation, and that’s a rule of thumb, there’s a lot of other factors that could change that. If you can get around that much traffic over the course of the test, you should be OK.
Roy: Singer isn’t talking about pages, he’s actually talking about his conversions, and that’s what we’re testing.
Trevor: Yeah, visits is one thing, conversions are the real thing here.
Roy: Right. Chris is asking if he can do more than one multivariate test on a page at the same time.
Trevor: On a single page, yes you could, but you probably wouldn’t want to. You would probably impact the results from both experiments, and the data would not be as true as you would want it. Technically, yes, but you probably don’t want to.
Roy: OK. Good question here by Jerry about using Google Website Optimizer, and let me just read this question. “Can this be used for static sites that I use to drive traffic to my Magento store, and how do I teach Website Optimizer what is a lead and what is a sale?” So, what does that mean; how do you conversion, I guess.
Trevor: For the first question; absolutely, you can use this on any website, including Magento, or what not. So, the way Web Optimizer records leads is using that conversion script. Generally, that is based on the load of a page, say a “thank you page”, like, “Thank you for joining my mailing list,” you just put that script at the bottom of that thank you page. You can also modify the script, there’s some instruction on how you can do that on the website on the support page. You can make it work on an on-click event, for example, clicking to go to the Magento store. That would work as well.
Roy: All right. This is more about newsletter testing. “I would love to test my subject lines in an email to our 20k opt-in list, but how? Subject lines are a big part of open RAID, so how can I use Google Website Optimizer for this?”
Trevor: Unfortunately, Web Optimizer can’t handle emails. There are a lot of tools that do apply in A/B testing, and there’s also a great amount of materials on what subject lines work best. I would recommend: 1) reading up, and 2) looking for a product that can handle it. But, Web Optimizer, unfortunately, is not that product.
Roy: OK. Next question, as we come to the conclusion here: “Are there not other parameters that decide conversion during a certain period of time?”
Trevor: I’m not sure I understand this question.
Roy: OK, maybe I’ll try to clarify it. Conversions are obviously impacted by much more than perhaps what we’re testing. How do you sort of balance that within an experiment?
Trevor: One easy way to balance it is just to make the period between… I guess, there’s two ways. One is when you design your experiment you want make the conversion event and the experiment event. You don’t want to put an experiment page if you’re testing your… I’m sorry. You’d want to have a lot of steps in-between your conversion event and your experiment. So, product page to check-out page is pretty good. Home page to check-out page, you’re getting a little further away a little more variants.
Secondly, you want to run your tests for a certain period of time. What will happen is Website Optimizer will do the statistics and show that there is in fact this variation that is making the difference, not just chance. So, part of that is running your test for at least a week, so you can avoid day of the week bias and things like that.
Roy: OK. We’re jumping over to…
Trevor: Yes you can.
Roy: We’re jumping over to…
Yoav: I just jumped in to listen for a certain time.
Trevor: Yes, you can test a series of pages. You may not be able to do this with Magento, just yet. Basically, you can do multi-page, multivariate testing where you put the different section scripts across. But, not at this time in Magento, but on a separate page you can do that.
Roy: OK. “Can the testing be used on a dynamic template page, such as a product page, that the variations will be shown across all products, or do you have to do one specific page at a time?” I think we answered that a few minutes ago. The answer is that you want to do this, and your able to do it on a per product basis and a per category basis. That’s what is available currently in Magento 1.17 and up.
Our final question, by Nicholas B., “Will the Optimizer integration be extended to product pages, cart, and check-out?” Yoav?
Trevor: So, it’s already on the product pages and ……
Trevor: Go ahead.
Yoav: So, definitely we have product pages, and what we are looking into doing in the near future is adding support for actually A/Biting, and maybe even multivariate testing on that. But that would probably be a bit harder. Definitely support for A/B testing on the cart and check-out, so you can actually present alternative cart pages and check-out pages for customers. This is something that we’re hoping to have in the next minor version of Magento.
We also might, in the near future, actually be able to serve different layouts off a page, which will then give you an ability to change layout off a specific product page. That will be across the whole catalog. You will be able to show one view of the product page and then have alternative views for that product page across the catalog as well.
Roy: Excellent. So, I think that’s it. Trevor, any last words from your perspective?
Trevor: I would just say, you should test that. You can try out Web Optimizer and read more about it at google.com/websiteoptimizer and through Magento. Thank you all for having me and I hope everything goes very well. I know a few of you are testing already and I’m very excited to see what happens.
Roy: Excellent. Trevor, thank you so much for taking your time this morning to join us, Yoav, thank you as well. With this, we’ll wrap up the webinar for this morning. Thanks again. If there’s any questions, of course, the community forum is a great channel. If you’re a support customer please do open up a ticket so that we can attend to your issue as soon as we can. Thanks folks, have a great day.
Yoav: Thanks everyone, bye.