Blogs about optimization from Pew Pew Laser Blog

What Testing and Optimization Can't Do.

Last time, I talked about what optimization testing can do for your site. Now, I'll talk about what it can't do.

• It can't generate new traffic for your site. But optimization testing can help you do more with the traffic that you do get.
• It can't get your web site / IT organization / content management system in order. Testing will require at least one change to the page; and optimization testing can't make a web development team more responsive.
• It can't clarify your web site's marketing objectives. It can tell you how well existing objectives are being met.

What Optimization Testing Can Do.

Optimization - testing to find out which elements drive desired visitor actions - can do a lot for your web pages.

• One of the most obvious benefits to optimization testing is to find out what your page's conversion rate is. And you can find out when that conversion rate drops or increases.
• A good optimization tool can can tell you what segments - specific niches of visitor traffic - exist within your overall traffic.
• Testing can tell you if those expensive Flash pieces are, in fact, more effective than their static counterparts. Wouldn't it be nice to justify those costs, or to know if you could get better results with something cheaper?
• Most importantly, testing will tell you what is most appealing to visitors; what content changes will drive visitors to conversion actions. How can you possibly know what your visitors like without testing your assumptions?

Ready to start? Very good. If you've got no money, Google has a free optimization tool. Of course, my company has an optimization tool too.

What I'd Test.

I've been imagining doing some optimization testing on Woot's new design. This is a bit of an intellectual exercise, and a bit of show-and-tell for designing a web page optimization test.

All tested elements should run concurrently, and for at least a week. Woot's conversion rate is probably highly dependent on what's for sale that day. If a test were run where the display of the elements were non-random (one element per day, for example), the results would be so heavily influenced by the desirability of the item for sale that the element's actual influence on conversion would be impossible to suss out. The site is highly dynamic, but even the dynamic elements can be tested using style changes and JavaScript.

I'd measure the results of 2 different actions - signing up a new user and purchasing the item. There's a good chance that the test elements would influence these events quite differently.

Above is a screen shot with an overlay of the elements I'd test for optimization. (Click for a full size screen shot) For each of these test elements, I'd test the original versus a new idea:

1. I'd try different colors on this Call to Action button. In subsequent tests, I'd test the language of the button.
2. I'd like to try a different treatment on this quick production information box. Creative styling (like what I've drawn, only good) that highlights the Call to Action button might prove effective.
3. Hiding the Discussions box as a test element will tell us whether visitors are being distracted from converting.
4. This smaller headline might prevent people from seeing the rest of content that would be more convincing.
5. Same for the larger headline.
6. I'd test hiding this advertisement box. Once it's influence on conversion rates is identified, we can look at the ROI of the advertising revenue to see if it's worth keeping the ad.
7. Lastly, I'd try moving this more straightforward and detailed description of the product near the top of the page.

All in all, that's 7 different elements to test. If we test only 2 versions of the Call to Action Button, we can do it in 8 experiments (the unique number of pages displayed) using a fractional factorial array. If Woot's traffic volume and conversion rate support 16 experiments, we can test 4 versions of the Call to Action button, as well as more versions of the other test elements, or a few completely new test elements.

300 Million Reasons to Optimize.

A recent article about web optimization and the visitor experience got a lot of attention - The $300 Million Button. It's a really exciting read - I recommend you check it out even if you're non-technical.

To summarize, by removing a "Register" button, and adding reassuring "you can just buy" language, the e-commerce site increased their conversions by 45%. (Forty five percent is huge - 300,000,000 dollars is ginormous.

One lesson I see in these results is that if you test big things, you can see big results. It probably took a ton of work to set up business rules and back-end support so that visitors could simply buy an item, without signing up. This effort paid off. The results of the testing are just another example of users despising the sign up. Think about it - do you need another password to remember?

(I suspect that the "major e-commerce site" is Barnes and Noble. I received the "you don't have to sign up" experience when I was shopping recently - and it was quite influential. It convinced me to just buy the item I was looking for, instead of hunting around for some easier way.)

Jared Spool, the usability tester and article author, is rather famous within the web design industry. I really appreciate Spool's sharing of his results, and the excitement that this article are bringing to the optimization industry. Both Spool and Luke Wroblewski will be speaking at a conference in Seattle in spring. I hope I get to go.

Re-Woot.

A couple of weeks ago, Woot launched a re-design of their website. Woot is known for selling just one thing a day - usually a deeply discounted electronics gadget. Nearly all e-commerce sites live or die by conversion rates - the percentage of visitors who purchase. I would expect that Woot is probably no exception. So I wonder if the new site design has had an effect on their conversions.

Above is a screenshot of Woot's previous design. (Full disclosure - I had to combine a couple of images from the Wayback Machine to get that exact image, but what you see is a fair representation.) If I'm interpreting the results from the Internet Archive's Wayback Machine correctly, this design had been in service since March of 2005. This image is a slightly compressed version of what's visible in my browser window; you can click the image to see it at full-size.

One thing about that page probably jumps right out at you - the "I Want One" button. It's big, it's orange, and it's surrounded by white space. It's friendly-looking, and you probably want to click it - just to see what will happen next. That's the call to action button - and it's the cirtical first phase of the "buying something" funnel.

Above is a screenshot from January 22nd, 2009 showing Woot's new design. Again, this represents what's visible in my browser window, and clicking will open a full-size screen shot. The new design certainly is slicker and more modern looking. I notice some detail-work attempting to highlight important parts of the site. But I would bet that the Discussions and the Ads on the page are detracting from the main content of the page.

Most importantly, the "I Want One" button seems to stick out less than with the old design, and I find myself less compelled to click it. In fact, I think the Discussions box is drawing a lot of attention away from "I Want One". As the eye reads left to right, it naturally slides right over the button and on to Discussions.

As a sidenote, it really bothers me that this new design doesn't fit in my browser window. You see, I do have a fancy-pants wide-screen monitor, but I don't let my web browser use all of that real estate. Firefox runs in a smaller window so that I can see other stuff beside it - important stuff like Dreamweaver, my Palm Desktop, and File Explorer. Each web page gets only 950 pixels of width, and 855 pixels of height before the scrollbars appear. The new Woot design demands 1096 pixels of width; any smaller than that, and out comes the hated horizontal scrollbars. Listen up web designers - even if my monitor is capable of displaying a certain resolution, you're not going to get all of it. Consider that even if a visitor runs their browser at full screen, you've got all of chrome and sidebars getting dibs on those pixels. (The bookmarks sidebar is very popular in some demographics.) I know that most statistics report screen resolution (Google Analytics, for sure), but I really wish they'd report available viewport size instead.

So, here's what I'd like to know? Did Woot test before they redesigned? The question isn't "Does page layout and design affect conversion rates?", but "How much?" and "Am I losing customers?".