Optimizely and jQuery – The new A/B Testing Platform

Being that I work in the eCommerce space, we talk a lot about A/B testing and it has also become a huge topic of conversation on the internet/twitter/facebook. Over the past two years, there have been more tools emerging that seem to blow away the previous competition. One tool that seems to be really pushing the boundaries is called Optimizely, which I have had the pleasure of trying out this weekend.

To give a little background on this company, it was started by Dan Siroker and Pete Koomen, both were product managers at Google. Dan Siroker eventually left Google to work on the Obama 2008 campaign as the Director of Analytics, where he worked on A/B and multivariate testing to increase signups, donations for the Obama website. The optimization was hard because of the amount of IT involvement that was required each time a campaign needed to be setup. I’ve experienced this too, you have a great idea to test – but the implementation can take 4 hours, it makes you rethink what to actually test, which shouldn’t be the case.

One more very interesting fact to mention is that these guys just received $23 million in Series A funding and they are growing at 400% year over year. This company is definitely headed somewhere interesting, excited to see what becomes of this platform. This is a company to keep a close eye on over the next year!

Account Setup

Creating an account on Optimizely was dead easy, I signed up for the cheapest plan ($19/month) to try some testing out on one of my websites (stamfordctguide.com). Within about 15 minutes, I had a live testing running, which to me is incredible after having experience with other platforms such as Performable and SiteSpect.

Optimizely Implementation and Peformance

The account I signed up for only includes 1 project, but each project can have unlimited experiments. The implementation is as easy as installing a JavaScript include in the head of your page once! One script for the whole website, it only has to be done once, I can’t say that enough. Optimizely uses a CDN (Akamai) to serve up it’s JavaScript snippet. If you look in the top right corner of the project that you are working on there is a box that shows the total download size of the JS snippet that will live on your site (see screenshot below).

Optimizely Account

This reminds me of the Typekit.com implementation, where they add all of your fonts to a JavaScript file which is included in your site, but you are shown the total size of the include. This will fluctuate depending on the size and amount of different fonts that you add. For Optimizely, the size of the library will grow depending on the amount of tests you are running and if you are including jQuery, the table below summarizes the difference in the size of the library. It wont be a big performance hit and the CDN will help with optimize the delivery to the end-user.

Project running 4 tests with jQuery included Project running 4 tests without jQuery
41kb 19kb

If you are curious about uptime and response time of the snippet, Optimizely publishes all of this information using Pingdom for the current day and also a past history. I love how open this company is about their software and performance!

Experiment Setup

This is the part of the software that I LOVE, LOVE, LOVE. There are two ways to setup experiments:

  • A firebug style inspector tool that highlights each element, then thru a contextual menu on the screen you can edit, move, resize, swap, etc. This will be easy for anyone to setup experiments.
  • A code editor that expects/accepts jQuery to write all of your experiments. Anytime you use the firebug-style inspector tool, it is actually writing jQuery behind-the-scenes. That’s MINT! It really puts the power of creating A/B experiments in the hands of the front-end developers and away from IT.

Support

The product support is excellent, they have created a great knowledgebase where you can view questions on setup, configuration of testing, reports, customization and you can also ask your own public questions. It’s pretty easy to find the answer you are looking for either via searching the existing questions or by posting a public question.

Video Walk-Thru

Optimizely has been publishing a lot of great video content on setup, etc. through their Youtube channel. If you are looking for a good intro, please check out the video below.

I will be writing more tutorials around setting up different types of tests using jQuery, so check back often.

Leave a Reply