When you track the impressions and click performance of a given page, the observed conversion rate might not accurately reflect the actual conversion rate. How do you know the unusually high number of clicks you measured might not have simply been a fluke or coincidence?

Confidence means different things to different people. Often, the consequences of an incorrect conclusion determines what level of confidence is required for a particular test. For instance, the military might require a 99% confidence level in a very tight confidence interval for weapons testing - when a cruise missile is launched at an enemy target, the commander wants to know with certainty that it won’t miss and end up hurting innocents or allies.

You’ll have to decide for yourself what level of confidence you’re comfortable with, but for website testing for a startup or small business, we believe an 85% confidence in a +/- 25% confidence interval is sufficient.  That is to say, if we had observed a conversion rate of 10%, we want to know there’s at least an 85% chance that the actual conversion rate was 10% +/- 2.5% or somewhere between 7.5% and 12.5%.

Let’s walk through the math.  As an example, let’s assume you’ve observed 300 impressions and 30 clicks for a conversion rate of 10%.

First, determine the Standard Deviation:

stdev = sqrt((p(1-p)/n)

stdev: standard deviation
sqrt: square root
p: observed conversion rate
n: # of impressions

stdev = sqrt((.1(1-.1)/300)) = .173 or 1.73%

To determine confidence, we want to determine the likelihood that the actual conversion rate is LESS THAN 12.5% and GREATER THAN 7.5%. To do so, we’ll effectively map the data to a normal distribution curve using a technique called a Z Transformation:

Z = (X-p)/stdev

Z: statistical variable used in further calculation
X: confidence (upper or lower) limit
p: observed conversion rate
stdev: standard deviation

Zu = (.125-.1)/.173 = 1.44 for the UPPER limit

Zl = (.075-.1)/.173 = -1.44 for the LOWER limit

In a statistics class, we would map out the distribution curve and find the area underneath the curve to determine the confidence level. But you’ll save a TON of time by using Excel’s NORMSDIST() function. In a spreadsheet, run the following formula:

conlvl = NORMSDIST(Zu)-NORMSDIST(Zl)

conlvl: confidence level
Zu: Z-value of upper limit
Zl: Z-value of lower limit

conlvl = NORMSDIST(1.44)-NORMSDIST(-1.44) = .851 or 85.1%

Now you can start to see why >30 conversions is a rule of thumb. As you adjust the inputs into the formulas to fit your needs, you’ll notice a couple of general trends.

• More conversions are required for higher confidence levels or tighter confidence intervals.
• Smaller conversion rates require a more sensitive test to detect than larger conversion rates and therefore need more data to achieve the same confidence.

In Fundamentals for Founders, we discuss statistical significance for website optimization in greater detail. When you purchase the ebook, you also get access to a tool that calculates confidence levels based on your impressions and conversions.  The same tool can also be used to determine confidence levels for A/B split testing.

This entry was posted on Monday, April 20th, 2009 at 9:16 am and is filed under Advertising, Statistics. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.