Archive for April, 2009
Tuesday, April 28th, 2009
Over the years Dan and I have built a lot of technology products and have internalized the product development process. Fundamentals for Founders is our first book, and while writing it we often joked about how it felt so much like shipping software again. No matter how different common perceptions might be about the types of people who become writers and engineers, we’ve noticed that these two professions share a lot of similarities. With hindsight it’s not that surprising: both are building a product, so many of the same principles apply. We thought of our book-writing process in terms of the following stages:

Looks a lot like an engineering project, doesn’t it? Here’s a breakdown of some of the shared points:
Project Management. Both writers and product teams need to trade off three dimensions:
- feature set
- quality
- schedule
For writers in particular, those dimensions are manifested as:
- the breadth and depth of content
- quality of the writing style
- timing of the book release
As they work on their book, they’re making decisions where they need to balance those three aspects. For example, we faced questions like: Should we add an extra section to this chapter? Is it worth our time to do another set of revisions on this excerpt, or is it good enough? This piece of content doesn’t meet our quality bar — should we fix it up, or remove it? Making those calls is also the daily job of product owners at technology companies — so we felt right at home.
Quality Bar. How do you decide if a piece of writing is good enough to sell to your readers? We established a couple of criteria for our book:
- no clichés (we’re allergic)
- The “Theron” test. My friend Theron is a super-smart engineer who recently quit his corporate job, so he fits perfectly into our target market of people looking for some guidance on how to start a business. For sections not directly related to product development (Theron’s home-field), we’d ask “Would Theron find this useful?” If not, cut it.
- The “Ripan” test. Our mutual friend Ripan is one of the smartest guys around and has a finance background. For anything outside of finance, the Ripan-Test asked “would we feel comfortable telling this to Ripan’s face over dinner?” That was a helpful bar not just for keeping an appropriate depth of content, but also for keeping our writing style fresh and targeted at a demanding reader.
- Last but not least, the “would we want to pay for this ourselves?” test, a good bar for keeping our content useful.
Keep Raising the Threshold for Accepting Changes. Fixing a bug in a software product always carries both the cost of having to re-test, as well as the risk of introducing new bugs (called regressions). It’s the same with a book. As you edit your writing, you’ll need to then recheck the whole section for consistency, and you’re running the risk that your changes have unintended consequences like changing meanings, removing context that’s required elsewhere, etc. So as you’re nearing the end of your project, you’ll need to keep raising that quality bar. When in doubt, err on the side of minimizing changes.
Write Ship-Quality From the Beginning. I started writing my parts of this book at a hostel in Taiwan and was having breakfast with a local newspaper reporter. Being new to writing myself, I asked her whether she goes for quantity first when writing a piece, or whether she polishes each sentence right from the start. Definitely the latter, she said, and apparently most other professional writers do the same. It took some time for that to sink in for me. At first I preferred just sitting down and writing from a stream of consciousness. It felt so productive to churn out thousands of words in a day! But we’d find out later that my high-quantity but unpolished content would take more time to clean up and rewrite than had I simply written it more carefully from the beginning.
In software terms, fixing the bugs took more time than writing the code in the first place. (That’s one reason for fascinating research results showing that software teams working 40 hours per week sometimes produce more work and at better quality than teams working longer hours. The over-worked teams write bad quality code that takes disproportionately more time to fix up.)
Lastly, an off-beat example about product quality that might serve as food for thought and discussion: I’ve been terrorized by my electric toothbrush recently. It turns itself on in the middle of the night, wakes me up, and then the OFF button does’t work. (Why does a toothbrush wake me up despite a wall and two doors being in the middle? I live in China and suspect that the toothbrush heads I purchased are fake… they make an enormous amount of noise when spun by my Oral B.)
So what to do? It’s 3am, you’re holding your vibrating toothbrush, and it won’t turn off? It’s an iPhone-style toothbrush — can’t remove the battery. Tossing it out the window isn’t an option. The solution: wrap it in something soft, then seal it in the washing machine. Nice and quiet. Charge it again during the day, then stuff TB back into the washing machine before you go to bed. What does this teach us about product development and keeping a high quality bar? Should Oral B have spent more time testing this toothbrush? Or generalized: faced with a choice to fix only one bug, which one would you pick: a low-severity issue that many people will see early on (such as an imperfection in the coloring), or a high-severity issue that only a few people will see and only after years of happily using the product (like the tooth-brush alarm-clock bug I ran into)?
Tags: Product Development, project management Posted in Product Development | 4 Comments »
 |
Monday, April 20th, 2009
When you track the impressions and click performance of a given page, the observed conversion rate might not accurately reflect the actual conversion rate. How do you know the unusually high number of clicks you measured might not have simply been a fluke or coincidence?
Confidence means different things to different people. Often, the consequences of an incorrect conclusion determines what level of confidence is required for a particular test. For instance, the military might require a 99% confidence level in a very tight confidence interval for weapons testing - when a cruise missile is launched at an enemy target, the commander wants to know with certainty that it won’t miss and end up hurting innocents or allies.
You’ll have to decide for yourself what level of confidence you’re comfortable with, but for website testing for a startup or small business, we believe an 85% confidence in a +/- 25% confidence interval is sufficient. That is to say, if we had observed a conversion rate of 10%, we want to know there’s at least an 85% chance that the actual conversion rate was 10% +/- 2.5% or somewhere between 7.5% and 12.5%.
Let’s walk through the math. As an example, let’s assume you’ve observed 300 impressions and 30 clicks for a conversion rate of 10%.
First, determine the Standard Deviation:
stdev = sqrt((p(1-p)/n)
stdev: standard deviation
sqrt: square root
p: observed conversion rate
n: # of impressions
stdev = sqrt((.1(1-.1)/300)) = .173 or 1.73%
To determine confidence, we want to determine the likelihood that the actual conversion rate is LESS THAN 12.5% and GREATER THAN 7.5%. To do so, we’ll effectively map the data to a normal distribution curve using a technique called a Z Transformation:
Z = (X-p)/stdev
Z: statistical variable used in further calculation
X: confidence (upper or lower) limit
p: observed conversion rate
stdev: standard deviation
Zu = (.125-.1)/.173 = 1.44 for the UPPER limit
Zl = (.075-.1)/.173 = -1.44 for the LOWER limit
In a statistics class, we would map out the distribution curve and find the area underneath the curve to determine the confidence level. But you’ll save a TON of time by using Excel’s NORMSDIST() function. In a spreadsheet, run the following formula:
conlvl = NORMSDIST(Zu)-NORMSDIST(Zl)
conlvl: confidence level
Zu: Z-value of upper limit
Zl: Z-value of lower limit
conlvl = NORMSDIST(1.44)-NORMSDIST(-1.44) = .851 or 85.1%
Now you can start to see why >30 conversions is a rule of thumb. As you adjust the inputs into the formulas to fit your needs, you’ll notice a couple of general trends.
- More conversions are required for higher confidence levels or tighter confidence intervals.
- Smaller conversion rates require a more sensitive test to detect than larger conversion rates and therefore need more data to achieve the same confidence.
In Fundamentals for Founders, we discuss statistical significance for website optimization in greater detail. When you purchase the ebook, you also get access to a tool that calculates confidence levels based on your impressions and conversions. The same tool can also be used to determine confidence levels for A/B split testing.
Posted in Advertising, Statistics | 4 Comments »
 |
Wednesday, April 15th, 2009
A hot new feature in the online advertising world: retargeting. It’s about showing ads specifically to people who have visited your website before. For example, someone comes to your site and views certain key pages, indicating some interest, but then doesn’t place any orders. Maybe he got distracted, had some unanswered questions, or didn’t need your offering quite yet, but maybe later. This person is now a very qualified sales lead. “Retargeting” features allow you to show ads to that person as he browses other websites.
For example, the next time he browses his favorite online magazine or news site, he may see an ad from you inviting him to come back. You know what he did the previous times he was on your site, so this ad can be very specific, maybe even include special offers to get this person to come back. Click-through and conversion rates are very likely to be much higher than for ads shown to people who’ve never been to your site.
How does this work? You partner up with a retargeting network such as fetchback.com (an aggregator of various ad networks that specializes on retargeting). As a visitor comes to your site, you’ll place a cookie that will now identify him as he browses other websites that belong to the ad network.
How is this different from advertising on Google Adwords or classic ad networks? You don’t need to specify targeting criteria. You target anyone who’s been to your site before. No need to filter it any further.
How is it priced? This depends on the ad network, but flat-rate pricing seems to be most common for relatively low-traffic sites. For example, we’ve seen introductory offers for sites with less than 20K unique visitors per month where $500 for at least 3 months will show ads to each of your <20K visitors as they browse other sites. Your actual CPM/CPC costs will vary based on the number of visitors and their browsing behavior once they leave your site. Ask your ad network what % of your visitors they expect to be able to cover (usually 50-70%).
When does this make sense to use? You’re investing a flat-fee per month in return for an increase in your conversion rate, so you should calculate what % increase in your conversion rate you need in order to justify that investment. The two most critical assumptions in those calculations are: 1) the percentage of your existing orders that come from return-visitors (depends on the business, we’re assuming 20% below), and 2) the percent increase to that rate delivered by retargeting. For the latter: Retargeting networks often see those go up by 50% or more. Let’s do the math, comparing 3 different scenarios with 3 different increases in the return conversion rate:
|
Scenario 1
|
Scenario 2
|
Scenario 3
|
|
Monthly Unique Visitors
|
10,000
|
10,000
|
10,000
|
|
Conversion %
|
2%
|
2%
|
2%
|
|
Total Conversions
|
200
|
200
|
200
|
|
% of conversions from return visitors
|
20%
|
20%
|
20%
|
|
Total Return Conversions
|
40
|
40
|
40
|
|
Average Order Profit
|
$100
|
$100
|
$100
|
|
Retargeting Increase in Return Conversions
|
10%
|
25%
|
50%
|
|
Conversions due to Retargeting
|
4
|
10
|
20
|
|
Retargeting Value
|
$400
|
$1,000
|
$2,000
|
|
Retargeting Cost
|
$500
|
$500
|
$500
|
|
ROI
|
($100)
|
$500
|
$1,500
|
We see now that an increase of 15% or above would be profitable for this business, and most ad networks will confidently exceed that number. So for the example above, retargeting is definitely something worth trying. Here is the spreadsheet containing the above calculations. Feel free to adapt this for your own purposes.
Posted in Advertising | 4 Comments »
 |
Tuesday, April 14th, 2009
Engaged in online advertising, or think you will soon? If you haven’t played with Facebook ads before, here’s a quick walk-through.
We’ll make this a practical example, where the goal of this exercise is to get more people to read this very article. We’ll start by targeting people who may be interested in reading this and try to get them to subscribe to our blog or even link to it.
Let’s brainstorm a list of characteristics for those people: They’re based in the US, are at least 22 years old, have graduated college, and have expressed some kind of interest in entrepreneurship. Even better: maybe they have their own blogs, and maybe they’ll be kind enough to link to us after reading this post. We created an ad with those targeting criteria and pointed it to this article just as we posted it. Here’s a video showing how we did it:
As a side note to all those who read this after clicking on the Facebook ad: found this slightly amusing or interesting? Feel free to subscribe on the right. Also, we know you enjoy blogging – how about a link from your blog to ours, done like this?
Hopefully we’ve shown how easy it is to get started and experiment with Facebook ads. Coming later: a post about measuring the effectiveness of an advertisement. Let’s see how to tweak the data around how many new subscribers we got based on this campaign.
Posted in Advertising | 5 Comments »
 |
Tuesday, April 14th, 2009
We’d like to start our blog series with some encouragement for all those who have been thinking about starting a business of their own yet keep putting it off. Perhaps this is due to concerns about costs and risk, or maybe questions regarding where to start or how to build more than just the product.
Our book is meant to address this. We strive to remove uncertainty and help turn paralysis into action by showing the different pieces that need to fall into place, and providing examples of specific tactics to get started for each of them. We think many readers will be surprised to see that, for many ideas, there exist fast and inexpensive ways to prototype a product or service, do some market tests and get real data to convince yourself of the revenue potential. Armed with that experience it’ll then be a much easier decision to throw your full efforts behind building the business.
Tim Ferriss’ Four Hour Work Week opened many eyes to the world of online market testing. Before you even have a product ready to sell, advertise your envisioned offering to your target market, then measure how many people are willing to buy. Your test subjects will see an “out of stock” message or an equivalent. Techniques like this can quickly give you hard data on the market potential for a new product or service, and creating the experiment can often be surprisingly quick and cheap.
For example, many such tests require only two primary components: a professionally designed website that pitches your offer, and an advertising campaign to drive a few hundred to a thousand pre-qualified visitors to the site. Each component can cost as little as $500, for a total of just $1000 over the course of 1-2 months to get real data and practical experience upon which to build your business.
Read our book to learn about how to setup such experiments and make sound decisions based on them. We cover many more techniques for getting started: Doing smart market research, testing sales potential, designing great products with the end-user in mind, working with outsourcing marketplaces to quickly get products developed, making data-based decisions with a financial model, etc. We also include spreadsheets that illustrate the examples used in the book and can be adapted for your own business.
Give it a read – we think you’ll like it.
Posted in Motivation | No Comments »
Friday, April 10th, 2009
At long last, we’ve launched! We’ve labored for months and months over this book, and believe we’ve packed a ton of insight and hands-on advice for anyone looking to start a business. Like any successful new business, we’re craving customer feedback, so feel free to share ANY thoughts you might have - about the book, the website, even our haircuts.
We’ve got a great series of posts planned for this blog. We’ll share insights on a range of topics relevant for entrepreneurs. Hit one of the subscribe buttons on the right and follow us!
Posted in Uncategorized | No Comments »
|
| |
|
|
|