Hidden costs of a/b testing 

There's plenty of literature on the merits of A/B testing, but little information on its pitfalls. Most people in ecommerce, tech and startups would have you believe A/B testing is a must do activity when, in reality, it's often unnecessary and counterproductive. 

At Sticker Mule, we recently stopped actively A/B testing and I thought it'd be helpful to explain why. I should preface this by admitting we ran more than 70 conversion tests before making this decision and had only 3 significant winners. I don't know a normal ratio of tests to wins, but I suspect our situation is common. This isn't to say we don't believe in constantly working to improve conversion--we've found different ways, which I detail later. 

![The Hidden Costs of AB Testing](https://storage.googleapis.com/sm-content/blog/64feff7f772f6956dd6752f42069c17b12f845f0)

## The 6 hidden costs of A/B testing

### 1) Time
Resources are finite. Time spent on A/B testing cannot be spent elsewhere. How you use your time determines your growth trajectory. A/B testing is a growth tactic, but it's not always the best use of your resources. 

### 2) Lost conversions
Most conversion tests fail and underperforming variations cause you to lose conversions while they run. These lost conversions are an investment you make to improve your long term conversion rate, but don't forget they are a real cost. 

### 3) Performance
Related to lost conversions, A/B testing tools make your site slower and this also reduces your conversion rate--even for the control. The additional conversions you lose because of this performance hit are another, often ignored, cost of A/B testing. 

### 4) Confusion
A/B tests potentially confuse your team and customers. Customer service may struggle to properly service a customer if they don't know which variation the customer is viewing. You can educate customer service about the various tests you're running, but, regardless, you're making their job more difficult. 
 
### 5) Speed
A/B testing slows down your organization's decision making. Some ideas are obviously good and adhering to a strict A/B testing process reduces your time to go live. Time is finite and the number of improvements you implement per year has a major impact on your growth trajectory. 

### 6) Focus
Usually, organizations congratulate themselves for investing heavily into conversion. This is evidenced by a Head of CRO or paying consultants. The largest cost is not the salary of those persons, but changing the focus of other team members such as designers, customer support, and developers. The success of a Head of CRO is usually at the cost of making other colleagues less successful in their primary roles. 

## How we improve conversion without A/B testing
Stepping away from A/B testing does not mean we have abandoned efforts to improve conversion. Rather we are improving our conversion through other methods (yes, A/B testing isn't your only option).

You can monitor your long term conversion trend rather easily via Google Analytics to gauge if you're making good decisions. And you can add more precision to this method of conversion rate monitoring by looking at your conversion rate trend by channel. 

With that in mind, we're doubling down on the following to improve our conversion rate:

### Usability principles
Adhering to thoroughly researched [usability principles](https://www.nngroup.com/) generally yields good conversion optimization results. 

### Design principles
In addition to usability principles, there are basic [design principles](http://www.amazon.com/Universal-Principles-Design-Revised-Updated/dp/1592535879/) that tend to improve conversion when followed. 

### The golden rule
"Do unto others..." is a guiding principle for most of our decision making and it works great with regards to conversion optimization.

### Solving problems
Solving actual pain points our customers experience is the primary way we've been able to improve our conversion rate and strengthen our brand. 

## We haven't abandoned A/B testing entirely
I should admit that we still A/B test some ideas. Rather than adhering to a strict A/B testing process, we let our team to A/B test ideas they're curious about. 

To minimize the likelihood of testing too much, we turned off our 3rd party A/B testing tool and integrated split testing into our code base. That means, in order to run a test, we need help from development to set it up. This extra effort discourages running careless tests and has the added benefit of eliminating the performance hit that comes with using a 3rd party tool. 

Finally, we're now able to A/B test entirely different versions of our code which is much more interesting than merely testing basic design changes via a 3rd party tool.

If our approach of improving conversion rate by solving problems rather than using an A/B testing tool appeals to you, you share some of [the characteristics we look for in team members](https://www.stickermule.com/blog/characteristics-we-look-for-in-team-members). 

Feel like you'd fit in well with our team? [Apply to join us](https://www.stickermule.com/careers). We'd love to hear from you.

A/B testing has multiple hidden costs. We decided to mostly stop a/b testing for 6 reasons. 

Prerequisites for rapid growth 

![Illustration of growing plants](https://cl.ly/2V3z1C1Z2k0f/prereq_blog.jpg)

To date, we’ve been fortunate to grow by focusing on our passion for manufacturing, design, software development and customer experience. This attitude let us grow for 5 years without a single person in marketing. Last year we started building a marketing team and noticed our growth curve bend.

> Our team is still small and, while there are obvious benefits to staying that way, there’s also long term risk associating with ignoring opportunities to grow faster.

Our team is still small and, while there are obvious benefits to staying that way, there’s also long term risk associated with ignoring opportunities to grow faster. Without outside investors forcing growth on us, we’ve been able to create an environment where talented people seem to enjoy working (our [eNPS](http://community.netigate.net/employee-net-promoter-score-how-engaged-are-your-employees-11861.html) is [90](http://cl.ly/2U2q3f3Q1J23)!). If we accelerate growth, it’s important that we don’t diminish the environment & culture that enables us to do great work.

Given our hesitation to make changes that could negatively impact our culture, we identified a few areas where we need to improve to grow faster. As we start 2016, to grow faster, we need to:


### Prioritize effectively organization-wide
Our future is determined by the projects we pursue. Time spent doing less valuable activities is not spent doing more valuable ones. The better we get at prioritization the faster we will grow.

### Minimize average time to go live
We spend resources upon starting a project and don’t get a return until we go live. It’s wasteful for good ideas to sit idle and worse to invest time in a project that get’s shelved or stalled.

### Learn to scale job responsibilities
New responsibilities accumulate quickly in fast growing organizations. We need to be flexible when it’s necessary to adopt new responsibilities or adapt to a new role.

### Identify, recruit and fill key positions
We can’t do anything without the right people. So our potential is constrained unless we identify significant roles and fill them with the best people.

### Develop an effective hiring processes
To hire at scale we need exceptional recruiting, hiring and onboarding processes and a commitment to caring about the success of new people. Productivity and morale deteriorate when companies hire without a plan.

### Raise standards
We must be committed to mastering new skills while continuing to improve where we already excel. People learn from each other and a deterioration in standards can quickly spread as we grow.

## Concluding thoughts
While these principles are particularly relevant to our current situation I suspect they’ll continue to matter indefinitely. Prioritization and execution speed are easily the two biggest determinants of long run growth.

Beyond that, it comes down to filling the right positions with the best people, enabling them to succeed and encouraging everyone to stay flexible so that the organization can quickly adapt to change. Finally, a commitment to constantly raising standards is perhaps the best safeguard to the decay the most organizations experience as they age.

If you agree with these principles and are seeking an organization where talented people enjoy working, [we are hiring](https://www.stickermule.com/careers?ref=blog).

We've identified a few areas where Sticker Mule (and other companies) need to change or improve in order to grow faster. 

The Sticker Mule guide to Asana 

![The Sticker Mule Guide to Asana](https://s3.amazonaws.com/assets.stickermule.com/blog/2016-02-12-the-remote-team-guide-to-asana/sticker-mule-guide-to-asana.png "The Sticker Mule Guide to Asana")

[Asana](https://asana.com/) is Sticker Mule's task management platform. We use it to create, track, assign, discuss and complete every task that isn't managed by a specialized system. Asana is flexible and feature rich which makes it intimidating at first. There's no definite way to use Asana correctly. To get full value requires upfront effort and a willingness to agree on conventions to standardize use.

We wrote this guide to explain how we use Asana at Sticker Mule. We designed it as a reference for our team and have published it for other teams that may find it useful.

##How we organize tasks in Asana
Asana encourages you to create teams of people that manage projects comprised of tasks and subtasks. We generally do not use it in this manner because most of our projects are small. It doesn't make sense for us to create hundreds of projects so we redefined Asana's conventions to categorize our work.

> It doesn't make sense for us to create hundreds of projects so we redefined Asana's conventions to categorize our work.

We use teams to create top level categories and projects to create subcategories. A task within a project (i.e. subcategory) can be a discrete task or a project that's organized using subtasks. Occasionally, we use Asana projects to track actual projects too. Our top level categories, which are setup as teams, are:

### Product design
The product design team categorizes improvements to our product (i.e. web site & related apps) by placing them in categories (via projects) for each functional area (i.e. cart, checkout, reorders, reviews, etc.). 

Periodically, we prioritize tasks within each functional area (i.e. project) using section headings to sort tasks by priority. When we start work on a particular area, we can quickly scan a list of related improvements sorted by priority. For example, if a developer is working on the cart, they may look at the cart project to quickly grab a few extra tasks to complete.

### Development
The development team categorizes development work by creating projects to track: defects, rough edges, high priority tasks and our roadmap. The roadmap provides a high level overview of the main improvements we intend to make. 

We also stay focused by using a "rough edges" project to categorize non-critical improvements that won't be noticeable by most users. Most "rough edges" are rather easy fixes and our team will tackle a few in between pursuing more substantial projects. 

Finally, we will sometimes create projects to track actual projects our dev team is pursuing if they are substantial in size.

### Operations
The operations team organizes work that's related to operational improvements. Since most operations projects are substantial, we use Asana projects within this team to track actual projects. The projects are roughly sorted by priority from top to bottom so anyone can monitor them. 

Additionally, we have a catch all project called "Ops ideas" that tracks ideas that may lead to operational improvements. Anyone is free to post an idea to the "Ops ideas" project. If a task within that project is green-lit, and seems substantial, we may convert it to an actual project.

### Marketing
The marketing team includes projects that categorize work by functional area (i.e. email, advertising, content, conversion, launches, etc.). Within each project we use section headings to sort tasks by priority. Additionally, we have a "Marketing ideas" that tracks tasks which don't fit into an existing category. 

Ideally, one person would own each area, but our team is still small so most people oversee a few areas.

### Customer Service
The customer service team includes a "CS ideas" project that tracks tasks that lead to customer service improvements, a "FAQs" project that tracks FAQs we need to create or revise and a "Rush" project that tracks orders which must ship early. 

Anything posted to the "CS ideas" project is intended to have 2 specific results: improve customer service efficiency or reduce the need for customers to request help.

### Human Resources
The human resources team includes a "Positions" project that tracks all open & potential positions. We use subtasks within each position to track additional tasks that need to be performed to fill the position (i.e. post ads, create job description, etc). 

We also have an "Onboarding & Training" project where we create tasks to represent each new hire and subtasks that reflect the onboarding and training tasks for that person.

## Asana tips & tricks

### Task naming conventions
1. Start task names with verbs (i.e. Consider, Send, Fix, Add, Design, Implement, Order)
2. Use sentence case. It's Easier To Read.
3. Be concise.

#### Example task names:
1. Consider using Google Tag Manager
2. Redesign invoice PDF
3. Design custom labels early access landing page
4. Implement custom labels early access landing page
5. Fix issue with address validator presenting identical addresses
6. Send email announcing custom labels
7. Order new chairs for the office.

### How to write a good task description
1. Use headings to give structure to a task description (CMD + B to bold)
2. Include screenshots & screencasts whenever possible using [Cloudapp](https://www.getcloudapp.com/)
3. Be concise and comprehensive. Needless words waste an assignee's time but so does making them ask for clarification.

### Turn off email
To speed up Asana adoption we encourage everyone to turn off email notifications to get used to relying on the Asana inbox.

### Assign tasks during onboarding
We give everyone Asana tasks as part of their onboarding so they start using the platform immediately. Example onboarding tasks we'll assign include "Place a free order using store credit", "Review and sign company handbook", and "Turn off email notifications in Asana".

> We give everyone Asana tasks as part of their onboarding so they start using the platform immediately.

### Use today, upcoming and later
Asana lets you organize your personal task list by putting tasks in either today (Tab+Y), upcoming (Tab+U) or later (Tab+L). Generally, we put current tasks in "today" even if they're not to be done today and use "upcoming" to track pending tasks. "Later" tracks tasks we might do.

### Projects aren't required
It's helpful to categorize work using projects but don't agonize trying to find the correct project for every task. The important thing is to create tasks, assign them to the proper person and notify the relevant people using the followers feature.

### Tasks can belong to multiple projects
We often put tasks in multiple projects. Most team members only care about a limited set of projects (i.e. task categories). To better coordinate, we sometimes categorize task in multiple projects so that relevant people can quickly view them. For example, a task could be in the "CS ideas" and "Checkout" project if it's both a benefit to our CS team and related to our checkout.

### Use Tab+Q
We encourage everyone to create tasks using the "Tab + Q" keyboard shortcut. It speeds up the task creation and helps you quickly put all of your tasks in Asana.

### Encourage task creation via other channels
We encourage each other to create tasks Asana when they're identified via another channel (i.e. in person or Slack). Usually, this is as simple as saying "Can you make a task in Asana and assign it to (me, yourself, another person)". This is especially important with new team members so that using Asana becomes habit forming quickly.

### Copy & paste templates
Asana's [templates feature](https://asana.com/guide/team/advanced/save-time-in-asana) is imperfect so we maintain our templates in a Google Doc. Often we'll post templates to a project description for quick access. For example, our "Onboarding and training" project includes a template of common onboarding tasks. A team members can copy & paste this list of tasks to create subtasks that need to be completed to onboard a new hire.

Hopefully Asana builds a real templating engine to make it easy to manage lists of re-usable tasks that can be added as subtasks to existing tasks or as tasks to projects.

> It's common for new team members to embrace Asana reluctantly only to become emphatic proponents within a few weeks.

## Concluding remarks
Since Asana is more complex than other task management platforms, it's common for new team members to embrace it reluctantly only to become emphatic proponents within a few weeks. Without Asana, and our conventions for using it, we wouldn't be able to gracefully coordinate and grow a remote team. 

While Asana is a big help, it can just as easily become a big mess without care to keep it well organized. As we continue to grow and launch new products (like [custom magnets](https://www.stickermule.com/products/custom-magnets)), our documentation and conventions will continue evolving. 

Our team works extraordinarily well together not just because we share the same tools, but because we share a similar perspective on work. We all tend to be [humble, independent, logical, thoughtful and autodidactic](https://www.stickermule.com/blog/characteristics-we-look-for-in-team-members). 

If you have these characteristics too, [we're hiring](https://www.stickermule.com/careers?ref=blog).


Although there's no definite way to use Asana correctly, Sticker Mule shows how it organizes and grows its remote team with the platform. 

Sticker Mule goes to Italy

Today we officially launched [Sticker Mule Italy](https://www.stickermule.com/it) so it seemed fitting to explain how we got here. 5 years ago when we started Sticker Mule our vision didn’t reach beyond the United States. International business is complicated and we were a small startup. We thought it best to focus on growing at home and set out telling everyone we could about Sticker Mule. 

Our background made us inclined to engage with designers and developers so we tried our best to befriend these communities. It didn't take long to realize that that the Internet is international and many of the best developers and designers live outside the United States. 

To properly engage these communities we needed to ship internationally at a minimum. So, early on, we contacted UPS to get rates, added international ship methods to our web site and naively thought we were done. Shortly thereafter, the complaints came in: "my order is stuck in customs" "my order is late" "I got hit with an unexpected $80 charge to clear customs." It turns out selling internationally isn't as simple as adding international ship methods to your web site. 

We eventually developed our roadmap on [how to become an international ecommerce business](https://www.stickermule.com/blog/building-international-ecommerce-company). This guided our progress as we quietly made numerous improvements to help our international customers including:

1. Shipping worldwide quickly and with tracking. 
2. Ensuring orders clear customs without problems.
3. Eliminating import duties (we pay them for you).
4. Reducing shipping costs worldwide.
5. Introducing free shipping to Western Europe.
6. Supporting the metric system.
7. Supporting the Euro.

Along the way, our team grew to become international as well. More than 50% of our core team lives outside the United States. [David North](https://twitter.com/davidnorth) has been with us since our inception and lives in England. 

[Brian Quinn](https://twitter.com/bdq) works out of Ireland. [Lucas Húngaro](https://twitter.com/lucashungaro) is in Brazil. And, [Andrea Giannangelo](https://twitter.com/Facens) & [Stefano Di Luca](https://twitter.com/pennadl) are in Italy. Already, we've seen tremendous growth internationally, year over year we grow faster abroad than in our home country and that will accelerate as we execute our roadmap. 

If it seems as though we stumbled into becoming international that's because we did. We didn’t plan to have an international team or sell internationally. Our sole intention was to build an amazing customer experience and it turns out expanding beyond the U.S. and hiring the best people in the world were necessary to pursue that goal. 

Admittedly, we even stumbled into starting our international expansion in Italy. One night, I came across the [Unixstickers](https://www.stickermule.com/unixstickers) website and started a live chat session with Stefano Di Luca who happened to be Italian. You don’t have to chat for long with Stefano to realize you would love to work with him so we kept in touch. 

Eventually he introduced us to his partner Andrea Giannangelo who also left an amazing impression on us. We immediately knew we wanted Andrea & Stefano on our team and I’d like to think they knew right away they’d enjoy working with us. A few months later Sticker Mule Italy was born. We haven’t decided what country we’re headed to next, but if you’re interested in being a part of our journey [we’re hiring](https://www.stickermule.com/careers "Sticker Mule careers"). :-)

We recently launched an Italian version of our website. Here's a brief overview of how we stumbled into the Italian market. 

How to order custom stickers 

<YouTubeVideo videoId="AaGNbXMv0PM?rel=0" />

We created a short video of our ordering process. A few highlights:

1. Every product page (e.g. [die cut stickers](https://www.stickermule.com/products/die-cut-stickers) or [vinyl decals](https://www.stickermule.com/products/transfer-stickers)) has a "custom size" option. Pricing changes automatically as you manipulate sizes.
2. You can use drag & drop to upload your artwork on modern browsers.
3. You can select from 3 delivery dates during checkout including a free shipping option for U.S. customers.
4. Proofs are provided after checkout within 4 hours.
5. We ask for your credit card upfront, but we do not charge it until you approve all proofs.
6. You can request unlimited changes to proofs and we'll make them for free until your happy.
7. You can cancel any time prior to approving your proofs at no cost.

Hope you like our 1st attempt at video production. We'd love to know what you think. Leave a comment or tweet [@stickermule](http://twitter.com/stickermule).


We created a short video of our ordering process. 

On phone support 

We discontinued phone support recently. Our rationale is explained below:

### Phone support is difficult to scale.
You can't easily put a new person on the phone since you have limited control over the difficulty of calls they take. By contrast, you can let newer support agents respond to easy emails until they are comfortable handling more difficult ones.

### Phone support is difficult to monitor.
It's time consuming to listen to calls, but scanning emails is quick and efficient. It takes me an hour per day to read all of our customer service communication, but it's incredibly useful. Our core team reads the majority of our emails as a source of ideas and to keep a pulse on how we're performing.

### Phone support increases head count.
Phone support is more human capital intensive than email support. It's difficult to maintain the quality of your organization when head count grows quickly. Erring on the side of needing fewer people makes it easier for us to ensure all of our people are top notch.

### Phone support is difficult to automate.
Email support can be automated by developing faqs, macros, videos and efficient routing systems. It's not possible to automate phone support to the same extent.

### Phone support is not concise.
We aim to be clear, concise and comprehensive in how we respond to your questions. Often that means taking a moment to research an answer and review our response with coworkers before replying. It's not possible to handle a difficult phone inquiry in this manner without putting you on hold or calling you back later.

### Phone support is not documented.
Special arrangements made on the phone must be documented by an agent who could record them inaccurately. By contrast, all email inquiries are associated to your account and can be viewed by our entire team so that we are all familiar with your needs.

### Final thoughts...
Since implementing this change, we have been diligently reviewing every customer interaction, documenting problems and developing solutions to prevent common problems from recurring. Most support inquiries exist because we failed in some regard. Either our process had a hiccup or our ordering experience was confusing. Already we have fixed dozens of small problems and more improvements are coming soon.

### One small, related bonus...
We removed the phone number field from checkout for all US customers. Filling out forms isn't fun, but we hope cutting one needless field makes checkout a bit more pleasant.

Contemplating a similar decision? You may want to read [Wistia's](http://wistia.com/blog/scaling-support-why-we-removed-our-phone-number-from-our-website/) blog posts on this topic.


We discontinued phone support recently and explain why in this blog post. 

Berichten van Anthony Constantino