Experiment Traps: 5 signs that your business experiment isn’t actually an experiment at all

Part of SeriesC’s Statistically Speaking series Over the past 40 years, the Harvard Business Review (HBR) has studied how companies conduct business experimentation and they often find that companies fail to learn from their tests because they never adopt the true discipline of experimentation.

Using J.C. Penney’s costly and disastrous 2012 overhaul as a key example, HBR pointed out that ­– had CEO Ron Johnson established a proper set of experiments to test his ideas to do away with coupons, double down on upscale brands, and use technology to eliminate cash registers – he might have discovered how customers would revolt and push store sales down by 44% that year.

Too often these days we hear business leaders in CEO and CMO roles declare that they need to “test their hypothesis” or “run an experiment” in hopes of discovering whether a new business model or product will succeed. The trouble is, they don’t actually form solid hypotheses or conduct experiments correctly. The right way to experiment involves five scientifically sound steps: form a specific hypothesis, identify the precise independent and dependent variables, conduct controlled tests in which you can manipulate the independent variable, and then do careful observation and analysis of the effects, leading you to actionable insights. If you follow the steps, they’ll always present you with a valuable answer. So, where do many seemingly smart companies go wrong when it comes to business experimentation?

HBR posits that businesses can fall down at various stages when running a business experiment. Here, we’ve taken HBR’s Checklist for Running a Business Experiment and included what we’re calling Experiment Traps that you should recognize and avoid throughout the process:

  1. Purpose – HBR asks: Does the experiment have a clear purpose?
    1. The Hypothesis Hypocrisy Trap – did you and your management team agree that a test was the best path forward? Why? Is your hypothesis specific and straightforward (A good hypothesis clearly identifies what you think will happen based on your "educated guess" ­– what you already know and what you have already learned from your research)? If not, you’ve already fallen into the biggest experiment trap: Hypothesis Hypocrisy
  2. Buy-in – HBR asks: Have stakeholders made a commitment to abide by the results?
    1. The Cherry-Picking Trap – are you entering into this experiment equally prepared to be delighted or disappointed in the results? Will you avoid the temptation to cherry pick results that support your preformed ideas? Avoid this trap by sitting down and agreeing how your company will proceed once the results come in. If you see the experiment as part of a larger learning agenda that supports the company’s overall strategy, then you’re off on the right foot.
  3. Feasibility – HBR asks: Is the experiment doable?
    1. The Unsound Trap – HBR says “experiments must have testable predictions” but complex business variables and interactions or ‘causal density’ can “make it extremely difficult to determine cause-and-effect relationships.” Avoid this trap by knowing your numbers. Start by figuring out if you have a sample size large enough to average out all the variables you’re not interested in. Without the right sample size, your experiment won’t be statistically valid. Engage SeriesC’s analytics team to help you determine the right sample size for your experiment.
  4. Reliability – HBR asks: How can we ensure reliable results?
    1. The Corner Cutting Trap – when conducting your experiment you’ll be faced with challenges of time and cost and other real-world factors that can affect the reliability of your test. Resist the pull to cut corners by adopting proven methods from the medical field, like randomization, control groups and blind testing, saving you time in the design of your experiment and producing more reliable results. Or tap into big data to augment your experiment so you can better filter out statistical noise and minimize uncertainty.
  5. Value – HBR asks: Have we gotten the most value out of the experiment?
    1. The Wrong Impression Trap – don’t go to the trouble of conducting an experiment without considering and studying not only the correlations – the relationship between one variable and another – but also the causality. Causality helps us to understand the connectedness of certain causes and effects that usually aren’t as immediately obvious. Make sure to spend just as much time analyzing the data from your experiment as you did setting it up and executing it.

The bottom line: why go with gut and intuition and past experiences that aren’t apples-to-apples when you could be informed by relevant and tested knowledge? Steer clear of these experiment traps in your process and you’ll avoid inefficiency, unnecessary costs, and useless results. Embrace the proper process and you’ll learn something valuable, increasing your chances of success. Statistically speaking.

Avoid these experiment traps

A Call for Research Participants

In our work with small, rapidly-growing clients, we have noticed a lack of benchmarks for marketing spend as they evolve from ‘idea to IPO.’ There are some great resources out there for large public companies to gauge themselves and stay up to date with spending trends. Reports from Forrester, Gartner, and IDC are incredibly valuable resource points and we would recommend them to anyone looking for help in creating a marketing budget. However, while these reports provide excellent data and are grand in scope, we feel that they are underserving a very important market. They lack data on high-growth, smaller, fast-paced companies. In other words, startups. We are aiming to solve that. We have set out to field a research study specifically targeted to inform “Pre-IPO”, “Pre-Exit” companies. This is important because startups are different. Year-to-year there is a tremendous amount of change. The current resources available to startup marketing leaders when trying to benchmark their spend are failing to capture that and this report will provide them with more reliable and targeted data that is actionable.

Our report will outline marketing budget trends and spend benchmarks of the startup universe. We are beginning by conducting a brief survey that will break down spend by industry, company size, and company stage. We will then further analyze spend trends by tactics and learn about what strategies might be helping companies meet their revenue targets. This research will be conducted in manner that puts the legitimacy of the data and privacy of our participants in the highest regard. As such, we’ve structured the research to be “double-blind” – thus eliminating the ability to trace specific answers back to individual participants or companies.

We are currently fielding participants for the research and are looking to include as many as we can in the final report. In exchange for participation, all of those involved will receive an advance copy of the final report, which we plan to publish by the end of the year. If you are interested, we ask that you please fill-out this quick sign-up form and pass along to anyone else you think might be interested. We will follow-up with those who fill-out the sign-up form and they can expect to receive the actual survey link from us within the week. If you or someone you know would like more information, please drop a note to me, Richard Dolezalek at or my colleague, John Volkmann at

5 W’s and 1 H of Getting Outside the Building

Steve Blank makes a set of simple but critical points in his recent Udacity video about the importance of getting out of the building and talking to customers. As we work with companies of all sizes and maturities at SeriesC, we often field questions about how to get this done, or see leaders who appreciate the need to talk to customers but aren't sure how to start.

There is an art to talking to potential and current customers. The first step is to ask yourself the right questions to find out the who, what, when, where, why, and how of getting the information Talk-to-your-customersyou need. Here is how to get that discussion started.

What? – What is it that you want to learn? This may seem obvious but dive a little deeper. Start by setting a hypothesis. Just like any other study using the scientific method, a hypothesis is necessary to help frame questions and determine the direction research needs to go.

Next, create goals for the study. This should be a list of facts that you would like to know – or, perhaps, things you think you know but would like to validate with outsiders who do know. Like any other goals you have set in the past they need to be SMART (Specific, Measurable, Actionable, Realistic, and Time-based). For example, a goal for a company trying to understand poor reviews in the app store might be to learn which features have been mentioned most frequently in negative reviews in the last 6 months. A goal for a large company trying to bring a new line of business to market may be to validate its hypotheses on what its best customers would most likely embrace in new service offerings.

Who? - Who knows the information you are seeking? Do you need to talk to current customers, potential customers, lost customers, or industry experts? Maybe a mix of all four? Your hypothesis and goals should inform this. You should already have a clue as to who might have answers to your questions about your business. You may find out this clue is wrong once you start talking to people, but your best guess is the right place to start.

There is a big WHO and a small who. The big WHO is your hypothesized target group. The small who is the target sampling of a few who will give you a good representative perspective without breaking the bank on time and resources. The internet has many great tools for helping to select the right size and type of sample to answer your research questions. Or, you could just make some informed gut decisions: Choose a sampling of ten people, aiming for a mix of prospects and customers, segments, or other demographics that matter to you.

How? - How should you talk to your target group? Basic data collection techniques include casual conversation, surveys, experiments, secondary data or research, observation, focus groups and in-depth interviews. In picking the right type of data think backwards from what you want to know. In what form would this information be easiest and most actionable? How important is recording and sharing the feedback within your organization? Would the feedback be more objective if you had a neutral third-party do the interviewing, or are you likely to get better results if you take this person out for coffee in person? Combine the answers with knowledge your target group and you will arrive at how to best collect your data. Don’t become paralyzed if this all seems to difficult to implement. Even a casual conversation can be fruitful. If you’re stuck, just pick up the phone and invite someone to lunch.

Where? - Where is closely related with how. Begin by thinking about where to talk to your target group. Is it easier to reach them through an email survey, at your next client quarterly review, or directly at a retail location? Where are they most likely to give you honest answers and where are they most likely to give you useful answers? The answer might be the same place, but it also might not. Remember to return to your goals when making the decision about the best way to reach your customers.

When? - Set a calendar for your data or information gathering. Plan out how long you expect each step to take and stick to it. Research can be daunting. Having a schedule will help you stay on track. As part of this schedule make sure not to overlook your data collection times. For example, when talking to customers at a retail location try different times of day. By only talking to them during only one window you may be missing out on potential facts that are time specific such as wait times during the lunch-rush or the difference in customers during the week and on weekends. If your customers are enterprise leaders, be respectful of time commitments. Be on time, and take no longer than the time you promised to take.

Why? - To be blunt, you don’t know it all. You never will. And just when you think you do know it all, rest assured that the market will change, and throw what you know back into question. Your best assumptions are only assumptions until they are validated by the people who matter most – the people who are buying what you exist to sell them. So start with smaller questions. Why did I collect this information in the first place? How can it benefit my organization? How can use what I now know to change my strategy to appeal to the right target market?

Hopefully these questions can guide your next marketing research project, whether it’s a simple brave trip out of the building or a full-blown objective study. Need help? Leave us a comment or contact our consultant team.