CEO

Experiment Traps: 5 signs that your business experiment isn’t actually an experiment at all

Part of SeriesC’s Statistically Speaking series Over the past 40 years, the Harvard Business Review (HBR) has studied how companies conduct business experimentation and they often find that companies fail to learn from their tests because they never adopt the true discipline of experimentation.

Using J.C. Penney’s costly and disastrous 2012 overhaul as a key example, HBR pointed out that ­– had CEO Ron Johnson established a proper set of experiments to test his ideas to do away with coupons, double down on upscale brands, and use technology to eliminate cash registers – he might have discovered how customers would revolt and push store sales down by 44% that year.

Too often these days we hear business leaders in CEO and CMO roles declare that they need to “test their hypothesis” or “run an experiment” in hopes of discovering whether a new business model or product will succeed. The trouble is, they don’t actually form solid hypotheses or conduct experiments correctly. The right way to experiment involves five scientifically sound steps: form a specific hypothesis, identify the precise independent and dependent variables, conduct controlled tests in which you can manipulate the independent variable, and then do careful observation and analysis of the effects, leading you to actionable insights. If you follow the steps, they’ll always present you with a valuable answer. So, where do many seemingly smart companies go wrong when it comes to business experimentation?

HBR posits that businesses can fall down at various stages when running a business experiment. Here, we’ve taken HBR’s Checklist for Running a Business Experiment and included what we’re calling Experiment Traps that you should recognize and avoid throughout the process:

  1. Purpose – HBR asks: Does the experiment have a clear purpose?
    1. The Hypothesis Hypocrisy Trap – did you and your management team agree that a test was the best path forward? Why? Is your hypothesis specific and straightforward (A good hypothesis clearly identifies what you think will happen based on your "educated guess" ­– what you already know and what you have already learned from your research)? If not, you’ve already fallen into the biggest experiment trap: Hypothesis Hypocrisy
  2. Buy-in – HBR asks: Have stakeholders made a commitment to abide by the results?
    1. The Cherry-Picking Trap – are you entering into this experiment equally prepared to be delighted or disappointed in the results? Will you avoid the temptation to cherry pick results that support your preformed ideas? Avoid this trap by sitting down and agreeing how your company will proceed once the results come in. If you see the experiment as part of a larger learning agenda that supports the company’s overall strategy, then you’re off on the right foot.
  3. Feasibility – HBR asks: Is the experiment doable?
    1. The Unsound Trap – HBR says “experiments must have testable predictions” but complex business variables and interactions or ‘causal density’ can “make it extremely difficult to determine cause-and-effect relationships.” Avoid this trap by knowing your numbers. Start by figuring out if you have a sample size large enough to average out all the variables you’re not interested in. Without the right sample size, your experiment won’t be statistically valid. Engage SeriesC’s analytics team to help you determine the right sample size for your experiment.
  4. Reliability – HBR asks: How can we ensure reliable results?
    1. The Corner Cutting Trap – when conducting your experiment you’ll be faced with challenges of time and cost and other real-world factors that can affect the reliability of your test. Resist the pull to cut corners by adopting proven methods from the medical field, like randomization, control groups and blind testing, saving you time in the design of your experiment and producing more reliable results. Or tap into big data to augment your experiment so you can better filter out statistical noise and minimize uncertainty.
  5. Value – HBR asks: Have we gotten the most value out of the experiment?
    1. The Wrong Impression Trap – don’t go to the trouble of conducting an experiment without considering and studying not only the correlations – the relationship between one variable and another – but also the causality. Causality helps us to understand the connectedness of certain causes and effects that usually aren’t as immediately obvious. Make sure to spend just as much time analyzing the data from your experiment as you did setting it up and executing it.

The bottom line: why go with gut and intuition and past experiences that aren’t apples-to-apples when you could be informed by relevant and tested knowledge? Steer clear of these experiment traps in your process and you’ll avoid inefficiency, unnecessary costs, and useless results. Embrace the proper process and you’ll learn something valuable, increasing your chances of success. Statistically speaking.

Avoid these experiment traps

Top Ten Best Practices for Managing Your Board

One of the most challenging things an entrepreneur must do is communicate effectively with his or her board. After spending countless hours with lots of entrepreneurs helping them prepare for board meetings and working with lots of boards to get what they need from their entrepreneurs, I’ve come up with a list of ten practices that make for good relations with this critical group of experts as you build your product and get it ready to go to market. How many of these do you regularly practice today? Choose the ones where you're deficient, and dig in. You'll get more out of your board relationships, and so will they.

  1. Understand how the board operates. Who is the alpha dog? How do the members communicate with each other? Know who your sponsor is. Respect the structure and fit in.
  2. Organize your thoughts ahead of time. Edit ruthlessly. Be concise. Use pictures, diagrams, infographics to convey messages.
  3. Never surprise your board. Reach out ahead of board meetings and run ideas past people. Use your sponsor to share good news and bad news ahead of board meetings and ALWAYS have a solution in mind if what you are sharing is a problem.
  4. Assign each board member a “role” in your mind to provide advice, counsel, context, whatever, and use them for that. Build a relationship with each one on a specific platform.
  5. Speak Metrics. Your preferred way of communicating may be words or pictures or even voice. But board members, especially venture capitalists,speak metrics. Learn the language and convey your points in it.
  6. Know more about your business than they do. Drop a “golden nugget” or two that they don’t know about your market, your competitors, your industry. Assume that they think you know everything about your business. Add something new each time you meet.
  7. Forecast accurately. A forecast is not a hope. And hope is not a strategy. Use a framework. Be conservative. Meet your forecasts. And if you will not meet them, let the board know ahead of time—why and what you are going to do about it. Start with your sponsor.
  8. Be confident. You are the CEO. There are people’s expectations and money resting on your decisions. Take the responsibility seriously and project accordingly.
  9. Set expectations. Be the voice of reason as it relates to goals and objectives that are first reviewed by and agreed upon by the board—before the clock starts ticking. Then remember to continue to set expectations in every conversation and especially every meeting. What is happening next and how does it fit in the plan and move the company forward?
  10. Provide follow-up that is also well organized, thoughtful and concise. Show your board that you took note of their questions and took the time to get the answers. Send them additional nuggets of good news between board meetings that show progress from the last meeting.