Product Marketing

Experiment Traps: 5 signs that your business experiment isn’t actually an experiment at all

Part of SeriesC’s Statistically Speaking series Over the past 40 years, the Harvard Business Review (HBR) has studied how companies conduct business experimentation and they often find that companies fail to learn from their tests because they never adopt the true discipline of experimentation.

Using J.C. Penney’s costly and disastrous 2012 overhaul as a key example, HBR pointed out that ­– had CEO Ron Johnson established a proper set of experiments to test his ideas to do away with coupons, double down on upscale brands, and use technology to eliminate cash registers – he might have discovered how customers would revolt and push store sales down by 44% that year.

Too often these days we hear business leaders in CEO and CMO roles declare that they need to “test their hypothesis” or “run an experiment” in hopes of discovering whether a new business model or product will succeed. The trouble is, they don’t actually form solid hypotheses or conduct experiments correctly. The right way to experiment involves five scientifically sound steps: form a specific hypothesis, identify the precise independent and dependent variables, conduct controlled tests in which you can manipulate the independent variable, and then do careful observation and analysis of the effects, leading you to actionable insights. If you follow the steps, they’ll always present you with a valuable answer. So, where do many seemingly smart companies go wrong when it comes to business experimentation?

HBR posits that businesses can fall down at various stages when running a business experiment. Here, we’ve taken HBR’s Checklist for Running a Business Experiment and included what we’re calling Experiment Traps that you should recognize and avoid throughout the process:

  1. Purpose – HBR asks: Does the experiment have a clear purpose?
    1. The Hypothesis Hypocrisy Trap – did you and your management team agree that a test was the best path forward? Why? Is your hypothesis specific and straightforward (A good hypothesis clearly identifies what you think will happen based on your "educated guess" ­– what you already know and what you have already learned from your research)? If not, you’ve already fallen into the biggest experiment trap: Hypothesis Hypocrisy
  2. Buy-in – HBR asks: Have stakeholders made a commitment to abide by the results?
    1. The Cherry-Picking Trap – are you entering into this experiment equally prepared to be delighted or disappointed in the results? Will you avoid the temptation to cherry pick results that support your preformed ideas? Avoid this trap by sitting down and agreeing how your company will proceed once the results come in. If you see the experiment as part of a larger learning agenda that supports the company’s overall strategy, then you’re off on the right foot.
  3. Feasibility – HBR asks: Is the experiment doable?
    1. The Unsound Trap – HBR says “experiments must have testable predictions” but complex business variables and interactions or ‘causal density’ can “make it extremely difficult to determine cause-and-effect relationships.” Avoid this trap by knowing your numbers. Start by figuring out if you have a sample size large enough to average out all the variables you’re not interested in. Without the right sample size, your experiment won’t be statistically valid. Engage SeriesC’s analytics team to help you determine the right sample size for your experiment.
  4. Reliability – HBR asks: How can we ensure reliable results?
    1. The Corner Cutting Trap – when conducting your experiment you’ll be faced with challenges of time and cost and other real-world factors that can affect the reliability of your test. Resist the pull to cut corners by adopting proven methods from the medical field, like randomization, control groups and blind testing, saving you time in the design of your experiment and producing more reliable results. Or tap into big data to augment your experiment so you can better filter out statistical noise and minimize uncertainty.
  5. Value – HBR asks: Have we gotten the most value out of the experiment?
    1. The Wrong Impression Trap – don’t go to the trouble of conducting an experiment without considering and studying not only the correlations – the relationship between one variable and another – but also the causality. Causality helps us to understand the connectedness of certain causes and effects that usually aren’t as immediately obvious. Make sure to spend just as much time analyzing the data from your experiment as you did setting it up and executing it.

The bottom line: why go with gut and intuition and past experiences that aren’t apples-to-apples when you could be informed by relevant and tested knowledge? Steer clear of these experiment traps in your process and you’ll avoid inefficiency, unnecessary costs, and useless results. Embrace the proper process and you’ll learn something valuable, increasing your chances of success. Statistically speaking.

Avoid these experiment traps

Five Ways to Market Your Tech When You're the Underdog

We were recently challenged by a client to re-position a mature product, a table-stakes part of a suite of B2B services.  The company’s sales and marketing teams believed the product was falling (or had fallen) behind the competition. Great development plans were in the works, but with delays and shifts, it didn’t appear there’d be much new to talk about in the coming six to nine months. That got our team to talking.  What’s the best way to market a technology when you know you’re the underdog in the marketplace? We shortlisted five of our favorite ideas, all of which we’ve tested with success in the course of our marketing careers.

1. Confirm, don’t assume. If you suspect you’re behind, confirm it with customers and prospects before you believe it as truth. Avoid the inferiority complex trap that can be very common, especially among weary and overwhelmed sales forces. Even when your product is terrific, sometimes all it takes is one competitor trumpeting boldly about their newest bell or whistle—however strategically unimportant it may be—to put your marketing and sales teams on the defensive. A non-confrontational way to learn what your customers and prospects truly think is to ask them these four simple questions when you talk to them next:

  • What are the features/functionalities of your current solution that you couldn’t live without?
  • What is your favorite feature/functionality of your current solution?
  • If you could change one thing about your current solution, what would it be?
  • Knowing what you know about our solution, would you recommend it today to a colleague?

These questions will give you insight into not just how you’re perceived, but also why people might think you’re behind, and what they most value in a solution.  If you find you are in fact competitive, a dose of sales training about your product’s strengths and differentiators could go a long way.

2. Change the product, or change the target market. It’s a beautiful thing about human beings: We don’t all want or need the same things.  What is “must-have” to one customer may be overkill to another. If you’ve confirmed your product has fallen behind in the eyes of your target market, then it’s time to change the product – or change the target. First, identify your product’s strengths. Perhaps it’s the price leader. Perhaps its differentiated by strong personal service and relationships.  Perhaps its more customizable. Then look at a gap analysis of what you’d have to build to re-emerge as a fierce competitor, and where.

  • Can you build a strong business case to invest in changing the product to close the gap and compete for your existing target market?
  • Is it more effective to shift target markets?  Consider if there is a different market segment that might rank your product highest based on your particiular strengths, at the product and company level.

Screen Shot 2014-05-06 at 4.28.44 PM3. Forecast your strategy to customers—boldly, rationally and transparently.  Develop a crisp point of view about what must be true about solutions in the future to meet your target market’s needs.  Then find ways to elevate the conversation to that strategic point of view, and to your intention to make all those things true.  Also, show them, don’t just tell them, what the future state might look like. If you have a concept in development that’s a year or less out, consider investing at least a small amount in conceptual user experience mockups or other visualizations. (Frankly, these could double as an aligning force for your development and product teams.) If something is more than a year out, conceptual artwork and thought leadership pieces, such as white papers, might do the trick. WARNING: Don’t be irrationally exuberant! If you promise something at a specific time, and you don’t deliver, you’ll do more to harm your credibility than any marketing may be able to repair.

4. Focus on your strengths.  Sometimes, it doesn’t hurt to acknowledge that your company is not all things to all people, and that your product cannot immediately be made to do all things.  Own up to your shortcomings when challenged by a customer. Then emphasize what does differentiate you—either with the product or with the way you deliver it: your people, your service, your processes, your accuracy, your consistency, your price, your geographic scope, whatever it may be.   Talk about your priorities, and discuss how they serve your customer’s needs best.

5. Play the Co-Opetition game. Ready to get really strategic and brainy about this?  Check out the classic ideas behind Co-Opetition, the game theory approach that Adam Brandenburger and Barry Nalebuff (of Harvard and Yale respectively) brought to the market in the mid 1990s.  You can shape the game, not just play the game, by using what they call the PARTS framework, which is explained pretty succinctly here.  This one takes some study and discipline, but it’s a great holistic exercise that could help align your executive team on the strategic priorities in front of you.

The Part About Product Launches that Isn't About Launching

Hey product managers and engineers: Do your customers complain about your product releases? Does sales suck up your time trying to deal with how to explain the point of the announcement? Does customer support dread new releases? Is the marketing department failing to get you the "buzz" you know you deserve for the great new stuff you've launched? If it's yes to one or more of the above, it might be your product launch process that's broken.

When you think of product launch, what is the first thing that comes to mind?

  • What the press release will say and where you want to see coverage?
  • Updating your Facebook page?
  • Previewing your app with "influencers" who you hope will write about it?

Screen Shot 2013-03-13 at 2.22.38 PMThat's all great, but it scratches the surface of what needs to be canvassed in a well-executed product launch. The most successful launches are executed when the people in charge ensure that:

  1. The launch meets the product/feature goals of the release
  2. The company is ready to market, sell and support the release
  3. The market is properly "primed" or ready to receive the product
  4. The current customers are ready to understand and use the product.

Too often, product launches focus just on # 1 and #2 above. Product management takes item #1. Product marketing is pulled in like a tactical partner to tackle #2, viewed as the department in charge of putting a nice bow on the pretty release and generating the "buzz" about it. When that happens, service and sales tends to get sent into reactive mode, having to scramble to try to manage through #3 and #4 as best as they can.

To be successful, product launches must be planned like a symphony, with marketing, account management, and sales involved and engaged.

So, how can you best build a holistic preparedness launch process?

1. Begin at the beginning. Collaborate with product marketing at the earliest planning stage to craft a go-to-market plan, rather than waiting to do so when the release is being put into production. Incorporate the goals of the launch in that early plan -- get precise on metrics.

2. As deliverables and milestones are scheduled and tracked to completion in engineering, so should they be in marketing. In fact, in the meeting that product management and engineering decides on a go/no-go decision, marketing's progress on its plan should be a part of the evaluation criteria.

3. Work as a team. In the end, what matters most is that putting these activities together facilitates collaboration between marketing and R&D. Marketing cannot be effective without being a part of the process, especially in high tech. If you treat marketing as a facade for your product, that's probably all you're going to get.

Your Policies are Features, Too

terms of useby Henry Hwong The huge uproar over the attempt by Instagram to update its Terms of Service is a sometimes painful reminder for product managers and marketers out there that when you have an Internet-based service, your product is more than just the features the engineers are creating.

This is not news for enterprise SaaS vendors that deal with Fortune 500 companies (and their lawyers) when negotiating contracts. That being said, the Instagram event has, yet again, raised awareness that will increase scrutiny on every company's data and privacy policies.

Product managers should own policies and terms of service (ToS), not leave them to the legal team alone. Effective product managers manage these along with other features, with implications for business model, customer acceptance and, as Instagram found out, change management during a release cycle. Remember, you're offering a service, and services come with a lot of non-engineering facets.

What should Instagram should have done differently? Clearly they did not spend enough time with the change management aspect of their rollout. Had they spend sufficient time and resources clearly articulating the improvements through their channels (e.g., app, website, marketing communications), with the understanding that Facebook has its own set of baggage, this could have been a complete non-event.

Furthermore, these improvements could have been implemented incrementally and done on a regular basis to get users to used to the idea that the ToS changes over time (i.e. put it on the roadmap!). Apple does a good job with this on iTunes and App Store, where they have gotten most everyone to accept changes to ToS on a periodic basis.