You are here: Home / On Demand / Making Tough Product Decisions – Gut Feel v. Quantitative Analysis

Making Tough Product Decisions – Gut Feel v. Quantitative Analysis

May 9, 2010 By
Since I started in venture capital four years ago, I have met with a lot of software entrepreneurs in early stage start ups. One consistent theme across all those meetings is that I have found entrepreneurs tend to rely fairly heavily upon qualitative v. quantitative analysis  to determine features, markets, pricing and positioning for their initial offerings.

I can certainly understand why this might be; most early stage entrepreneurs are subject matter experts in their domain and have a strong point of view with respect to what their application/product should do.  In addition, quantitative analysis can take time and money – something that is in short supply at an early stage startup.

That said, even the best product team can get it wrong — delivering a product that may not be exactly what the market wants, or is priced and/or positioned incorrectly. And, unlike larger software companies, a mistake in an early stage software company can be fatal – or at least cause the company to have to take in more capital and require more time to get into market.

So, I am surprised that more start up software companies (and big software companies, for that matter) don’t incorporate more quantative analysis to help them make critical product, market, features, pricing, and positioning decisions.

In my opinion, the best methodology for making such trade-offs is conjoint analysis. A definition of conjoint analysis from Survey Analytics follows:
Conjoint analysis is a popular marketing research technique that marketers use to determine what features a new product should have and how it should be priced. It requires research participants to make a series of trade-offs. Analysis of these trade-offs will reveal the relative importance of component attributes.
I will give you an example of how I used conjoint analysis to help me with the  CRMOnDemand division at Siebel Systems.

I took over the CRMOnDemand division in early 2004 after returning to the company after a 20 month hiatus. The CRMOnDemand division had been created by Tom Siebel to go after Salesforce.com and the SMB market – a market Siebel hadn’t really participated in until that point.

The CRMOnDemand division had two products to offer: UpShot’s “on demand” CRM products as well as our own internally-developed ”on demand” CRM products  (Note: Upshot was a SaaS-based CRM provider that Siebel acquired to gain access to its personnel who had demonstrated they knew how to compete against Salesforce in the “on demand” market).

Unfortunately, the CRMOnDemand division had been struggling since its initial launch 6 months prior to my return and had achieved very little traction with customers and Siebel’s own internal sales organization.

In my first weeks, I sat down with the product, marketing and sales organizations and listened to what the teams thought about UpShot v our internally-developed on demand product, key product features, market positioning, sales strategy, etc. As a result, it didn’t surprise me why we were getting our clock cleaned; we were competing against a strong competitor in its market of strength with a confusing 2-product, ”me, too” message and we were primarily using Siebel’s enterprise sales organization to go after small (SMB) accounts.

I decided to engage a small research firm (Incyte Group) to help us reach out to prospects and customers to better understand how they perceived our “on demand” products, pricing and messages.

From these surveys, we captured a significant amount of data that drove our conjoint analysis. We were able to identify many different issues that were not necessarily obvious going into the study. This data enabled us to quantitatively understand specifically what our engineers should be building, what our marketers should be saying and what our sales organization should be selling.

Consequently, the long and heated debates inside Siebel changed from subjective opinons regarding what should we build, how should we price it, how we should sell it, etc. to instead focusing on solving all the issues surfaced from the surveys. Engineering and Products were now working in alignment. Sales was confident we were building the right product and could sell with confidence. Marketing had a solid messaging platform to work from. And, I personally felt I had the data to show Siebel’s senior executive staff and the Board what we needed to do in order to win in the market.

The result: in 18 months, from relatively little revenues, we were able to grow the organization into an $80M annual bookings business.

While the study certainly wasn’t cheap (it took about 4 months and $100,000+ ), I think about how much it would have cost us had we not done it at all. And, today, with my own early stage start ups, I consider the risk associated with not doing conjoint analysis; how much time/capital will we spend if we get the product, market, pricing, go to market model wrong?

I recently read an interesting article on conjoint analysis by Brett Jarvis at Sawtooth Consulting. I think he makes a powerful case for why each of us involved in developing and/or investing in new product offerings – SaaS or otherwise – would be well served by using conjoint analysis to verify and/or refute our plans.

So, the next time you are faced with a critical product, pricing, messaging, go to market decision, rather than relying solely upon ”gut feel”,  I would suggest you initiate a little conjoint analysis. It doesn’t have to be perfect, take a long time, or a lot of money – and, there are a lot of small consulting firms that can help you do it. I think you will find it illuminating, clarifying, uniting and it might well be one of the single most important decisions you can make as a start up.

Filed Under: On Demand, SaaS, Software, Software as a Service
  • http://www.softwareindustryinsights.com/ Glenn Gruber

    Bruce, nice post overall. I like tools like conjoint that can help ensure the product development roadmap is properly prioritized to ensure a strong market impact and reduce feature creep. However, those who use such tools must also keep in mind that conjoint is good for analyzing the affect of different feature bundles and pricing trade offs on purchase intent, but doesn’t do such a good job for taking into consideration other elements like brand. Which leads me to the next comment.

    Your headline of using data instead of your gut, did strike another chord. I’m currently reading Steven Sinek’s “Start with Why”, which I would highly recommend. Sinek provides a framework that companies and individuals can use to make sure that the elements of the product are consistent with the core mission or personality of the brand which can impact the decision process of customers who will often pay more and/or overcome other purchase obstacles because of the shared values and affinity they have for the brand. Otherwise, marketers can get caught in the trap of just competing on features and have to rely on classic manipulations like dropping your price, which only have short term effects and don’t encourage loyalty.

  • http://www.interwest.com Bruce Cleveland

    Great suggestion and insights, Glenn. You are absolutely right — brand matters. At Siebel, the process we used with the Incyte Group did include evaluating our brand; that confirmed the fact that the Siebel brand worked well in enterprise but worked against us in the SMB market. We changed some of our positioning and our go to market strategy to help us address that issue.

  • http://www.softwareindustryinsights.com/ Glenn Gruber

    I had to run out last night and didn’t finish my post. I did want to underscore your point about companies being too quick to build what they think (hope) customers want — essentially spending their R&D dollars after playing pin the tail on the donkey with their PRD. But they bristle when faced with spending $100K or $250K to be sure that the next several million dollars is committed to the right things. I had a similar issue many years ago while working at a Japanese peripherals company. I was sure the existing roadmap was flawed yet they wouldn’t spend $250K before committing $20M in NRE and manufacturing costs. Just crazy. Luckily I was able to persuade them to make some changes based on homegrown analysis, but we would have been even better off utilizing data direct from the target customer.

    As a VC, how do you “encourage” your portfolio clients to do some homework before they start spending your money? Some of the great entrepreneurs do have the best intuition, but so many others believe they are great entrepreneurs and subsequently miss the mark.

  • http://www.interwest.com Bruce Cleveland

    Glenn: I find that I have very little “persuasive” power with any of my portfolio companies — certainly far less than I expected before becoming a vc. It was much better when I was the evil GM of a division where I could impose my will! :)

    I think the reason most high tech companies, large or small, reject using quantitative analysis is that they believe they know better what the markets/customers want than the markets/customers themselves. That, and the fact, that many market research firms don’t use deep mathematics and statistical analysis to generate correlates that provide enough unique insight.

    The large companies are large because they’ve been successful so their ego/behavior is reinforced — they know better than anyone else. Small companies generally feel they don’t have the luxury of time/money to make the investment. They need to get in the market quickly so they can start competing — if they need to change, they will iterate.

    I believe both these behaviors are wrong and end up costing investors (and management) millions in delivering the wrong product, with the wrong positioning and/or the wrong price. It’s the dirty secret that companies just don’t talk about.

    I haven’t generally found people with deep quantitative research skills inside any of my firms – large or small. And, unfortunately, not many Product teams are used to using this type of a process. So, this is definitely a skill I would contract for and I would get a number of solid references before engaging.

    I can tell you that if I were to go back to an operating role, I wouldn’t build any product ever again without this type of analysis done in advance.

  • Ken

    I think a more important consideration for start ups is in building an “agile” culture.

    The assumption should be that you will make mistakes and not get it right the first time. You should have a mechanism in place to make decisions in timely manner and then have an ability “course correct” if necessary based on quantitative analysis.

  • http://www.interwest.com Bruce Cleveland

    I like your notion of an “agile culture”.

    I think a “mechanism” to such a culture is strong instrumentation built into the fabric of the business (KPIs = sales cycle times, discount rates, win/loss rates, on boarding times, adoption rates, usage rates, support calls/trouble tickets, churn rates, etc.).

    And, there must be a process that supports the mechanism. The process should include individuals who own the KPIs and a weekly executive staff meeting where the CEO reviews the KPIs with the team so that the company doesn’t miss key market signals.

    To me, an “agile culture” is one where a company is both willing and able to move quickly — and in the right direction; which is where the quantitative analysis plays a critical role.

  • Mary W

    Great article. What happens when ownership (privately-held, no investment or outside advisors) won’t really invest into marketing or lead nurturing (at the pace of the competitive SaaS market) and expect to grow near exclusively through a telesales force? Selling in a commodotized market but have a strong offering & in higher-end of market. What do you suggest if the owners plan to throw peanuts (1-2% of revenue) at marketing/brand positioning/lead gen but expect to sell/close enterprise clients thru phone sales in a pretty competitive space? $6M business. I’m in a vp/leadership position but am not one of the majority owners (so not my $$) and feel like I’m trying to push the company uphill w/ this challenge & their gravitation to cold calling & controlled costs w/ rep salaries & commission. Goal is an exit in a few years. We’ve grown but have been relatively flat for 2.5 yrs (top-line), but owners just have resistance to investing in the types of things you talked about (referencing a new division that is 12 mos new in a 4 yr old business). What is one to do?

    If you’re willing, would love to hear your thoughts.

    • http://www.interwest.com Bruce Cleveland

      Mary:

      You might want to read my article titled “SaaS: Lead Generation Not Sales Capacity Drives the Model”.

      http://www.interwest.com/software-as-a-service/market-leadership/saas-lead-generation-not-sales-capacity-drives-the-model/

      Additionally, you might search the blog at Marketo (www.marketo.com) regarding the role of lead gen/lead nurturing. There may be useful data you can use to make a case with your colleagues there.

      From the little that you’ve said, it sounds like your colleagues are skeptical that investments in marketing really pay off. You may want to offer doing a “trial” where they do a little investing in lead gen and if it pays off, they agree to do a little more.

      The fact is that if you are a SaaS company and your ASP is below about $100K ARR, if you’re not investing in lead gen to drive the top line of the funnel you’re never going to make the model work.

      So, if they still won’t listen – and you probably don’t want to hear this — you may want to consider looking at other companies now vs waiting for the inevitable conclusion.

  • http://customer-enlightenment.com Sylvie Leotin

    Great article, Bruce! I’ll need to look more into this type of analysis…

    I am seeking to explore techniques to light the path towards understanding and elucidating customer needs, in the context of product innovation in my blog. I call this “customer enlightenment”. As entrepreneurs and product innovators, how do we reconcile our creative vision with the needs of others in order to bring to market product innovations that delight customers.

    Thanks for the suggestion, I will look into this.