Make It Happen
My Tipsheets are chock full of ideas. They are all aimed at translating knowledge into a quick, action-oriented 60-second nugget.

First Name:
Last Name:
Tipsheet Archive
Randall's Resources
Whenever I speak or write, I often prepare extra "bonus" materials.
Enter the Resource Code to access this special content:
Resource Code:
Try this example Resource Code: eventplanning

When Market Research is insufficient… or ill-advised

by Randall Craig on March 14, 2014

Filed in: Blog, Make It Happen Tipsheet, Marketing

Tagged as: , , ,

Yes, we do market research.  In fact, each year we (108 ideaspace and Bramm Research) have done industry benchmarking and best practices research in the area of Social Media.  So it might be surprising to read a post about when – and why – market research should not be done.

At its core, the rationale for market research is to help make better decisions.  Market research can discover, validate, and reduce risk.  It can improve ROI, improve management confidence, and expose competitive threat.  And it can also be the worst thing that an organization can do.

Even when done perfectly, traditional market research – both qualitative (focus groups, interviews, etc) and quantitative (surveys) – has one inherent flaw:  it takes time.  The market research instrument is designed, tested, and approved.  The respondent sample set is recruited and surveyed.  The results are tabulated, tested for statistical significance, and analyzed.  Finally, a report is written, presented, and discussed.  It takes time and costs money.

Traditional market research works by examining a sample, and then extrapolating these findings to the “universe”.  There is no absolute certainty that a particular sample actually does represent the actual universe.  And even if it did, testing only a sample means that invaluable “outlier” findings would not appear.  Today’s competitive dynamic and sense of urgency has meant that organizations are now adopting two alternative approaches:  DIY (Do-It-Yourself) Market Research, and Agile Crowdsourcing.

DIY Market Research:  In this approach, the organization turns to the free: Survey Monkey, Informal web polls, Likes, and blog or Twitter feedback.  The rationale is that some data is better than none, and therefore it is best to ask questions and see the results.  While this sounds reasonable, it introduces what market researchers call bias, and can result in precisely the wrong decisions being made.

For example, a web poll asking about future service preferences will only capture results from those who already have at least some affinity to the organization.  (They did, after all, make it to the web site, and they did spend the time and effort filling in the web poll.)  But is it not just as important to learn what others think – especially those who didn’t ever get to the site?  If you put this concern aside, market research always exists in context: who are the respondents, anyway?  Are they from a geography that the organization doesn’t serve?  Are they children?  Are they machine-based spammers?

Mechanical bias can also skew results.  Consider a survey delivered using Survey Monkey,  Are the results biased because of the order of the questions?  Or because of the order of the answers within each question?  Or is the sample set too small, or non-representative?  Are the results biased because certain types of respondents (senior management, non-native English speakers) get frustrated by the survey length, and drop out prematurely?

Yes, DIY market research when done properly, can help. Sadly, though, the results can mislead. When the care taken to remove bias in traditional market research is neglected in favor of urgency, this isn’t surprising.

Agile Crowdsourcing:  This approach is different.  Instead of traditional market research, the organization pilots an early version directly in the marketplace.  Based on real feedback, it makes mid-course improvements, releasing better and better versions, or more and more focused products and services.  This approach, made popular by Google and their forever-in-beta products, is powerful.  It posits that by the time traditional market research would have provided basic insights, far more can be learned by going directly to the market and  experiencing first-hand feedback plus several product development-release-feedback cycles.

This same iterative direct-to-market approach is at the core of a Pay-per-click advertising strategy.  Skip the market research, and directly test which of multiple ads generate greater click-through.  Then iterate by dropping the poor performers, and release another generation of ads that are variants of the best.  Then repeat.  This same technique – typically called A-B testing – is also at the core of improving email response and web page conversion.

While big data and Agile Crowdsourcing are exceptionally appealing, there are downsides.  For example, making changes based on observed behavior does not explore the underlying motivation to transact.  It also doesn’t evaluate the motivation of those who abandon their journey along the way – or those who are not even aware of the product or service’s existence.  Agile Crowdsourcing cannot explore the “could be” for products or services not yet on the market.  And finally, using this approach gives your competitors a sneak-peek at your capabilities and direction, reducing your competitive advantage.

This week’s action plan:  Is traditional market research dead?  Absolutely not.  But the advantages (and risks) of the alternatives (DIY and Agile Crowdsourcing) are too important to ignore.  This week, consider your most critical management decision: how might you use each of traditional market research, DIY market research, and Agile Crowdsourcing to make a better decision.

Marketing Insight:  Traditional market research, just like almost every other area of marketing, is a fast-changing field.  In the hands of an experienced market researcher, DIY and Agile crowdsourcing are two of the most exciting developments around.  Yet just as the early Mac allowed anyone to be a desktop publisher (often with horrible-looking results), DIY and Agile crowdsourcing allow anyone to do “market research.”  Watch out for business decisions that are made based on a biased market research approach.


Note: The Make It Happen Tipsheet is also available by email. Go to to register.

Randall Craig

@RandallCraig (follow me)
:  Professional credentials site
.com: Web strategy, technology, and development
:  Interviews with the nation’s thought-leaders


Randall has been advising on Digital Strategy since 1994 when he put the Toronto Star online, the Globe and Mail's GlobeInvestor/Globefund, several financial institutions, and about 100+ other major organizations. He is the author of eight books, including Digital Transformation for Associations, the Everything Guide to Starting an Online Business, and Social Media for Business. He speaks and advises on Digital Transformation, Digital Trust, and Social Media. More at

Leave a Comment

Previous post:

Next post: