Editor’s note: William C. Pink is senior partner, creative analytics at research company Millward Brown. This is an edited version of a post that originally appeared here under the title “Liberating research: a manifesto for change.”
I have previously argued that big data is not replacing research, it is liberating it. Researchers are liberated from generating a new survey for each new learning occasion; instead, ongoing big-data assets can be leveraged for many topics, allowing subsequent primary research to go deeper and fill in the gaps. Researchers are liberated from needing to rely upon bloated surveys and instead can keep surveys short and focused on those variables that they are ideally suited for, resulting in better data quality.
I stand by that argument, and we have many examples of forward-looking brands adopting this approach for their research programs. However, we still see far too many brands clinging to research practices that are out-of-date. For example, the default approach remains to ask each survey respondent all possible questions. Building research in this manner is convenient and comfortable but it does not encourage consideration of alternative sources of insight.
The challenge of the new
This hesitation should not be surprising; well-established behaviors and practices are hard to change. As any observer of human behavior will tell you, the best predictors of an individual’s future decisions are his or her past decisions. In other words, you can’t teach old dogs new tricks. When we examine our actions and decisions as researchers who study consumers, brands and marketing effectiveness, we see that far too often we are still acting like the proverbial old dogs. I say this not to offend but rather to ignite a movement throughout the research community to revisit our first principles of design and data quality. Liberated research can only deliver to its highest potential and promise if we actually liberate ourselves from practices that are not working. Otherwise, we risk bringing about our own obsolescence.
Let’s talk about specifics. What do we need to do in order to liberate research?
Shorter surveys. First, we need to stop burdening consumers with long surveys. The evidence is overwhelming that shorter surveys yield better data quality and better consumer engagement. In a recent example, Kantar, TNS and Millward Brown collaborated on parallel studies for the same consumer target. The first study matched the historical design and took over 25 minutes to complete. It was built to ensure a completed data set of respondent-level information for each consumer. The second study was purposefully designed to be much shorter, taking around 12 to 13 minutes to complete. Each consumer was asked only those questions deemed core to understanding the category and meeting the primary analytic objectives.
The results were startling. For one product category, 3.5 times more attributes were identified as important in the shorter survey than in the longer survey and the average level of endorsement for brands in this product category was 31 percent in the shorter survey, compared to 17 percent in the longer survey. In effect, the contextual differences of the survey environment generated very different results and consumers were willing to share more information in the shorter survey.
Shorter is better. Yet, we are very slow to reduce the length of our questionnaires for fear of giving up information that we are used to having. How many competitive brand sets are lingering to ensure consistency with the past, even though we know the past is not a reflection of the current market? Why do we cling to information generated by a long survey that is familiar and comfortable but potentially inaccurate?
Elimination of redundancy. Second, we need to stop asking consumers redundant questions. What makes questions redundant? When consumers give the same patterns of response to multiple questions. Data reduction techniques have existed for years to detect this but how often are we implementing those findings by removing redundant content? Taken further, data reduction techniques provide a line of sight into the themes that consumers perceive. Given the maturity of many markets and categories, we expect to see very stable themes emerge from our analysis – themes of product quality, corporate reputation, consumer motivation, etc. These themes are typically few and rarely change. However, we often see 20 questions reduced to only two themes in consumers’ minds. In that case, why are we continuing to ask all 20 questions? If, year after year, we see so few themes emerge from so many questions, then we are missing key opportunities to optimize our questionnaire designs.
We know redundancy only increases consumers’ frustration levels and reduces the quality of their responses. The bottom line is that we are running suboptimal research designs by keeping the status quo. We should remove redundant questions without hesitation.
Meaningful measurement. Third, we must ensure that we have the most consumer-friendly and accurate mechanisms for capturing relevant insights about what matters, even if this means changing the historical measurement system and implementing a better, more appropriate measurement system for today.
Every day we work with brands to improve their measurement programs. This ranges from linking different data assets for new perspectives on old phenomena to utilizing the latest protocols for survey design and measuring brand equity. We bring our data and new learning to the table and our clients share their category-, brand- and market-specific experiences. Our conversations typically revolve around how to bring our collective expertise together for improved market measurement. However, at some point, clients often worry that implementing changes to their measurement programs will result in changes to historical trending – and this is usually when the air deflates from the progressive tires.
* * *
Of course, giving up historical trends from a long-built and well-invested data asset is a tough pill to swallow. It is difficult to explain to executives that we are making changes to a research instrument and losing comparability to history, even though we are quite good at devising statistical protocols to preserve trends and can make the old data act like the new or calibrate new results to match the old. The fundamental point is that we must opt to change. Trend maintenance should not be our first principle of design; we should abandon instruments and approaches that may have been suboptimal in the first place.
Liberation in practice
Millward Brown’s framework for understanding brand equity is an example of the benefits of this kind of thinking. First, we designed the framework so it can be asked quickly, averaging about three minutes per consumer, and we only ask the questions that consistently link to behavioral success across categories and markets. Second, we designed the question format to mirror the competitive context that consumers experience in their daily lives. This is done through a design called “associative scale and rank.” For the key dimensions in the model, the consumer positions each brand on a 0-10 scale and overlays all the brands on that scale to give a relative ranking. This provides the sensitivity of leveraging a full scale for each dimension while being much more consumer friendly and engaging than a separate assessment of each brand, so we get the information we need quickly and accurately. Most importantly, our brand measurement framework serves as connective tissue across research solutions and data assets. The new model is designed in a short, engaging, repeatable and standardized manner that can be implemented across a range of research scenarios.
Time to change
This brings us back to why we want researchers to embrace the opportunity the modern data landscape provides and liberate themselves from lengthy, single-source surveys. To suggest that this is a necessary step to effectively utilize mobile devices for surveys is valid but misses the essential point: Shorter, focused surveys are better surveys. If we properly frame business problems and think about them from a research-program mind-set, then we will be empowered to enjoy the benefits of liberated research. What business problems can only be answered by having all the information from the same individual in one survey? What learning objectives can be better addressed across a suite of connected research solutions? Which questions are redundant and repetitive?
Of course, we raise the stakes when we remove the security blanket of asking each consumer about all the pieces of the business puzzle. This requires clarity of planning and purpose but it is a challenge that we should embrace. The evidence shows that we are kidding ourselves if we think that analyses of long surveys with poor-quality data can provide accurate stories and actionable recommendations.
Stand and deliver
That is why I see this as a manifesto for change for the research community. We know what works: shorter surveys that respect consumers’ limited availability in a time-pressed world; research tools that engage consumers in a dialogue using everyday language; and research solutions that encourage participation, not frustration.
We can no longer reasonably claim, “I don’t want to rock the boat,” as an excuse for not rethinking research designs that don’t meet these criteria. The boat has already been rocked.
As an industry, we talk a lot about moving from backward-looking insights to research with foresight and forward-looking actionability. I endorse and hope to amplify those goals in this post. With shorter surveys run as part of a larger research program that includes both big and small data assets, we will be well-positioned to deliver on those goals. But if we don’t actually speed up our implementation of shorter surveys and move away from bloated, historical survey designs, our talk will simply be hot air.