Is methodological rigor getting neglected?

Editor’s note: Edward Appleton is European consumer insights manager at a major multinational based in Munich, Germany.

Market research is undergoing an exciting phase of change – the survey is under attack as a shallow method of accessing people’s real opinions; many new tools are being actively promoted to provide different, more authentic angles.

Innovation is all around us, which is a healthy sign.

But are we in danger of assuming that just because something is new it’s better? Similarly, are we in danger of ditching the “MR baby with the bathwater” when we engage in what sometimes feels like radical renewal?

Here’s my take: 

• We neglect methodological rigor at our peril. That’s what we’re respected for. Validation of any new technique is critical. To quote Andrew Ehrenberg ( “A result can be regarded as routinely predictable when it has recurred consistently under a known range of different conditions.”

Research-on-research is essential to understand what differences new data collection modes – such as mobile – make. I applaud those who engage in such efforts.

• It’s generally recognized that MR needs to shift from being a data provider to insights advisor, with an ability to create a compelling data narrative. This new skill set means that we are attracting people from outside the core industry – folks from marketing, PR and advertising are increasingly occupying senior MR positions on both the agency and client sides.

This is great. However, none of us should forget that a great narrative is only as powerful as the robustness of the data and the methodology behind it. Better be perceived as boring than wrong or misleading.

• There is a huge pressure on budgets on the one hand but on the other a recognition that gaining a balanced understanding of a given perceptual situation might well actually require more budget to handle a mixed-modal approach.

The onus is on us all to find ways to carefully reduce unnecessary cost and manage the understanding of risk.

• DIY research is a trend that’s in all likelihood here to stay. I view this positively – it allows more companies to do more research. However, bad DIY research can be very misleading.

Understanding what good research practice is versus bad is important and educational efforts such as those done by Kathryn Korostoff, Annie Pettit and Ray Poynter are to be applauded. I miss Jeffrey Henning’s excellent blogs from his time at Vovici.

• Many of the existing arsenal of tools in MR rely on direct questioning. Done sensitively, there’s definitely a role for that in my view – customer satisfaction, brand tracking, concept testing, for example. Access to historical databases and norms allows one to upweight or downweight sensitively and accurately as required.

As it becomes increasingly possible to introduce context into this equation – via mobile, for example – and we embrace ways of using visuals as a counterbalance to words in our survey design, then direct questioning becomes potentially even more powerful.

• Experts may not always (or often) be right – neither is an amateur perspective. It’s modish to attack expert opinion – it’s blinkered, doesn’t move with changing circumstances, narrow in focus. I’m indeed a fan of boosting the role of intuition, gut reaction and referencing broader audiences early on in any MR process – but as a basis for forming a hypothesis that requires validation, not as a substitution.

It’s commonplace to say that MR tools are becoming commoditized, that we should concentrate on the impact, not the method.

Yes, the days of black-box modelling seem largely behind us. But that doesn’t mean the role of advanced analytics is finished – on the contrary. The people who have the expertise and experience in helping marketing folk (and many in insights as well) leverage the power of sophisticated analytic techniques are extremely valuable.

I enjoy refreshing my fundamental understanding of both basic and advanced statistical and sampling techniques as much and as often as I can – it’s empowering and invigorating. It’s just not something I talk about much, in the interests of maintaining my current circle of friends…;) – which in a way is one of our conundrums as an industry.

Curious, as ever, as to others’ views.

This entry was posted in Market Research Best Practices, Qualitative Research, Quantitative Research, State of the Research Industry, The Business of Research. Bookmark the permalink.

3 Responses to Is methodological rigor getting neglected?

  1. Lucas Borja says:

    Great article! Is a good elaboration of the current state of Market Research and its flaws. I do sense a feeling of alarm by MR professionals at the idea that market research is in fact become a tool in the arsenal of the smallest of organizations. And there are loud voices pointing at the tittle of this article. In my view, the use of ‘simpler’ but yet valid statistical models and sampling algorithms do not make the findings any less robust; although you are right to point out the storyteller should not veer far form what these methods allow her/him.

    My take is that MR is becoming a more accessible service for all to use, but MR professionals have molded themselves so much to the status quo that they do not see the enormous favor technology platforms are doing them by ‘popularizing’ market research. After all, wouldn’t we rather approach a client with some understanding of the power (and risks) of MR (learned through a DIY approach perhaps) and who is ready to go ‘pro’ through an agency? I am yet to meet the agency celebrating and promoting the use of DIY research.

  2. Trevor Cooper says:

    I couldn’t agree more about the testing and peer reviewing of new techniques. Just because they provide an answer doesn’t mean it is the right one.

    I would also like us to pause and consider whether the use of existing techniques are rigorously applied.

    How many times have you seen small r squared’s and little variance explained in multiple regression models and the ANOVAs family and have still been forced to use the results commercially because you promised the client a result?

    Perhaps it is time for a good hard look at the industry.

  3. Edward- The idea that “MR tools are becoming commoditized, that we should concentrate on the impact, not the method” is the profound and only answer to how we, as professionals, continue to explain our value to clients. I also concur with Lucas that we could benefit from embracing the DIY trend among clients in a similar manner. I attended a presentation/round table on DIY methods and trends at a recent MRA symposium in Washington, DC. Clients like the idea for its obvious cost savings’ but they really are too busy to do the job as well as we can. Plus, they fear the precedent this sets when those outside the company’s marketing areas do what they call “going rogue” ans cause even greater trouble by misinterpreting or misapplying results from poorly-designed studies. This can have a net negative PR effect for legitimate MR. It seems time for us to own the issue and take the leadership role.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>