When good research goes bad

November 19, 2013

As Director, Brand Intelligence at CoreBrand, my focus when it comes to brands and brand strategy centers on finding the empirical evidence behind a brand, and what drives behaviors around that brand. Nothing excites me like doing quantitative research to reveal the underlying dimensions that influence, motivate or drive a successful brand or brand strategy.

Of course, for any type of quantitative research to be successful, it has to be done right, or there is a risk of doing irreparable harm. And if you’ve been around research for any length of time, you know that there can be all kinds of unforeseen errors, miscalculations, or simple mistakes that can derail it, resulting in seriously negative consequences.

One example of a seemingly minor detail that had a major negative impact on the research results was one that took place in 1992 when Roper Starch Worldwide conducted a poll for the American Jewish Committee (AJC), which was going to release the findings in conjunction with the dedication of the U.S. Holocaust Memorial Museum. In the poll results, it was reported that more than one out of five (22%) Americans believed it was “possible that the Nazi extermination of the Jews never happened,” and another 12% indicated they were unsure. In effect, the research suggested that fully one-third (34%) of Americans were not convinced that the deaths of millions of Jews at the hands of Nazis actually happened!

To say that the world found this finding shocking is an understatement. America was derided as a country filled with fools, and many called for the need to strongly educate Americans on the realities of history.

However, what was at the root of the surprising finding had less to do with America’s seeming indifference to the suffering of the Jews in WWII, and more to do with the way the research was conducted.

The question that was asked, and that was the basis for the result, read….”Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?”

Now think about that question for minute. How would you answer it?

For the respondent to answer in a way that affirmed their belief in the Holocaust taking place, they would have had to select what amounted to an answer that was a double negative response, i.e. that they thought it was impossible that the holocaust never happened!

Having discovered the convoluted wording, Roper then did a follow up survey that simply asked “Do you doubt that the holocaust actually happened or not?” The results? Only 1% answered that they had a doubt that the holocaust occurred, 8% were unsure, and 91% answered that no, they had no doubt that the holocaust had taken place.

Needless to say, the whole episode was quite an embarrassment for Roper Starch Worldwide, a very respected polling firm at that time.

But it also provides a sobering lesson in the need to not only think through the research being done carefully and completely, but to also pre-test it thoroughly in order to catch the type of error that occurred in the Holocaust Museum poll. Taking a short cut to save time or money may seem like a good idea at the time, but the consequences if something then goes wrong can be far reaching and damaging in ways no one ever imagined.

Have you experienced, or heard of other examples where flawed research impacted a brand?

blog comments powered by Disqus
Brandlogic and CoreBrand have become Tenet Partners — Where brand meets innovation®. More