ChangeWave survey validity should be questioned

Comments

When we wrote about ChangeWave’s most recent survey, we tried to make it very obvious that the survey was in no way indicative of any statistically accurate trends, and that it was to be taken with a grain of salt. In fact, the exact wording was:

“Since the survey sample is so small, and we can’t be sure of how representative the sample is of the total North American market, we should obviously take these results with a grain of salt.”

What should have been included in the post, was a better explanation of why the ChangeWave survey isn’t accurate.

It’s important to take into account when these surveys are published. The timing of these surveys reflects either a total ignorance on the part of ChangeWave regarding when devices are released, or a ulterior motive behind fudging the numbers.

Take for example the last survey which showed some very inaccurate data. ChangeWave’s survey polled 4,255 consumers between September 14th and the 21st. This is exactly the wrong time to poll users if you want to make any kind of BlackBerry and iPhone comparison. The poll date is a couple months after the launch of the iPhone 3Gs, allowing it to accumulate sales, and before the launch of several major BlackBerry devices (Curve 8520, Storm2 and Bold 9700).

According to the survey, 36% of respondents said they would purchase an iPhone, and 27% said a BlackBerry. With the iPhone 3Gs circulating among consumers, and the only newish BlackBerry around town being the Tour, of course their survey respondents are going to say the iPhone. If this study were done today, I can guarantee you that the numbers would favor BlackBerry.

The fact that ChangeWave computed smartphone OS market share based on their sample size of 4,255 (used to reflect a population of 304,059,724), says a lot about the validity of their survey.

According to ChangeWave, RIM holds 40% of the US smartphone market, while Apple holds 30%. Compare this to a respected research firm such as NPD group, who say RIM’s consumer smartphone market share increased 15 percent to nearly 50 percent of the smartphone market in Q1 2009 versus the prior quarter, as Apple’s and Palm’s share both declined 10 percent each.

The results are reminiscent of an equally unreliable source AdMob. AdMob published their August 2009 Mobile Metrics Report which was just laughable in its findings.

Aug-blog-pic

Again, the survey sample only includes data from the AdMob network, a sample size that is not reflective of the global smartphone market. Do you really think Apple has a greater worldwide market share than Nokia? The numbers are so off you wonder why they even bother publishing the results.

The most likely scenario here is that the “analysts” at ChangeWave are just lazy, and didn’t take the time to release a survey that provided accurate market share percentages and analysis. There are also some who think that ChangeWave has something to gain financially from misrepresenting Apple’s dominance.

The biggest problem with these surveys is that they are republished with no warning to their validity, and mainstream media picks them up, which could make the surveys a self fulfilling prophecy. The mainstream media, who write about technology as though it is something from outer-space, are easily duped by these bogus surveys. Just look at how the Financial Post republishes the ChangeWave survey with no critical eye whatsoever:

“While RIM’s BlackBerry devices are still the most popular smart phones for consumers and non-business users in the United States, Apple Inc. and its iPhone are closing the gap, according to a new survey from U.S.-based market research firm ChangeWave Research.”

We will continue to publish surveys by research firms such as ChangeWave, but unless they become more scientifically prepared and accurate, the next time we publish it won’t be pretty.

  • http://surveys.com.sg/ Singapore Survey

    Hello, I found this website when i was looking for web sites related to survey. I wanted to tell you, your site is good. I like the logo too, its refreshing. I don’t have the time at the moment to slowly read your blog but I have bookmarked it and I also subscribed for your RSS feed. I will be back in a night or two. Thanks for an informative web site.

  • http://surveys.com.sg Singapore Survey

    Hello, I found this website when i was looking for web sites related to survey. I wanted to tell you, your site is good. I like the logo too, its refreshing. I don’t have the time at the moment to slowly read your blog but I have bookmarked it and I also subscribed for your RSS feed. I will be back in a night or two. Thanks for an informative web site.