by Charles Platt
I don’t think the Harris polling company has violated any laws, here. But it does look to me as if they have been, perhaps, unethical.
A widely quoted Harris poll claims that 57% of sampled Republicans think that Barack Obama is a muslim. This has been a great gift for liberal commentators who are predisposed to fear and loathe the Tea Party and its supposed influence. But did this gift just materialize from dispassionate and objective data, as if by accident? Or was it a premeditated attempt to create a misleading picture that would demonize a segment of the American public?
First take a look at the actual poll. Here’s the apparent source:
Browsing through it, we find that Republicans are not the only ones who doubt the president’s religion. Even among Democracts, 15 percent supposedly think that he is a Muslim. That translates as more than 10 million people, if we’re sampling the national population of around 70 million registered democrats. This doesn’t make sense to me. It cannot be true–and in fact, it isn’t true.
To find out why, we need to browse a little further.
Here, buried on page 4, is the methodology. It states that the entire poll was conducted online. In other words, this poll was not done by telephoning a random sample of the population and asking them to respond.
The online population is not representative of the general population. Therefore, we are starting with a non-representative sample, probably weighted toward those who have strong political opinions, both left and right, since political extremists seem more numerous online.
But the sampling problem has only just begun. There are millions of Internet users. How were the respondents chosen? Were they a random sample? No, apparently not. They seem to have been allowed to choose themselves. It says in the Methodology: “Respondents for this survey were selected from among those who have agreed to participate in Harris Interactive surveys.” In other words, the data are based on the views of volunteers.
I suggest it is extremely likely that people who hold strong opinions are more likely to want to talk about them than people who have moderate opinions or are simply not interested.
Worse still, Harris does not even reveal the process by which the final 2,320 people were chosen from among all the respondents. Harris simply says they “were selected.”
To me, this looks like a travesty of polling. It makes a final selection by means that are not revealed, from a group who just happened to respond to this poll, who are not necessarily representive of all the people who participate in Harris polls, who are not representative of Internet users, who are not representative of the general public.
If you conclude that the poll therefore has a very high probability of error–well, no problem! Harris agrees with you! In the Methodology section we find this artfully worded disclaimer: “Because the sample is based on those who agreed to participate in the Harris Interactive panel, no estimates of theoretical sampling error can be calculated.”
I take this to mean that there is no accuracy. This is not, and was never intended to be, a picture of American political opinion.
But it gets even worse, because when the people at Harris ended up with the numbers, they changed them.
The numbers do not reflect the population as a whole, but have been changed as if they do. The Methodology section states: “Figures for age, sex, race/ethnicity, education, region and household income were weighted where necessary to bring them into line with their actual proportions in the population.” So, for instance, if the selection of respondents didn’t happen to include many women, the prevalence of their views was apparently multiplied by a suitable factor–which of course is not revealed.
The press release does sneak in a reference in its third paragraph stating that the poll was of “2,320 adults surveyed online,” but the real meaning of this reference is never spelled out, and is outweighed by the document’s first paragraph, which draws attention to “the large numbers of Americans who hold extreme views of President Obama.” This implies that everyday Americans are the real subject of inquiry. Certainly that is how the polling data have been presented by commentators, and I have not seen any press release from Harris correcting this misapprehension.
Why did Harris do it this way? Well, “internet polling” is cheaper than telephoning people, but is that the only reason to do it? Surely the people at Harris must know that they will harvest more extreme views via this methodology, and since extremism was the focus of this poll (note the term “wingnuts” in its title), the methodology was virtually guaranteed to give the most sensationalistic results that anyone could hope for.
I don’t believe the people at Harris are stupid. I think that they knew what they were doing. Sensationalism sells, and thus the results were sure to be quoted widely. Does this mean the objective was just to get media coverage? Or could it be even worse than this? Did Harris specifically *want* to portray Americans this way, feeding the agenda of commentators who are eager to demonize anti-Obama groups such as the Tea Party?
I don’t want to think that a respected national polling organization has perpetrated such a piece of gross trickery. And indeed I would welcome any rebuttal disproving my conclusion. But given the disclosures in the poll itself, what other conclusion is possible?
