Thank you Martin for a really informative and objective post. I’ve copied it to my own archives and will use it as a reference.

Martin, just to clarify about the use of the iPhone; presumably it was handed over to the respondent and they filled the questionnaire out on there, rather than being asked questions?

The first time I read through I didn’t get that, and thought it was a sort of online poll only available to people who had an iPhone.

@Atropos – Thanks 🙂

@ekcol – Yes, it was handed over to the respondent! Apologies if that wasn’t clear.

Awww, Martin, this was a good article until the end.

You don’t have a background in social sciences, and have consequently ventured outside your expertise and given dodgy advice – particularly disappointing as there are some easily accessible guides available online.

What you have actually done is a list of ‘Excuses You Can Use To Dismiss Research That You Don’t Like’.

Point 1 is a particular problem. “Where is the control? Am I seeing the full picture? If the poll is talking about the attitudes of one group, or was conducted in one time frame, do we know what the results of other groups or other time frames are?”

An opinion or attitudes survey is not an RCT. There may not be comparison groups, and you cannot give placebo in a survey. ‘Control’ is the wrong word to use. Longitudinal surveys – those conducted over a time period – are very difficult to carry out because you need access to the same sample population over a period of time. They cost a lot of money and are subject to considerable sample attrition. Just because a survey is not carried out longitudinally does not mean it can be dismissed. In the case of the CSC poll, it would have been helpful if we could see what the whole population, or a population of people of different faiths had said, but the fact that we can’t does not automatically invalidate the data – it does make a difference to interpretation, though.

Points 3 and 4 are also misleading or incomplete, but having typed, at great length, some very dull explanations, I think they’d bore everyone rigid. Have some references instead.

This isn’t a bad preliminary guide to the points you were trying to make:

http://www2.sfasu.edu/polisci/Abel/PollEvaluation.html

And here is the slightly longer guide from the British Polling Council – point 13 is particularly pertinent.

http://www.britishpollingcouncil.org/questions.html

Bah, double post thanks to personal incompetence. I blame the Daily Mail.

Martin, this is just to point out that you are wrong.
https://liberalconspiracy.org/2009/05/04/smearing-british-muslims-through-opinion-polls/#respond
The CSC results are available online and they did have a control sample – 632 Muslim students were asked the same questions as 831 non-Muslim students. There are some pretty strong contrasts between what the two groups said.

For example, on the question of killing in the name of religion:
Yes in order to preserve and promote that religion: Muslim students, 4%, non-Muslim students, 1%
Yes but only if that religion is under attack: Muslim students, 28%, non-Muslim students 1%
No it is never justifiable: Muslim students, 53%, non-Muslim students, 94%
Not sure: Muslim students, 15%, non-Muslim students 4%

And the survey was carried out by YouGov so you can be pretty sure that it was representative.

Which leaves press coverage, I agree with you on that. It is ridiculous.

Interesting to see that the results are online now – they’ve obviously moved them since last year as I cited them then.

Regarding the control, no they don’t have a control sample. A control in science doesn’t just mean “a different group”, it means controlling for specific confounding variables – something which they have failed to do. The survey is also not representative – it is restricted to university students.

As an example of what I mean by failing to control above, note that there is for example no comparative figure for Christian students. Note also that Christian students in the UK aren’t a minority, and don’t tend to have as many friends and relatives in war-torn areas. A more scientific comparison would look at a Christian minority in a country like Turkey perhaps.

Note: Reading back through the article, I can see didn’t make that control point very clear – apologies.

Martin, you’re claiming that a survey of Muslims in the UK should be compared with a survey of Christians in Turkey?

You’d regard Turkish Christians as a ‘control group’?

The point of a ‘control group’ is that all the other conditions should be identical other than the one you wish to measure: Turkey is hardly the same as the UK for about a million reasons.

After a reasonable article you are suddenly defending into twaddle.

Sorry, ‘decending’.

I hope those surveyed by iPhone remembered to switch off their auto-corrective text.

Martin, it was a poll of university students, the headlines were about university students and the CSC survey was called “Islam on Campus”. The results are, therefore, only representative of students. Nobody has made the claim that they are representative of all Muslims. For that reason, your comparison of the CSC results with those of the iPhone polls in Birmingham is invalid.

I think the CSC website has been redesigned recently, but I find it a little surprising that you did not look at the “Publications” section of their website.

But I take your point about not breaking the results up by religion and allowing an exact comparison. It would have been preferable if they could have looked at representative groups of Christian, Hindu, Sikh, Jewish or whatever university students and then contrasted them with Muslim students.

I don’t think you necessarily need a control group for a survey – generally speaking surveys are designed to asnwer the question ‘what % of people agree to A or B in population C’ – and the standard sample sizes used (around 1000 people) are intended to give an answer with 95% chance of being +- 3%.

In the case where you then want to claim that Muslims (say) are uniquely extremist, then you’d need a control group (usually the rest of the national population would seem reasonable to me). But then you’d need statistical tests for significance – and that’s way beyond the usual statistical competence of polling organisations and their media employers.

Unfortunately, this magical ‘around 1000 people’ convention leads polling companies to pretend that, say, 700 people is close enough to 1000, to ignore the fact that the +-3% only applies if they have randomly sampled the population concerned (which they almost certainly haven’t, and weighting by demographics doesn’t make that any better – indeed it may make it worse – YouGov, for instance, can only poll people with internet access – giving more weight to those poor people who do have internet access does not make up for this, those individuals may well be highly unrepresentative of poor people without internet access, in which case you make the situation even worse), and to think that they can then slice up the sample into teeny tiny subsets and give the figures for multiple choice answers (A, B, C, D, etc) and still pretend that the errors are +-3%, when they are often +-10% or more, they’d need 1000 people per subset for that to be true.

But, of course, we all know that the key to a good headline grabbing survey is (a) to have massively ambiguous and preferably leading questions, and (b) not to publish the data or questions in full, so that (c) people can interpret the questions to mean things that may well not have been intended by the respondents (or even questioners) but no one can check.

RS is right – YouGov were regarded with suspicion from their inception because of the self-selection bias in their methodology. That said, they are better than they were.

He is also right on the fact that you do not necessarily need a control group for a survey to be good and for a variety of reasons it is not always possible. Many people underestimate the time and expense of good qualitative research, and for a face-to-face interview with good questions, it is not uncommon (at all!) for individual surveys to take more than half an hour. It thus becomes clear how difficult it is to get qualitative research that meets quantitative research standards.

The Opinion Poll Evaluation Guide at the end of the article is also somewhat misleading and definitely incomplete. For those wanting an expert view, might I suggest this: http://www.britishpollingcouncil.org/questions.html and this: http://www.mrs.org.uk/media/opinionpolls.htm – they also cover the codes of conduct expected by survey organisations. Note that a survey organisation who is part of the British Polling society, is bound to send you the questionnaire of a survey if asked.

They will equip you with the basics to tell the difference between a poll you just don’t like the look of, and one that is poorly conducted.

YouGov is far from uniquely compromised in that regard though. Conventional polling agencies carry out surveys during the day when, let’s face it, a very particular subset of the population are available.

@Gsirrah “Nobody has made the claim that they are representative of all Muslims.”

Yes, they have – note the Daily Mail headline.

@ Shatterface “Martin, you’re claiming that a survey of Muslims in the UK should be compared with a survey of Christians in Turkey? ”

No, I’m not. My point is that the survey is inadequately controlled. The results are being used to portray Muslims as more violent than other groups, but it is impossible to draw that conclusion from the evidence presented.

@ Ken McKenzie

The final few points aren’t intended to be a comprehensive guide to evaluating primary data from opinion polls, they’re just presented as a few basic questions to ask yourself when you see an article based on an opinion poll in the press, and while I appreciate I’m not a social scientist, I don’t really see where your objection lies with them.

If you see an opinion poll that is used to make a claim about Group A, then I think it’s very important to see if that conclusion is backed up by adequate comparison with other groups. That’s just common sense, in any field. As is making sure that the questions reported match up to the questions that were really asked in the poll – clearly they don’t in the case of many newspaper articles – seeing what sort of questions were asked, and whether the sample was representative. I’m sure there are others you can add, out this isn’t supposed to be a guide for Ph.D. sociologists doing peer review, it’s just what to look out for in newspaper articles making claims based on opinion polls.

Thanks for taking the time to write such a well-informed and excellent piece.

Martin, my objection is that you have set yourself up as an authority, by giving advice on an area where you do not yet have the expertise to give advice.

So, for example: “If you see an opinion poll that is used to make a claim about Group A, then I think it’s very important to see if that conclusion is backed up by adequate comparison with other groups. That’s just common sense, in any field.”

No, because if the survey is sampling Group A, assuming a good sample, it is perfectly adequate to make claims about Group A.

You compound this error in your reply in post 17. “The survey was inadequately controlled”. No. No it wasn’t. All you need say is your final sentence in post 17 – that the results are not adequate to reach the conclusion that Muslims are more violent to reach the conclusion.

You are eliding the survey itself with the reporting of that survey. This is a subtle but important error. If you want to accuse social scientists of methodological error, you need to be very sure of your ground.

@Ken “if the survey is sampling Group A, assuming a good sample, it is perfectly adequate to make claims about Group A.”

I think we’re talking at cross-purposes. The issue is that you can’t make a claim about Group A relative to any other group. You can say that e.g. “group A tend to like red”, but you can’t say that group A have a particular liking of red that other groups don’t have, and that’s all that’s being said in this piece and the subsequent comments.

@Ken “You are eliding the survey itself with the reporting of that survey”

Absolutely I am. The poll was commission specifically to provide the newspaper articles. From the point of view of an exercise by CSC, it was inadequately controlled. From the point of view of YouGov, they simply went and asked the questions they were told to ask – YouGov did not design this poll, the CSC did, and they did it in order to draw conclusions that are not supported by the results.

@Paul Sagar: “Thanks for taking the time to write such a well-informed and excellent piece.”

Many thanks 🙂

I’m from Alum Rock in Birmingham. I remember seeing 2 guys doing this poll a couple of months ago. They were asking people if they wanted to take part in a survey.
It was a bit weird because we don’t usually get people doing surveys in this area, especially by a white guy and a muslim guy together. (Although it was nice to see).

Some people here seem to think the poll was carried out only using people who had iphones. That’s not true. What they were doing was handing an iphone which had the questions on it. You then just clicked A, B or C to each question. It was pretty cool.

I was a bit shocked that they were trusting people with their iphones. Alum Rock isn’t exactly known for being crime-free. I wasn’t sure about them at the time but now I know what they were doing, I say good on them. Its about time our real opinions were counted. We just want to fit in.

RS, YouGov used to have some very specific population issues, unique to them, which predisposed them towards getting certain responses.

I understand that they have ironed them out, though. Agreed with the other issues of survey response bias.

Annoyingly, because of my line of work, I am both very keen to participate in opinion polls, and barred from doing so.

Martin – good article, but then I think the Centre for Social Cohesion has an agenda anyway, as do people like Policy Exchange who commission these kinds of slanted polls every couple of years.

There is a lot more to the CSC that I would like to expose and plan to do so here soon. Douglas Murray isn’t a character I’d really listen to on issues to do with social cohesion since he uses that agenda to push his own goals, which are at cross-purposes with what he claims is his agenda.

Martin, you are a very strange person to discuss things with, you veer from the insightful to the wildly inaccurate. The headline (as cited by you above) is “One third of British Muslim students say it’s acceptable to kill for Islam.

And later in this piece you write:

The Daily Mail headline I mentioned above claims that a third of British Muslims think it’s “acceptable to kill for Islam,”

No it doesn’t. It makes claims about Muslim students.

I came across a poll a few months ago,which appears to be extensive,comprehensive and covers,Muslim attitudes towards Western society,and Westerners attitudes to Muslims.
It covers many countries,and interesting comparisons between attitudes from Muslims in both Western and Muslim societies.

http://www.pewglobal.org

Europes Muslims More Moderate; The Great Divide; How Westerners and
Muslims View Each Other.

Released June,2006
The Pew Global Attitudes Project
A Pew Research Center Project

Non-Partisan “fact tank”
Washington D.C.

It would be intersesting to hear if any polls/ surveys have been taken recently on this subject by non-right wing / conservative / neo-con / organisations.
If not,then that in itself should raise a few interesting questions.

This might also assist as an interesting comparison;

Taken from; Capacent A / S. Research / Consulting.
Survey taken for DR,News ( Danish Television )

Young Muslims generally have the same attitude to the family model of education and democracy as their parents.
Though the younger generation are much better educated than their parents generation,there is no big difference in their attitudes on a series of important issues.

Denmark,
55% think critizing of religion should be forbidden.
64% support curtailing of free speech.
60% don,t go to a mosque,imams not representative.
18% want to see Sharia law implemented.
and….wait for it……90% would vote for the left,which means we don,t need a cystal ball to figure out ( taking U.N demographic statics into account ), what our futuristic Utopia will be like in 2030,and which political wing is going to win every election.
A strange alliance aye,until one Utopian vision has no use for the other.
And we all know who is going to win that one.
Should this game of cross your fingers,touch wood and swear on Tiny Tims Weekly,
that this dreaded future worse scenerio cannot conceivably take place,be little more than just a slower version of “Russian Roulette”,your apologies Martin Robbins,will not be accepted.
I know which right wing ideology I am more worried about.

The problem here is the spin applied to the results to the YouGov survey, not the survey itself. I would particularly take issue with the idea that the fact it was commisioned by the CSC casts doubts on the results: Reputable pollsters typically don’t reveal who commisioned the poll until the end (if at all) and the commisioners have no involvement in its actual conduct. The poll must be taken on its own merits, and there is no obvious flaw in its methodology — the wording and order of the questions are reasonable in the context of a poll of Muslim students on their attitudes to politically controversial aspects of Islam, and YouGov’s methods are better suited than those of traditional pollsters for reaching such a small target population (how many randomly dialled phone numbers would you need to get 632 Muslim students?). Unfortunately the BBM survey measures up very poorly; It won’t be widely reported because it doesn’t deserve to be reported — enthusiasm and a large sample size don’t make up for an utter disregard for the basic principles of social research. Despite the biased and sensationalist reporting of the YouGov poll by the Daily Mail et al. it *does* reveal significant differences in the attitides of muslim and non-muslim students on certain questions, and the best response to this is not to try to rubbish the underlying data but instead to rebutt their interpretation of it by, for example, pointing out that the vast majority of British muslims have no actual involvement in Islamist violence.

Trouble is, every time anyone on the left produces evidence that negative attitudes towards Islam are the product of the machinations of a hostile media, rather than inherent features of Islam itself, along comes some Islamophobe to throw a spanner in the works:

http://www.independent.co.uk/opinion/commentators/yasmin-alibhai-brown/yasmin-alibhaibrown-whod-be-female-under-islamic-law-1678549.html

I thought that this article made some points that were well worth reading, in an amusing and engaging way. The biggest flaw about any opinion polling is the format of the questions, the order the questions are asked and the interviewees perception of what is expected of them. The Yes Minister clip should be required viewing for anyone who is about to have their opinion canvassed. And intelligent people will always spin public opinion polling to their own agenda. That is the nature of media.

It would be quite amusing to have a survey button on here. Was this a good article yes / no?

I’d vote yes, but you knew that anyway.

Worthwhile topic, though I sense some shooting of the messenger.

Slightly strange emphasis on methodological rigour on the one hand, while offering the “four guys from Alum Rock” survey on the other.

There were some odd results there. 98% would accept a gay child, though heaven forfend they date a white kid – 53% oppose that!

The Pew data cited above – a very respectable and long-established charity – seems to suggest that there is less of a problem than the CSC, though a problem nonetheless.

In fairness to the 28% of Muslims who said they would fight in defence of their religion – or members of any other religion facing a similar situation for that matter – if I’d been asked whether violence would be justified in defence of my rights as an atheist, I’d have also said yes too.

If the teaching of Darwin was outlawed, if blasphemers were imprisoned or executed, if I was forced to pray to a god I do not believe in, or deny the rights of women and homosexuals, or made to grow a beard (especially the latter: they itch) then I’d say yes, it would be justified to fight.

The question is whether either religious groups are facing such repression – and in the West, at least, they are not.

Many thanks, Martin.

No doubt one or other of the MSM will seize on this poll to announce that a majority of young British Muslims don’t think that white kids are good enough to marry theirs. (The poll says 52%)

Perhaps of more interest is the expectation that British Muslims will have effectively integrated (i.e. secularised, if the Jewish experience is any precedent) in three generations – say sixty years. The fact that only 27% look to their religious leaders to “speak for” the British Muslim community (only twice as many as those who expect a non-Muslim to do so!) suggests that this may be an over-estimate of the timescale.

A shame, given the sample size, that there are no subsets (e.g. gender of respondent, education level/employment status, location of family roots etc) but hopefully this will be published later or collected in future surveys.

What we need is a Govt inquiry. I suggest Baroness Uddin, Labour’s first Muslim Peer, should lead it.

http://www.insidehousing.co.uk/story.aspx?storycode=6504453

Ala Uddin is Spitalfields Housing Vice-Chairman.

This seems on topic. My first thought was to ask whether the issue of self-identity as a Muslim was cultural or religious, and whether this is the same in each country.

Trofim:

“Trouble is, every time anyone on the left produces evidence that negative attitudes towards Islam are the product of the machinations of a hostile media, rather than inherent features of Islam itself, along comes some Islamophobe to throw a spanner in the works”

I wonder if these people may actually have played a bigger role in creating ‘negative attitudes’ to Islam. This list could have been 10 times as long…

British Convert To Islam Reveals Plot To Kill Saudi King
ww w.rttnews.com/forex/gblnews.asp

Afganistan’s Prez Karzai a Misunderstander of Islam? Signs law “legalizing rape in marriage”
ww w.aina.org/news/2009030793841.htm

Afghanistan: Misunderstander of Islam kills 30 on Kabul bus
news.yahoo.com/s/afp/20070929/ts_afp/afghanistanunrestblastkabul_070929115207;_ylt=AvxXSyk_PWbo53tX2sy_PkrOVooA

Singapore: Misunderstander of Islam admits to airport crash plot
ww w.adnkronos.com/AKI/English/Security/?id=3.0.2908517215

German Misunderstander of Islam gets 18 years in prison for synagogue bombing
ww w.google.com/hostednews/ap/article/ALeqM5ix8mre8AMWUk9p0n1UGQFIekhULgD965MHO81

Kenya: Misunderstander of Islam blows himself up while clutching Qur’an
ww w.abcmoney.co.uk/news/11200785567.htm

and more ‘Misunderstanders of Islam’ – strange that soooo many Muslims around the world misunderstand eh?

UK: Convert and Misunderstander of Islam gets life sentence for botched jihad attack
hww w.guardian.co.uk/uk/2009/jan/31/life-sentence-nail-bomber

USA: Ohio Misunderstander of Islam to plead guilty to joining Al-Qaeda and plotting to bomb resorts and military bases
news.yahoo.com/s/ap/20080602/ap_on_re_us/overseas_terror_plot

Canada: Misunderstander of Islam had detonators at home
ww w.canada.com/nationalpost/news/story.html?id=9642cd35-ceff-4c59-b006-bf3e60980b77&k=87068

Iran – Islami scholar and Imam: Ayatollah Khomeini said:
Islam makes it incumbent on all adult males, provided they are not disabled or incapacitated, to prepare themselves for the conquest of countries so that the writ of Islam is obeyed in every country in the world.

Islamophobes mount multiple suicide attacks in Afghani mosque — no, wait…
story.irishsun.com/index.php/ct/9/cid/2411cd3571b4f088/id/349617/cs/1/

France: Former imam misunderstands Islam, convicted in France for Chechnya-related jihad plot
ww w.kyivpost.com/bn/27830/

Detroit: Anti-Jewish, anti-Christian fliers found on cars – “Kill Jews and Christians if they don’t believe in Allah and Mohammad.”
ww w.macombdaily.com/stories/092807/loc_fliers001.shtml

UK: Allah “loves us to die and kill in his path”
edition.cnn.com/2008/WORLD/europe/04/10/terror.uk.ap/index.html?iref=mpstoryview