Unscientific Rubbish: Fox News Makes You Stupid? FDU’s Data and Methodology are an Incoherent Mess
Firstly, this entry will be prefaced by noting that The Reactionary Researcher (TRR) has no stake here; TRR has never and will never utilize FoxNews for his information concerning world events. For that matter TRR does not obtain his news from any television based sources.
A “study” has been making a big hit on the internet with its claim that “Some News Leaves People Knowing Less”. Within the first paragraph of this Fairleigh Dickinson University PublicMind Poll, the authors note that some news outlets,
especially Fox News, lead people to be even less informed than those who say they don’t watch any news at all.
This has been enthusiastically picked up by a variety of sources that have gleefully highlighted the point regarding FoxNews:
- Poll: Fox News Viewers Less Informed Than Those Who Read No News
- Surprise! Fox News Makes You Stupid
- Study Confirms That Fox News Makes You Stupid
Literally hundreds of other websites make similarly irresponsible, uninformed, and ridiculous claims. TRR is doubtful that any of these sources actually bothered to consult the original FDU poll.
Let’s straighten up a couple of things right off the bat: Firstly, this is no “study”, it’s a phone poll conducted by Fairleigh Dickinson University (FDU) that asked 612 random people in NJ exactly four questions, the answers to which were subsequently used to extrapolate this information.
Secondly, it is noteworthy that the fact that Talk Radio, the true news source for most conservatives, had an allegedly positive effect on people’s degree of ‘being informed’ went unnoted by any of the outlets that highlighted the story. Go figure.
Irrespective of what the results say, they are an incoherent and indecipherable mess, that as a consequence of being based on flawed methodology, are ultimately, meaningless.
TRR has reproduced some of the actual data from FDU PublicMind Poll below to illustrate the serious flaw in their methodology:
The instructions seem relatively straightforward:
“say “yes” if you got news from that source any time in the past week.”
Look at the data, note that each news source adds up to 100%, meaning that when they were on the phone with the respondents, the interviewer simply went down the list of sources, the respondent replied with a “Yes” if they had consulted any particular news source, and the interviewer simply scored it as positive or negative.
The implication of this is that nearly every respondent in this survey consulted more than one news source, and more importantly, the methodology employed by FDU did not isolate people who viewed only FOXNews, indeed it’s not likely there were many who viewed only a single source..
The statistics are right there: The Probability that a person views FoxNews and a consults a local newspaper can be calculated (statistically speaking, not in reality), and is 0.64 x 0.74, or about 47%. Similarly the probability that a person viewed Fox News, the Daily Show with Jon Stewart, and NPR is 0.64 x 0.18 x 0.21, or roughly 2.5%.
Allow me to re-emphasize the most important point: Because the respondents simply responded “Yes” when they had consulted a particular news source in the past week, it is impossible to isolate persons who consulted FoxNews and only FoxNews as a source of information. Had the interviewers isolated who viewed which single news source, the data associated with question one would be hard numbers, indicating the number of respondents who viewed each news source, instead we’re given relative viewing percentages.
In other words, given their methodology it is impossible to correlate knowledge of anything specific with FoxNews, or any other news outlet for that matter.
Given their methodology with respect to the first question, one wonders how they were able to extrapolate the information that was used to construct the table associated with question two, which is shown below:
To break these data down, or at least what the authors are attempting to imply with the data is that, for example, of those who listen to NPR, 68% believe the coup in Egypt was successful, 18% believe that it wasn’t, and 14% don’t know. Similarly, of those who watch The Daily Show with Jon Stewart, 68% believe the coup in Egypt was successful, 20% believe that it wasn’t, and 13% don’t know, etc.
Again, the problem with this is that the data provided by FDU don’t permit one to extrapolate this information. There is no information here regarding which information was derived from FoxNews, specifically.
The remaining questions and data suffer from this same fallacy.
The methodology employed by FDU as published in their online report, makes it impossible to discern which respondent derived what information from which source, and ultimately makes this poll and any and all associated conclusions, Unscientific Rubbish.
Rather than consulting FoxNews making a person stupid, reading, believing, and standing up for a “study” such as this, is the type of thing that will truly render a person, stupid.
The FDU study never passed the “smell test” for me, but you provided the reasons why.
Thanks.
Your criticism is formally correct, but you miss the point: The data does indeed show that people who watch Fox know less. (Irrespective of what other sources they consult.)
What you’ve written is true only given a dishonest assessment of the data. As my blog entry clearly indicates, the questioners weren’t separated based on which news source they consulted; the data regarding news sources was simply lumped together. That said, I’m perfectly willing to concede that I’ve made an error here, but you’ll need to point out, specifically, where I’ve made that error.
This was controlled for in the study:
–page 5
Of course I read that.
However, that has nothing to do with my complaint regarding FDU’s methodology. Either FDU’s methodology is seriously flawed based on their first questions, or they simply haven’t provided a full disclosure of their methods or results. In either case, it makes the study completely unverifiable, and completely pointless.
“Again, the problem with this is that the data provided by FDU don’t permit one to extrapolate this information. There is no information here regarding which information was derived from FoxNews, specifically.”
They say they use logistic regression to separate the responses, I have no understanding of logistical regression, but I am inclined to believe you can use it to do that unless you either demonstrate that logistical regression does not work or that they performed it wrong.
I do understand logistical regression, and the data they’ve published don’t permit one to undertake such an analysis. This isn’t to say that they didn’t do this, only that they didn’t make their data available, and didn’t provide a roadmap by which one might evaluate their conclusions. As I’m sure you’re aware, a hallmark of good science is repeatability, as well as clear methodology, and this poll has neither. Indeed, the data they’ve published are incoherent, and don’t permit one to determine anything at all. In fact, and I pointed this out in the blog entry, you can’t even extrapolate the information they said they did using the data provided.
You have a look at those data, and tell me how they correlated answers to Q1 with answers to any subsequent questions.
The FDU press release says the sample size is over 1,185 and national, not 617 people in New Jersey.
Sampling approach looks pretty good to me (a non social science researcher nor statistician, but I do work with them):
Random selection is achieved by computerized random-digit dialing. This technique gives every person with a land-line phone number (including those with unlisted numbers) an equal chance of being selected. Landline households are supplemented with a separate, randomly selected sample of cell-phones, including cell-phone only and dual cell-phone and land-line households, interviewed in the same time frame. The total combined sample is mathematically weighted to match known demographics of age, race and gender among the voting population.
The full methodology (here: http://publicmind.fdu.edu/2012/confirmed/final.pdf) also asks respondents to identify how much they used the different sources.
My analysis stands. Please note the methodology to which you refer was published in 2012, however the study I commented on was published in 2011. They may expanded or revised this study, but my analysis is correct based on what FDU published at the time.
I may take the time to have a look at what you linked, but my legitimate paid writing gigs are keeping me away from the blog currently.
Thanks for reading.