Evaluation: Vaccine News Quality Questionnaire

Last updated: June 30, 2022 (Version 1)

For details on the development of this questionnaire, please refer to the Analysis Framework: Vaccine News Quality page.

  • The questionnaire is made up of three sections, with a series of questions within each section that pertain to either the article or the source.
  • When reading an article, keep in mind the two major aspects:
    • The sources of information mentioned in the article
    • The major claims the article is making
  • Answer by marking a “1” under either Y (Yes), N (No), or NA (Not applicable/Don’t Know).
  • Each question also has tips or “hints” to further explain the question and help responses to be more uniform across multiple people.
  • Tally and assess:
    • Tally the answers to gain a general sense of the quality of the news. The result will fall in one of three groups:
      • Great: High-quality (majority “Yes” answers)
      • Fine: Medium-quality (mix of “Yes” and “No”)
      • Not so good: Low-quality (majority “No” answers)
  • Questions are included for both article-level and source-level assessment.
    • For an article-level assessment, the questionnaire should take less than 10 minutes to complete. If a quicker assessment is needed, focus first on answering the “Key Questions” at the beginning of each section. If you answer “N” to any of these questions, there are most likely problematic elements to the article.
    • For a source-level assessment, the questions should be combined with several article-level assessments on a periodic basis. The aim of the source-level assessment is meant to identify outlets who have been able to provide quality reporting on a regular basis.

Note: We are open to feedback on this first version of the questionnaire; in particular, on the source-level questions and the scoring balance. Please contact us at artt [dot] hackshackers [dot] com with the subject line “Vaccine News Questionnaire.” 

Article-Level Questions

Key Questions Yes No NA
Are there NO existing reputable fact-checks on this information that refute the article that you can find?  
Search “fact check” and the URL of the article or video. Examples of reputable fact check are often conducted by member organizations of the International Fact Checking Network (IFCN), which has standards.
Does this piece base itself upon/cite more than one independent source of information (not just itself)?  
To answer this question, focus on the articles or studies referenced. Then, do a quick check to see if any of these sources are directly connected to each other by being at the same organization. (Definition of an independent source) 
General Questions Yes No NA
Does the piece avoid circularity in sourcing? 
To answer this question, check the source list to see if there are times where it looks like two different sources are being cited, but they are really the same source. (Definition of circular reporting)
Does the piece substantiate the important claims it makes? 
Is there at least one source that you noted above that is connected to a major point the article is making?
Is this piece NOT a press release? 
A press release is defined here. Some organizations directly put terms such as “press release,” “media release,” “for immediate release,” in the document. A press release might for example announce a recent scientific discovery made by a faculty member in a university, but is not required to double check or verify the discovery itself with external reviewers.
Is this clearly marked as a news report or an opinion piece of journalism? 
Look around the page or in the URL of the piece and see if you can find the words “opinion” or “news.” Even if you don’t see a clear label on the piece, look at the outlet to see if this falls under a news section. One way to look for this is also to see if you can find a clear opinion section, or clearly marked opinion pieces.
Is this a well-written piece?  
Tip: Typos? Convoluted? Hard to follow? If it overall reads clearly and contains no significant typos, grammatical errors, feel free to mark it as yes.
Does the title capture the main point of the piece or report? 
Does the title reflect the most important claim(s)?
Is the title generally neutral (not sensational) in its sentiment? 
Tip: Check the text with an online sentiment analysis.

Key Questions Yes No NA
Does this report refer to an expert in the specific field of scientific study?
For example, an article on vaccines should cite a vaccinologist, infectious disease, or directly related specialists, either directly or through cited references (and not rely on the word of a different kind of doctor.) In a similar vein, an article on the Holocaust should refer to experts in modern European history rather than a different era or location.
If this report cites or links to an academic preprint, does it treat the reference differently than fully reviewed and longer standing research?
Are any references within places such as arxiv.org, biorxiv.org or other places that collect pre-prints. A reference to a preprint should mention this fact and/or that its results have yet to be vetted by a peer community. Readers/listeners should understand that any conclusions need to be taken with some caution. 
General Questions Yes No NA
Does the author have a background on the science beat or in science, or does the outlet/organization have a science editor/reporter?
Look at the description of the reporter and also other reporters of science articles at the outlet/organization. Science reporting is complicated, and good reporting takes a combination of skill and experience. If the outlet/organization has a science editor or reporter that can review, guide, and/or edit pieces, this can also be helpful in producing quality scientific reporting.
If a scientific conclusion is discussed, does the article explain how evidence is built through testing and/or possible uncertainty about results?
Absolute certainty in science is very difficult, as it depends on the consensus of many participants who are reviewing results over and over again. This is why some conclusions can be overturned in the future. 
If the report mentions medical/scientific cause(s), are the terms or ideas about data in fact discussing causation correctly (as opposed to correlation)?
Explanations of causation need to make sure that evidence is greater than mere chance, and against the possibility that there are other factors influencing the results. Here’s an explanation of correlation versus causation to keep in mind from Khan Academy, an educational website.
Is the main purpose of this piece to provide scientific research to communities, to educate the public, or to explain something scientific?
The central purpose of the article should be public-oriented missions of science reporting to benefit the public and public health, rather than entertainment. 

Key Question Yes No NA
Does the article/report express confidence in the overall efficacy of vaccines? If the answer to this question is “N,” mark the “N” and then skip the rest of the questions in this section.
An article is problematic if it questions if vaccines overall have been shown to work, or describes those with concerns as having equal evidence for their doubts. For example, “false balance” in reporting can occur when unsupported, minority, or implausible views are given an equal voice in journalistic reporting with expert consensus and credible information. So, an article is problematic if it questions whether vaccines overall have been shown to work (there is broad, overwhelming scientific and medical consensus over the efficacy of vaccines), or portrays those who have concerns over the general efficacy of vaccines as possessing equivalent evidence that supports their doubts.
General Questions Yes No NA
If discussing vaccine risks, does the article/report contextualize them appropriately (including risk of disease)?
If this piece mentions risk, based on this piece, do you find yourself worried about taking a vaccine?
Does the article/report reference at least one of the WHO’s Vaccine Safety Net (VSN) list of vetted websites as a trusted source of vaccine information?
Look to see what organizations are referenced in the article. Check to see if one of these organizations is referenced/mentioned as a VSN member. (List of current VSN members)

Source-Level Questions

Key Question Yes No NA
Does the article/report come from an outlet with a high factual reporting history (eg. Media Bias/Fact Check, absence within EN Wikipedia’s Perennial Sources list)?
Check the outlet in MBFC or another vetted list for its rating.
General Questions Yes No NA
Does the article or report come from an outlet that labels its opinion pieces differently from other news items?
Even if you don’t see a clear label on this one piece, look at the outlet to see if this piece falls under the news section. You can also see if you can find a clear opinion section, or clearly marked opinion pieces.
In cases of breaking news, does the outlet indicate that the situation may be evolving and information may be inaccurate or incomplete? 
In the case of developing events (~48 hours), breaking news may not have full or accurate information. Look for 3 breaking news pieces in the past year to see whether the outlet indicates this uncertainty through updates and disclaimers.
Can you find pieces in the past year where the outlet has indicated author/reporter disclosure?
Tip: Search for “full disclosure” text on the outlet site and see what pieces appear.
Is there a way to notify the outlet of the need for a correction or a complaint?
Search for email addresses or forms on the outlet/organizational site connected to corrections. If this doesn’t exist, also see if this is possible through the general “Contact” information.
Can you find multiple instances where this outlet has recently issued a correction? Do not include corrections about breaking news. 
Search on the outlet/organization site for the terms “correction.” When the outlet indicates that a previous version of the article had incorrect information and amends the text. In addition, outlets may issue pieces to address egregious issues. Outlets with many articles are likely to need correction, and indicates that the outlet is committed to releasing accurate information. While there is no specific number of corrections, a general guideline is five instances in the past year.

General Questions Yes No NA
Does the outlet self-identify as a science outlet?
Look for a focus on science in the outlet’s/organization’s title or mission.
Does the outlet have a section dedicated to science?
Look at the sections of the outlet to see if there is one or more dedicated to science topics. Do not count special reporting related to COVID-19.
Does the outlet have a desk, department, or editors dedicated to science?
Look at the staff of the outlet/organization in order to determine if there are writers and editors assigned to science.

General Question Yes No NA
Does this outlet frequently publish stories about different kinds of vaccines and not just one (e.g., smallpox, the Zika virus, or COVID-19)?