Analysis Framework: Vaccine News Quality

How high-quality is this information?

Last updated: June 30, 2022 (Version 1)

This framework was developed to assess the quality of a given news article — vaccine reporting in particular. The framework is grounded in a questionnaire: recognizing that specialized topics benefit from the specificity of answerable questions, we developed a questionnaire that evaluates news quality through a layered approach. 

This layered framework for evaluating news quality is visualized as three concentric circles; the circles represent how each dimension of quality evaluation builds upon the other.

  • Dimension 1: General Journalism Quality: This foundational dimension assesses the general journalism quality when the piece aims to be journalism.
  • Dimension 2: Science/Health Journalism Quality: The second dimension assesses the science/health journalism quality when the piece mentions health and/or well-being, and/or makes mention of science or scientific methodology, 
  • Dimension 3: Vaccine Journalism Quality: The third dimension assesses the quality of vaccine journalism when the piece is about or discusses vaccines.
This figure is a visualization of the Vaccine News Quality Framework. The three circles on the left represent the three quality dimensions of general journalism, science/health journalism, and vaccine journalism. The boxes on the right represent the indicators associated with each dimension. Finally, the arrow on the far right shows how as the dimensions become more specialized, so do their associated indicators.

This figure is a visualization of the Vaccine News Quality Framework. The three circles on the left represent the three quality dimensions of general journalism, science/health journalism, and vaccine journalism. The boxes on the right represent the indicators associated with each dimension. Finally, the arrow on the far right shows how as the dimensions become more specialized, so do their associated indicators.

Then, each dimension is associated with a set of indicators, which are determined through questions. These questions capture characteristics commonly found in high-quality journalism, and encourages the reader to consider the accuracy, bias, genre, language, and transparency of the content and the content’s source. The questions provide general guidelines for reliability and credibility, and highlights key points for a reader to keep in mind when assessing the general quality of a news article. 

Overall, this evaluation should serve as a broad guide to helping writers, editors and readers feel confident in their vaccine news quality assessment skills. There is also a possibility this questionnaire can be used as a guide for new research in order to build news quality datasets through annotation and labeling, and to crowd-source automated techniques that can detect key quality indicators. Additional uses will be determined through further testing and future work in this area.

Indicators and questions can be seen in the expandable sections below. The formal questionnaire can be accessed either in an online version or as a PDF.

This foundational category of news quality is largely derived from traditional journalistic standards, and can be applied to news of any topic or genre. This first dimension represents an integrated approach to quality, emphasizing how the areas of accuracy, transparency, impartiality, reliability of sourcing, and language work together to create high-standard public information.

Indicator (Level) Indicator Question Citation
Accuracy (Article) Are there NO existing reputable fact-checks on this information that refute the article that you can find? 1, 3, 6, 13, 14
Multiple (Independent) Sources (Article) Does this piece base itself upon/cite more than one independent source of information (not just itself)? 9, 13
Circularity (Article) Does the piece avoid circularity in sourcing? 9
Claim Substantiation (Article) Does the piece substantiate the important claims it makes? 13
Press Release (Article) Is this piece NOT a press release? 9
News / Opinion Labeling (Article) Is this clearly marked as a news report or an opinion piece of journalism? 3, 7, 9, 15
Linguistic Grammar and Style (Article) Is this a well-written piece? 6, 13
Title Appropriateness (Article) Does the title capture the main point of the piece or report? 17
Sensationalist Language in Title (Article) Is the title generally neutral (not sensational) in its sentiment? 17
Reliable Source (Source) Does the article/report come from an outlet with a high factual reporting history (eg. Media Bias/Fact Check, absence within EN Wikipedia’s Perennial Sources list)? 10, 17
News / Opinion Separation (Source) Does the article or report come from an outlet that labels its opinion pieces differently from other news items? 7, 9
Transparency — Breaking News (Source) In cases of breaking news, does the outlet indicate that the situation may be evolving and information may be inaccurate or incomplete? 8
Disclosure (Source) Can you find pieces in the past year where the outlet has indicated author/reporter disclosure? 2, 8
Correction/Complaint Notification (Source) Is there a way to notify the outlet of the need for a correction or a complaint? 2, 8
 Evidence of Corrections Practice (Source) Can you find multiple instances where this outlet has recently issued a correction? Do not include corrections about breaking news. 2, 8

Science and health reporting, which deals with scientific issues and public issues involving science, can be understood broadly as “how society talks about science.” In this type of specialized journalism, it is critical to communicate often-complex scientific information in an accurate yet accessible way. Accordingly, this second dimension of quality includes questions regarding subject matter expertise, the inclusion of context, and the handling of uncertainty.

Indicator (Level) Indicator Question Citation
Science/Health Expert Reference (Article) Does this report refer to an expert in the specific field of scientific study? 11
Pre-Print Context (Article) If this report cites or links to an academic preprint, does it treat the reference differently than fully reviewed and longer standing research? 5, 11
Specialist Author/Editor Expertise (Article) Does the author have a background on the science beat or in science, or does the outlet/organization have a science editor/reporter? 11
Uncertainty (Article) If a scientific conclusion is discussed, does the article explain how evidence is built through testing and/or possible uncertainty about results? 5, 12
Correlation vs. Causation (Article) If the report mentions medical/scientific cause(s), are the terms or ideas about data in fact discussing causation correctly (as opposed to correlation)? 12
Science/Health Mission Alignment (Article) Is the main purpose of this piece to provide scientific research to communities, to educate the public, or to explain something scientific? 4, 12
Science/Health Outlet (Source) Does the outlet self-identify as a science outlet? 4
Science/Health Section (Source) Does the outlet have a section dedicated to science? 4
Science/Health Department (Source) Does the outlet have a desk, department, or editors dedicated to science? 4

Vaccine journalism, as a highly unique type of science communication, requires additional considerations.This third dimension of quality considers the use of specialized sources, the avoidance of equal presentation of scientifically unequal claims (“false balance”), and the representation of (un)certainty about the strength of scientific evidence for or against a particular risk.

These indicators are new contributions to news quality standards. They were derived from conversations with science and public health journalists and members of the World Health Organization’s Vaccine Safety Net (VSN), and validated through review workshops. We plan to continue to validate these indicators through additional testing and research.

Indicator (Level) Indicator Question
Vaccine Confidence (Article) Does the article/report express confidence in the overall efficacy of vaccines?
Risk Context (Article)  If discussing vaccine risks, does the article/report contextualize them appropriately (including risk of disease)?
Vaccine Safety Net Source (Article) Does the article/report reference at least one of the WHO’s Vaccine Safety Net (VSN) list of vetted websites as a trusted source of vaccine information?
Vaccine Topic Variety (Source) Does this outlet frequently publish stories about different kinds of vaccines and not just one (e.g., smallpox, the Zika virus, or COVID-19)?

Note: We are open to feedback regarding these indicators and questions. Please contact us at artt [dot] hackshackers [dot] com with the subject line “Vaccine News Questionnaire.” 

Methodology and References

The development of this framework for evaluating vaccine journalism quality was informed through academic research, consultation with journalists and technologists, and input from collaborative workshops. The structure was first derived from earlier work on classification around news taxonomies related to scientific reporting, which emerged from panels hosted by the News Quality Initiative in 2020 that built upon earlier work in the field.

Drawing on these findings, the ARTT team determined an initial set of indicators and drafted a corresponding questionnaire. Experts in a variety of related fields were then consulted for review and feedback, either directly or in semi-structured workshops. This group included science and public health journalists, members of the World Health Organization’s Vaccine Safety Net (VSN), experienced editors within the Wikipedia community, and a stakeholder from a technology platform. Related questions were also reviewed by participating Wikipedians in two workshops in early 2022. The questionnaire was also presented in a workshop at the International Journalism Festival (IJF) in Perugia, Italy in April 2022. Throughout this iterative process, questionnaire categories and indicators were continually revised and validated, resulting in the first version of this framework.

We would like to thank our partners at MuckRock Foundation, the Vaccine Safety Net (VSN) of the World Health Organization, and Wikimedia DC for their support and input. We would also like to thank the individual participants of our workshops and consultations, as well as the attendees at IJF Perugia.

[1] Anderson, Peter James. 2013. “Application of a News Quality Monitoring Methodology.”

[2] Bhuiyan, Md Momen, Hayden Whitley, Michael Horning, Sang Won Lee, and Tanushree Mitra. 2021. “Designing Transparency Cues in Online News Platforms to Promote Trust: Journalists’ & Consumers’ Perspectives.” Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2): 1–31.

[3] Bogart, Leo. 2004. “Reflections on Content Quality in Newspapers.” Newspaper Research Journal 25 (1): 40–53.

[4] Chen, Caroline, Randi Hutter Epstein, Jennifer Kahn, Katherina Thomas, Rick Weiss, Wudan Yan, and Yemile Bucay. 2021. “Towards Healthier Science and Health News Feeds: NewsQ Panel on Science and Health Journalism.” NewsQ (blog). January 26, 2021.

[5] Fleerackers, Alice, Michelle Riedlinger, Laura Moorhead, Rukhsana Ahmed, and Juan Pablo Alperin. 2021. “Communicating Scientific Uncertainty in an Age of COVID-19: An Investigation into the Use of Preprints by Digital Media Outlets.” Health Communication 0 (0): 1–13.

[6] Gladney, George Albert. 1996. “How Editors and Readers Rank and Rate the Importance of Eighteen Traditional Standards of Newspaper Excellence.” Journalism & Mass Communication Quarterly 73 (2): 319–31.

[7] Harloe, Kate, Nancy Ancrum, Karen Attiah, Maria Bustillos, Sewell Chan, Elena Gooray, and Arwa Mahdawi. 2021. “Separating Quality News from Quality Opinion,” NewsQ.

[8] “JTI | Journalism Trust Initiative.” JTI Website. Accessed June 2, 2022.

[9] Kahn, Gabriel, Meredith Clark, Al Cross, Claudia Irizarry Aponte, Mandy Jenkins, and David Kroman. 2020. “The Local News Landscape Is Broken: NewsQ Panel Review of Platform News Products.” NewsQ.

[10] Liotsiou, Dimitra, Bence Kollanyi, and Philip N. Howard. 2019. “The Junk News Aggregator: Examining Junk News Posted on Facebook, Starting with the 2018 US Midterm Elections.” ArXiv:1901.07920 [Cs], April.

[11] Ordway, Denise-Marie. 2020. “Covering COVID-19 and the Coronavirus: Harvard Professor Offers 5 Tips.” The Journalist’s Resource (blog). March 6, 2020.

[12] Secko, David M., Elyse Amend, and Terrine Friday. 2013. “Four Models of Science Journalism.” Journalism Practice 7 (1): 62–80.

[13] Shapiro, Ivor, Patrizia Albanese, and Leigh Doyle. 2006 “What Makes Journalism ‘Excellent’? Criteria Identified by Judges in Two Leading Awards Programs.” Canadian Journal of Communication 31: 21.

[14] Shapiro, Ivor. 2010. “Evaluating Journalism.” Journalism Practice 4 (2): 143–62.

[15] Vincent, Subramaniam, Patricia Lopez, David Agraz, Leona Allen Ford, Jon Allsop, Rochelle Riley, and Rebecca Traister. 2020. “Our Opinion: Recommendations for Publishing Opinion Journalism on Digital Platforms.” NewsQ.

[16] Vir, Jason. 2021. “The Relevance of Impartial News in a Polarised World.” Reuters Institute for the Study of Journalism, University of Oxford.

[17] Zhang, Amy X., Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B. Adams et al. 2018. “A structured response to misinformation: Defining and annotating credibility indicators in news articles.” In Companion Proceedings of the The Web Conference 2018, pp. 603-612. https://doi/abs/10.1145/3184558.3188731.