Vaccine News Quality

This framework was developed as part of the News Quality (NewsQ) Initiative to assess the quality of a given news article -- vaccine reporting in particular. Over the course of two years, NewsQ worked with journalists and health communicators to better understand what elements make for a better news article generally, for science/health journalism, and for vaccine-related news.

The framework is grounded in a questionnaire: recognizing that specialized topics benefit from the specificity of answerable questions, we developed a questionnaire that evaluates news quality through a layered approach.

We’ve arranged these markers into three categories of quality:

• Dimension 1: General Journalism Quality
  ‍This foundational dimension assesses the general journalism quality when the piece aims to be journalism.
• Dimension 2: Science/Health Journalism Quality
  ‍The second dimension assesses the science/health journalism quality when the piece mentions health and/or well-being, and/or makes mention of science or scientific methodology,
  
• Dimension 3: Vaccine Journalism Quality
  ‍The third dimension assesses the quality of vaccine journalism when the piece is about or discusses vaccines.
  

This layered framework for evaluating news quality is visualized as three concentric circles; the circles represent how each dimension of quality evaluation builds upon the other.

This figure is a visualization of the Vaccine News Quality Framework. The three circles on the left represent the three quality dimensions of general journalism, science/health journalism, and vaccine journalism. The boxes on the right represent the indicators associated with each dimension. Finally, the arrow on the far right shows how as the dimensions become more specialized, so do their associated indicators.

Then, each dimension is associated with a set of indicators, which are determined through questions. These questions capture characteristics commonly found in high-quality journalism, and encourage the reader to consider the accuracy, bias, genre, language, and transparency of the content and the content’s source. The questions provide general guidelines for reliability and credibility, and highlight key points for a reader to keep in mind when assessing the general quality of a news article.

Overall, this evaluation should serve as a broad guide to helping writers, editors, and readers feel confident in their vaccine news quality assessment skills. There is also a possibility this questionnaire can be used as a guide for new research in order to build news quality datasets through annotation and labeling, and to crowd-source automated techniques that can detect key quality indicators. Additional uses will be determined through further testing and future work in this area.

Indicators and questions can be seen in the expandable sections below. The formal questionnaire can be accessed either in an online version or as a downloadable PDF.
‍

General Journalism Quality

This foundational category of news quality is largely derived from traditional journalistic standards, and can be applied to news of any topic or genre. This first dimension represents an integrated approach to quality, emphasizing how the areas of accuracy, transparency, impartiality, reliability of sourcing, and language work together to create high-standard public information.
‍

Science/Health Journalism Quality

Science and health reporting, which deals with scientific issues and public issues involving science, can be understood broadly as “how society talks about science.” In this type of specialized journalism, it is critical to communicate often-complex scientific information in an accurate yet accessible way. Accordingly, this second dimension of quality includes questions regarding subject matter expertise, the inclusion of context, and the handling of uncertainty.
‍

Vaccine Journalism Quality

Vaccine journalism, as a highly unique type of science communication, requires additional considerations. This third dimension of quality considers the use of specialized sources, the avoidance of equal presentation of scientifically unequal claims (“false balance”), and the representation of (un)certainty about the strength of scientific evidence for or against a particular risk. These indicators are new contributions to news quality standards. They were derived from conversations with science and public health journalists and members of the World Health Organization’s Vaccine Safety Net (VSN), and validated through review workshops. We plan to continue to validate these indicators through additional testing and research.
‍

Methodology

The development of this framework for evaluating vaccine journalism quality was informed through academic research, consultation with journalists and technologists, and input from collaborative workshops. The structure was first derived from earlier work on classification around news taxonomies related to scientific reporting, which emerged from panels hosted by the News Quality Initiative in 2020 that built upon earlier work in the field.

Drawing on these findings, the ARTT team determined an initial set of indicators and drafted a corresponding questionnaire. Experts in a variety of related fields were then consulted for review and feedback, either directly or in semi-structured workshops. This group included science and public health journalists, members of the World Health Organization’s Vaccine Safety Net (VSN), experienced editors within the Wikipedia community, and a stakeholder from a technology platform. Related questions were also reviewed by participating Wikipedians in two workshops in early 2022. The questionnaire was also presented in a workshop at the International Journalism Festival (IJF) in Perugia, Italy in April 2022. Throughout this iterative process, questionnaire categories and indicators were continually revised and validated, resulting in the first version of this framework.

We would like to thank our partners at MuckRock Foundation, the Vaccine Safety Net (VSN) of the World Health Organization, and Wikimedia DC for their support and input. We would also like to thank the individual participants of our workshops and consultations, as well as the attendees at IJF Perugia.
‍

Note: We are open to feedback regarding these indicators and questions. Please contact us at artt [dot] hackshackers [dot] com with the subject line “Vaccine News Questionnaire.”
‍