Online Misinformation Harm Questionnaire

September 2022

The “Urgent” questionnaire is meant to help you make an assessment of the potential or relevant urgency of a specific piece of content according to certain dimensions of magnitude. Here are some guidelines and notes:

  • The questionnaire should take less than 10 minutes to complete. If a quicker assessment is needed, focus first on answering the “Key Questions” at the beginning of each section
  • Answer by marking a “1” under either Y (Yes), N (No), or NA (Not applicable/Don’t Know).
  • Feel free to answer “Not applicable/Don’t Know” for a question if, after reflection, you cannot say.
  • “Not applicable/Don’t Know” answers can be subtracted from Total “Yes” answers to gain a relative magnitude of urgency across different pieces of content.

For details on the development of this questionnaire, please refer to the Analysis Framework: Online Misinformation Harm. A .pdf version of the questionnaire can be downloaded here.

A piece of misinformation is more harmful the more that it spurs directly harmful actions. The questions in this section try to ascertain whether characteristics or factors related to the message(s) make the content likely to spur directly harmful actions, particularly physical ones.

 

Key Questions Yes No NA
Does the message content include an explicit call to action?
The message asks the reader/listener to directly do something; it might ask the reader to post, share, tell others about something, join an event.
Does the piece of content incorporate coordination efforts, such as dates/times or other arrangements for follow-up?
Does the message provide a name or otherwise any identifying information about an individual, an address, or a place of work in such a way that people might be directly harmed?
General Questions Yes No NA
Does the message content include a tone of urgency or mention of time sensitivity?
Does the message content include any threats of violence?
Violence includes mention of physical or emotional attacks on others, weaponry, assault.
Does the message lay blame or cast aspersions or hatred on a particular group, such as a particular religion, gender, sexual orientation, race, country, or culture, that has been harmed in the past by the audience of the content?
Does the message invoke a sense of injustice or moral outrage, including on behalf of a vulnerable individual or group such as children or women?
Does the direct target or current audience members directly addressed of the message have a recent history of taking actions that cause harm?
These are individuals or groups that may be on certain watch lists, and known more locally. Also consider doing an internet search for the name of the individual or organization and see what events occur in their recent history.
Is this message associated with/similar to other messages that are also actionable?
Look for linked messages, comments, or other associated discussion that score “Yes” with the above questions.

A piece of misinformation is more harmful the more the message seeks to exploit human or a group’s weaknesses, including a lack of resources. This dimension recognizes that factors can contribute to the target audience’s vulnerability to misinformation, ranging from emotional manipulation to a lack of available resources. The questions in this dimension strive to examine when aspects of the message directly engage in exploitation.

 

Key Questions Yes No NA
Does the message directly address or reference children or use language aimed at a younger audience?
Look for child-friendly themes such as toys or direct discussion about children.
Does the message introduce a degree of fear or feelings of uneasiness?
E.g., do you think the content predict any undesirable outcomes/consequences of an event, a decision, or a situation? Does the message employ any kinds of scare tactics?
Is the message content complicated to understand?
Do you have a hard time reading the content all the way through and explaining it to others?
Does the message directly address or reference elderly community members, or discuss topics aimed at them?
Look for generational cues such as retirement or health issues, or discussions around aging parents or relatives.
General Questions Yes No NA
Does the message directly address or reference military veterans, or discuss topics aimed at them?
Look for references to past military service, engagement in wars, language around those “who have served.”
Does the message make mention of a reader’s feelings of isolation?
Do you think this content appeals to/ address someone who may be isolated or experience feelings of isolation?
Does the message make mention of a reader’s feelings of powerlessness?
Does the message make mention of a reader’s feelings of disenfranchisement?
This can include discussion about one’s rights or vote being denied.
Is this message being shared by within an online group, community, or thread that has a recent history of discussing conspiracy theories or viral misinformation?
Look for recent posts by the same author/account and within the same social media space, to see if narratives include discussion of conspiracy or claims that have been fact-checked by reputable sources.
Is the language of the intended audience neither a UN language (English, French, Spanish, Mandarin Chinese, Russian) nor on the top 5 list of most popular languages?
Less popular languages may have fewer resources to corroborate or fact-check misinformation. A list of popular languages can be found here.
Is the message presented in a region where the local context might amplify its harm?
For example, anti-vaccine misinformation may be more harmful in countries where vaccine hesitancy is prevalent, and misinformation spreading communal hate could be more harmful in regions with existing social conflicts.

A piece of misinformation is more harmful the more platforms and people are exposed to it. This dimension ascertains whether characteristics/factors related to the message(s) make the content more likely to spread or be discovered. It focuses on questions related to magnitude of exposure or potential exposure, rather than analyzing the message for its credibility.

 

Key Questions Yes No NA
Is the content already spreading far and/or fast on a multitude of platforms?
Do the people or entities who are spreading the piece of content have a broad reach (size of following on social media, “influencer,” presence on TV or other news media)?
Are the people or entities known to be repeat spreaders of questionable information?
Search for the author/poster within a fact-checking website and see if their claims have come under review more than once.
General Questions Yes No NA
Is there evidence of coordination activity (whether bot/automated or not) to encourage spread?
Signs of coordination might include: searching for the title of the message/article in a search engine to find the same repeated text, or actual discussions, posts, or upvotes related to creating an online campaign.
Is the content publicly accessible (posted on a public platform, addressable URL)? Is the content posted on a popular platform?
Is the content spreading on multiple platforms?
Search the claim or the related URL link on a couple of search engines, and see if you can find it on more than one platform (e.g. YouTube, Twitter, Telegram, Facebook).
Does one of the platforms upon which the content is shared have tools to support amplification (e.g. reshares, algorithmic feeds, recommendation engines)?
Does the message make direct appeals to audience members that it in their financial, political, or social interest to spread the content further?
Does the message directly call audience members to share the content further?
Count this if there is an explicit call to share, repost, amplify. Do not count this if the only evidence of this is a social media button for sharing.
Is the tone of the content striking enough in ways that encourage sharing?
Things to look for include sensational language, humor, emotionally arresting imagery.
Does the content contain an image, audio-clip, or other richer formats that are easy to remember, visually or aurally arresting, or seems interesting to share?
Does the message impart a sense of exclusivity or novelty (“breaking news”)?
Are there hashtags associated with the message?
For example: #vaccinated. Hashtags are evidence of trying to associate the message with other social media streams.
Is the message difficult to fact-check or prove false?
Is the message related to a current event or a topic that is being reported on actively by many news outlets?
For example: look at a news aggregator such as Google News or Apple News to see what topics and stories appear near the top.

A piece of misinformation is more harmful the more believable its message is to a specific community. This dimension’s measures are related to topics in which either authoritative consensus is difficult to achieve or is affected by the perceptions from a specific community.

 

Key Questions Yes No NA
Is there a lack of high quality information that is publicly accessible and is refuting the message’s claim?
Do an internet search for the same claim and/or actors and see if there is another report by a mainstream or public information source.
Does the poster and/or organization/outlet have a noteworthy number of social media/community followers?
Is the content published by an organization/outlet with uncertain editorial control (e.g. is not a recognized news publisher)?
Check the outlet in MBFC or another vetted list for its rating.
General Questions Yes No NA
Is there a lack of consensus on the part of experts regarding the claim?
Look for discussion around books, articles by relevant credentialed experts in the field (e.g. scientists, historians) and find whether or not their is general agreement. For example, infectious disease experts agree that vetted vaccines are effective; historians agree that the Holocaust occurred.
Does the message fail to include external citations, links, or language about evidence to support its claim?
Look for whether the poster mostly links back to self or relies on its own witness.
Does the message contain richer formats as part of its evidence that lay people consider to have low falsifiability?
Image- and sound-based messages (such as video, audio recording, screenshot) are considered to have lower falsifiability than plain text.
Is the message written or communicated in a personal or persuasive tone?
Things to look for are personal appeals that include emotion, encouraging language, colloquialisms from a community.
Does the message make reference to the broad believability of the claim or topic?
Look for cues such as “everyone knows” or references to outlets or communities that agree with the poster.
Does the message appeal to a specific community identity by mentioning a shared set of values or beliefs?
Look for cues in which community identities are invoked, as well as specific values such as “freedom,” “justice,” “tradition.”
Does the poster have credentials that represents some kind of expertise?
Examples include academic credentials such as MD, PhD, JD, PMH, titles such as Professor, Reverend. They can also include unusual titles as long as the credential makes a claim about the expertise of the poster.
Is the content posted by an imposter individual or counterfeit outlet that could successfully pass as a different person/account based only upon a quick glance?
Examples of this are rare, but for example a website claiming to be ABC news that isn’t abcnews.go.com or abc.net.au or a Twitter handle that is close to a recognizable figure’s expected Twitter handle.
Does the content have the graphics and styling of a legitimate news agency or mainstream information source?
Look for high high-production values such as a professional looking banner or logos that seem similar to existing mainstream information sources, (e.g. national level outlets such as BBC, New York Times, Al Jazeera).

A piece of misinformation is potentially more harmful the more that it addresses, or is part of, a community’s societal, cultural or institutional relationships. This dimension measures how deeply embedded the content’s narrative is to the community’s existing context and history.

 

General Questions Yes No NA
Does the message fit into a larger narrative that has been existing for some time?
A larger narrative means that there is story touching the information that crosses platforms and time. This may include stories about communities, race, political parties.
Does the message question trust in or the functioning of public institutions?
Does the message question trust in or the functioning of the scientific community as a whole?
Does the message question the functioning of or trust in news sources/ the media in general?
For example, does the message encourage distrusting all online sources? Does it suggest that everything is not verifiable?
Does the message question the trustworthiness of other people in general within a community or society?
In a democratic country where there are elections, does the message directly attack the election process?