Tracer Newsletter #43 (10/02/20)- Twitter announces new policy addressing synthetic and manipulated media

Published in

Sensity

7 min readFeb 10, 2020

Welcome to Tracer, the newsletter tracking the key developments surrounding deepfakes/synthetic media, disinformation, and emerging cybersecurity threats.

Want to receive new editions of Tracer direct to your inbox? Subscribe via email here!

Twitter announces its new policy addressing synthetic and manipulated media on the platform

Twitter’s safety team announced its new policy for addressing synthetic and manipulated media on its platform, following a period of user consultation on how the policy should be implemented.

What does the policy state?

The new policy is the result of an initial period of user suggestions and a feedback round on a draft of the policy, as well as a consultation process with global experts. The final policy, which will be implemented from the 5th March 2020, is filed under the “Authenticity” category on Twitter’s rules and policies page:

“You may not deceptively share synthetic or manipulated media that are likely to cause harm. In addition, we may label Tweets containing synthetic and manipulated media to help people understand their authenticity and provide context”

In a blog-post announcing the new policy, Twitter explained the three-point criteria that will be used to assess whether a tweet violates the policy, and how offending tweets may be removed, labelled, have their visibility reduced, or be accompanied by contextual explanations in response.

A positive move, but one that may take time to be implemented reliably

The announcement of Twitter’s finalised policy follows Facebook’s own manipulated media policy announcement in January ahead of this year’s US presidential elections. Both acknowledge the significant threats manipulated media poses, and represent the first on-platform policy moves by social media companies to address these threats. While Twitter’s policy appears more comprehensive than Facebook’s on account of its inclusion of shallowfakes and crude forms of manipulated media, it has also been criticised for leaving significant grey areas, particularly surrounding a tweet’s context. However, Twitter acknowledged that there will likely be errors with the policy’s enforcement at first, but that they are committed to “doing this right”.

Facebook researchers create “radioactive data” that leaves an imperceptible mark on images generated by models that train on it

Facebook researchers published a paper detailing a new “radioactive data” technique for detecting whether a particular image dataset has been used to train a specific model.

How does the technique work?

The researchers’ approach was inspired by the analogous medical practice of using radioactive markers to see conditions more clearly on computer scans. The technique works by inserting an imperceptible identifying mark into an image, such that any images generated by a model trained on the dataset would also bear this mark. This mark can then be easily detected by the technique with a high degree of confidence, without the inserted identifying mark perceptibly impacting the images generated by a model. This differs from previous approaches that “poison” a dataset imperceptibly, but result in poor generalisation by models trained on that dataset.

A new way to potentially detect AI-generated synthetic images

The development of radioactive data provides a new way to track where training data is being used and how a model itself has been trained. The researchers emphasise that the utility of radioactive data lies detecting potential causes of biases across models, but also where data is being misused to train certain machine learning models. In the case of deepfakes, radioactive data could be used across datasets to help ensure that highly realistic synthetic images and videos generated are easy to detect via the identifying mark. However, it is likely that bad actors would circumvent such measures by simply creating datasets that do not include these “radioactive” identifying marks.

Google releases new “near-human” conversational chatbot Meena

Google released a neural network powered chatbot that it claims is the best conversational chatbot ever created, based on a new human evaluation metric that assesses how “natural” conversations appear.

How has Google supported its claim about Meena’s superiority?

The basis of Google’s claim is that Meena can learn how to generate context-appropriate and natural responses to almost any query made by a human agent. This is in part due to the model behind Meena, which consists of 2.6 billion parameters and was trained on 341GB of public domain social media conversations. Compared with OpenAI’s generative text model GPT2, this represents a 1.7 times model size increase, and an 8.5 times increase in training data. To test Meena against other chatbots, Google developed the Sensibleness and Specificity Average (SSA) a new crowdsourced human evaluation metric that they claim best captures a chatbot’s ability to engage in the “key elements of a human-like-multi turn-conversation”. In these SSA tests, Meena scored 79%, with the next best chatbot scoring 56%.

A powerful example of how convincing chatbots are becoming

Examples of Meena in action, including the joke seen in the above image, impressively illustrate how dynamic chatbots’ conversations have become. However, as previously seen with Microsoft’s Tay chatbot, they are susceptible to weaponisation in a variety of malicious contexts, such as spreading hate speech in high volumes or enhancing social engineering attacks. Google is yet to release a public demo of Meena due to a safety and bias vetting period, where safeguards will hopefully be introduced to address these potential misuses.

This week’s developments

1) Donald Trump tweeted an edited video that misleadingly showed US House Speaker Nancy Pelosi ripping up his state of the union speech in response to featured individual audience stories. (The Verge)

2) The US Federal Trade Commission (FTC) held an expert workshop examining the dangers of deepfake audio and synthetic voice cloning in relation to fraud and social engineering. (Venture Beat)

3) The US House Ethics committee issued a notice to House members warning them that posting misleading deepfakes or other manipulated content on social media may violate House rules. (The Hill)

4) Maine lawmakers proposed a bill that would prohibit the publication and distribution of political deepfakes 60 days before an election, mirroring a similar law passed in California. (Press Herald)

5) Youtube published a blog post outlining the platform’s policies for removing deceptive deepfakes and other forms of manipulated media ahead of the US Presidential elections. (Youtube)

6) A family lawyer claimed that deepfake voice audio was submitted as legal evidence against his client during a child custody case, in an attempt to make the client sound threatening. (Daily Telegraph)

7) Researchers from MIT created a new form of adversarial attack that tricks NLP systems into misunderstanding a piece of text by replacing words in a sentence with synonyms. (MIT Tech Review )

8) The Digital Forensic Research Lab released “Dichotomies of Disinformation”, an initiative for providing a shared language to categorise and describe different disinformation campaigns. (DFR)

9) Chinese state media and a government official spread a false image of an apartment building that they claimed show a newly constructed hospital in Wuhan for coronavirus victims. (Buzzfeed News)

10) Twitter introduced a feature that allows US users to report tweets containing voter suppression or misinformation to help “protect the public conversation” ahead of the US presidential election. (Politico)

11) Reuters announced a new proof of concept prototype for automatically generating news reports in real-time using synthetically altered iterations of pre-recorded footage of a news presenter. (Reuters)

12) Youtuber Denis Shiryaev synthetically upscaled and ‘sped up’ the 1894 black and white film “Arrival of the Train at La Ciotat” into a 4k 60fps version using neural-network powered tools. (Gizmodo)

Opinions and analysis

Making sense of deepfake policy

Aviv Ovadya outlines a “knowledge pipeline” framework for understanding the threats posed by deepfakes, and how the implementation of “levers” in this pipeline is key to mitigating these threats.

How activists can help inform effective synthetic media policy

WITNESS’ C orin Faife summarises activists’ reflections on the key factors social media platforms must consider when forming policies for countering the malicious uses of deepfakes and synthetic media.

A 2020 guide to synthetic media

Paperspace contributor Su dharshan Chandra Babu provides a primer on the different kinds of synthetic media, including a detailed overview of different synthetic video techniques and their applications.

Why we need social media platforms to be deepfake detectors

J ared Schroeder argues that social media platforms are best positioned to counter the threat of deepfakes, due to their policies avoiding some of the limitations facing government legislation.

Can synthetic media drive new content experiences?

BBC technologist Ahmed Razek outlines how the organisation is experimenting with synthetic media to create innovative content, specifically custom weather reports delivered by digital presenter avatars.

Why America needs a ministry of (actual) truth

Josh Wilbur argues that a new federal agency providing a “connecting tissue” between key stakeholders could help counter the threats posed by deepfakes and AI-generated disinformation.