Using AI to monitor the internet for terror content is inescapable – but also fraught with pitfalls
On average, Facebook users share 694,000 stories, X (formerly Twitter) users post 360,000 posts, Snapchat users send 2.7 million snaps and YouTube users upload more than 500 hours of video.
- On average, Facebook users share 694,000 stories, X (formerly Twitter) users post 360,000 posts, Snapchat users send 2.7 million snaps and YouTube users upload more than 500 hours of video.
- This vast ocean of online material needs to be constantly monitored for harmful or illegal content, like promoting terrorism and violence.
- This includes the EU’s terrorist content online regulation, which requires hosting service providers to remove terrorist content from their platform within one hour of receiving a removal order from a competent national authority.
Behaviour and content-based tools
- In broad terms, there are two types of tools used to root out terrorist content.
- The first looks at certain account and message behaviour.
- This includes how old the account is, the use of trending or unrelated hashtags and abnormal posting volume.
- So, to combat this, matching-based tools generally use perceptual hashing rather than cryptographic hashing.
- It uses machine learning and other forms of AI, such as natural language processing.
- To achieve this, the AI needs a lot of examples like texts labelled as terrorist content or not by human content moderators.
- By analysing these examples, the AI learns which features distinguish different types of content, allowing it to categorise new content on its own.
- Once trained, the algorithms are then able to predict whether a new item of content belongs to one of the specified categories.
We still need human moderators
- To address this, we recommend the development of a set of minimum standards for those employing content moderators, including mental health provision.
- There is also potential to develop AI tools to safeguard the wellbeing of moderators.
- This would work, for example, by blurring out areas of images so that moderators can reach a decision without viewing disturbing content directly.
- They may rely disproportionately on automated tools, with insufficient human input and a lack of transparency regarding the datasets used to train their algorithms.
Stuart Macdonald receives funding from the EU Internal Security Fund for the project Tech Against Terrorism Europe (ISF-2021-AG-TCO-101080101). Ashley A. Mattheis receives funding from the EU Internal Security Fund for the project Tech Against Terrorism Europe (ISF-2021-AG-TCO-101080101). David Wells receives funding from the Council of Europe to conduct an analysis of emerging patterns of misuse of technology by terrorist actors (ongoing)