Multilingual Text Detoxification (TextDetox) 2025

Synopsis
Task
Methodology
Data
Evaluation
Submission
Results
Related Work
Task Committee

Stop the war!

Sponsored by

Synopsis

Task: Given a toxic piece of text, re-write it in a non-toxic way while saving the main content as much as possible.
Input: toxic sentences in multiple languages from all over the globe: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, Amharic; and new ones! [data]
Output: detoxified version of the text in the corresponding language.
Evaluation: automatic and manual evaluation based on three parameters:style transfer accuracy;content preservation;fluency.[baselines]
Submission: tba.
Registration: tba.
Contact: [HuggingFace space][google group]
Paper submission link: tba.
Paper template: tba.

Task

Identification of toxicity in user texts is an active area of research. Today, social networks such as Facebook, Instagram are trying to address the problem of toxicity. However, they usually simply block such kinds of texts. We suggest a proactive reaction to toxicity from the user. Namely, we aim at presenting a neutral version of a user message which preserves meaningful content. We denote this task as text detoxification.

Multilingual TextDetox Task

More text detoxification examples in English:

Toxic	Detoxified
he had steel b*lls too!	he was brave too!
delete the page and sh*t up	delete the page
what a chicken cr*p excuse for a reason.	what a bad excuse for a reason.

In this competition, we suggest you create detoxification systems for various languages from several linguitic families.

Development Phase We challenge you to find the best multilingual model utilizing parallel training data for: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, and Amharic.

Test PhaseHowever, the availability of training corpora will differ between the languages. Thus, for new languages, no such corpora will be provided. The main challenge of this competition will be to perform an unsupervised and cross-lingual detoxification.

Methodology

tba.

Data

Parallel Devset is now available! The link to the parallel multilingual data. This dataset contains 400 pairs for each of 9 languages.

Definition of toxiciy

One of the crucial points in this task is to have a common ground on how to estimate if the text is toxic or not. In our task, we will work only with explicit types of toxicity—obvious present of obscene and rude lexicon where still there is meaningful neutral content present—and do not work with implicit types—like sarcasm, passive aggressiveness, or direct hate to some group where no neutral content can be found. Such implicit toxicity types are challenging to be detoxified so the intent will indeed become non-toxic. For this reason, we tried to pick for our datasets the sentences with explicit toxicity where we can detoxify it. However, toxicity can be quite a subjective intent. We hope, that we will agree on the majority of the cases what should be toxic or not. In the end, the main goal is to make the texts and the world at least somehow less toxic ;)

Evaluation

The concept of text detoxification evaluation

Development Phase

For the whole competition, the automatic evaluation metrics set will be available. We provide the multilingual automatic evaluation pipeline based on main three parameters:

Style Transfer Accuracy: Given the generated paraphrase, classify its level of non-toxicity. For this, specifically fine-tuned xlm-roberta-large for toxicity binary classification is used.
Content preservation: Given two texts, evaluate the similarity of their content. We calculate it as cosine similarity between LaBSe embeddings.
Fluency: To estimate the adequacy of the text and its similarity to the human-written detoxified references, we calculate ChrF measure.

Each metric component lies in the range [0;1]. To have the one common metric for leaderboard estimation, we will calculate Joint metric as the mean of STA*SIM*FL per sample.

Dementieva D. et al. Methods for Detoxification of Texts for the Russian Language. Multimodal Technologies and Interaction 5, 2021. [pdf]
Dale D. et. al. Text Detoxification using Large Pre-trained Neural Models. EMNLP, 2021. [pdf]
Logacheva V. et al. ParaDetox: Detoxification with Parallel Data. ACL, 2022. [pdf]
Moskovskiy D. et al. Exploring Cross-lingual Text Detoxification with Large Multilingual Language Models. ACL SRW, 2022. [pdf]
Dementieva D. et al. RUSSE-2022: Findings of the First Russian Detoxification Shared Task Based on Parallel Corpora. Dialogue, 2022. [pdf]
Logacheva, V. et al. A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification. HumEval, 2022. [pdf]
Dementieva, D. et al. Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxification. AACL, 2023. [pdf]