Trigger Detection 2023

Synopsis

  • Task: Given a document, assign all appropriate trigger warning labels.
  • Input: A Fanwork with 50 to 6,000 words length. [data]
  • Output: The ID of the work and a list of appropriate trigger labels, selected from a set of 32.
  • Evaluation: F1 [code]
  • Submission: tira.io preferably as a software submission

Task

In trigger detection, we want to assign trigger warning labels to documents that contain potentially discomforting or distressing (triggering) content [1]. We ask participants to develop a software or model that determines if a document contains triggering content. In particular, we model trigger detection as a multi-label document classification challenge: assign each document all appropriate trigger warnings, but not more. All warnings are chosen from the author's perspective, i.e. the work's author decided which kind of trigger the document contains.

Data

Our dataset contains works of fanfiction, extracted from archiveofourown.org (AO3). Each work is between 50 and 6,000 words long and has between 1 and many trigger warnings assigned. The label set contains 32 different trigger warnings with a long-tailed frequency distribution, i.e. some labels are very common, most labels are increasingly rare. Our training dataset contains 307,102 examples, with 17,104 in validation and 17,040 in the test split.

Training Dataset

The train and validation splits contains two files each: works.jsonl and labels.jsonl. Both files have 1 example on each line, encoded in a json object. The works.jsonl can be loaded with the huggingface datasets library. 

{
    "work_id": 1234,                 # The id of the work.
    "labels": [1,0,0,1,...]          # A list of labels (binary format). Check the utility code for conversion.
    "text": "The text of a fanwork." # This contains <em>inline html</em> to preserve the original formatting.
}
works.jsonl
{
    "work_id": 1234,                         # The id of the work.
    "labels": [pornographic-content, abuse]  # A list of labels (string format). Check the utility code for conversion.
}
labels.jsonl

Test Dataset

The test split contains one file: works.jsonl. Its format is as described above, but without with labels. Each solution to this task should take this works.jsonl as an input, and produce a complete labels.jsonl as described above.

Label Set

All labels are based on the freeform content warnings added to a fanwork by its author. Detailed definitions are included in the dataset download. 

pornographic-content, violence, death, sexual-assault, abuse, blood, suicide,
pregnancy, child-abuse, incest, underage, homophobia, self-harm, dying, kidnapping,
mental-illness, dissection, eating-disorders, abduction, body-hatred, childbirth,
racism, sexism, miscarriages, transphobia, abortion, fat-phobia, animal-death,
ableism, classism, misogyny, animal-cruelty

Evaluation

Our ranking criteria are macro (primary) and micro F1. We also compute most other common evaluation metrics used in multi-label classification.

Submission

All submissions are handled through tira.io. We encourage all participants to use software submissions (via docker) for the sake of reproducibility. In a software submission you upload a docker image, which is then built and executed on the test data on tira's servers. Please follow tira's extensive documentation (and forums) for instructions.

Task Committee