PAN at CLEF 2024

Shared Tasks

Important Dates

  • January 31, 2024: Training Data Released
  • May 05, 2024: Software submission deadline
  • May 31, 2024: Participant paper submission [submission] [paper template]
  • June 24, 2024: Peer review notification
  • July 08, 2024: Camera-ready participant papers submission
  • September 09-12, 2024: Conference

The time of all deadlines is Midnight CEST.

Keynotes

Julio Gonzalo
Will an A.I. be the Shakespeare of the XXI Century? Experiments and Thoughts on Large Language Models as Creative Text Writers [slides]
Julio Gonzalo
UNED

One of the most remarkable aspects of Large Language Models is their ease of writing creative texts; under certain conditions, they have been shown to match and improve average human writing skills. But is their writing truly creative, or just a repetition of the clichés they have been pre-trained with? Do they have a distinctive creative writing style? What is the role of (human) prompting in the creation process?

In the talk, I will discuss the creative writing potential of LLMs and their intrinsic limitations, paying special attention to the experiments carried out at UNED. These include a contest between GPT-4 and one of the best contemporary novelists in Spanish, Patricio Pron. The contest is inspired by past AI duels (such as DeepBlue vs Kasparov and AlphaGo vs Lee Sidol), and was designed to test whether LLMs can already challenge a top (rather than an average) fiction writer.

Julio Gonzalo apart from being the sax of Trevithick (see picture) and one of the fastest reviewers of the NLP & IR circuit (aka Julio Speedy Gonzales), he is the director of the UNED Research Center in Natural Language Processing (NLP) and Information Retrieval (IR) in Madrid. Along his career he has worked on topics such as online reputation monitoring, toxicity and misinformation in Social Media, interactive cross-language search, computational creativity and semantic similarity. He has also worked extensively in the design and assessment of evaluation metrics for a wide range of Artificial Intelligence problems, which led to a Google Faculty Research Award (together with Enrique Amigó and Stefano Mizzaro). He has recently been general co-chair of ACM SIGIR 2022 and of IberLEF 2019-2022, the annual evaluation campaign for NLP systems in Spanish and other Iberian languages. He is currently leading ODESIA (odesia.es), a Spanish government initiative to measure the state of the art of language technologies in Spanish.

Read more… Read less…

Program

PAN's program is part of the CLEF 2024 conference program. All times are Central European Summer Time - CEST.
Tuesday, September 10
09:00-10:40CLEF Session: Lab Overviews (ImageCLEF, CheckThat!, JokerLab, Pan, qCLEF)
11:10-12:40Keynote and Lab Session, Chair: Paolo Rosso
11:10-12:10Keynote: Will an A.I. be the Shakespeare of the XXI Century?
Julio Gonzalo
12:10-12:40Overview: Voight-Kampff Generative AI Authorship Verification
Janek Bevendorff, Matti Wiegmann, Efstathios Stamatatos, Benno Stein, and Martin Potthast
15:40-16:40Poster Session
16:40-18:10Lab Session, Chair: Benno Stein.
16:40-17:00Generative AI Authorship Verification at Foshan University
Leilei Kong
17:00-17:10BaselineAvengers at PAN 2024: Often-Forgotten Baselines for LLM-Generated Text Detection
Ludwig Lorenz, Funda Zeynep Aygüler, Ferdinand Schlatt, and Nailia Mirzakhmedova
17:10-17:20Team aida at PAN: Ensembling Normalized Log Probabilities
Pablo Miralles, Alejandro Martin, David Camacho
17:20-17:40Overview: Multi-Author Writing Style Analysis
Eva Zangerle, Maximilian Mayerl, Martin Potthast, and Benno Stein
17:40-17:50Team fosu-stu at PAN: Supervised Fine-Tuning of Large Language Models for Multi Author Writing Style Analysis
Jiajun Lv, Yusheng Yi, and Haoliang Qi
17:50-18:00NYCU-NLP at PAN 2024: Integrating Transformers with Similarity Adjustments for Multi-Author Writing Style Analysis
Tzu-Mi Lin, Yu-Hsin Wu, and Lung-Hao Lee
18:00-18:10Continual Transfer Learning with Progress Prompt for Multi-Author Writing Style Analysis
Zhanhong Ye, Yutong Zhong, Chen Huang, and Leilei Kong
Wednesday, September 11
14:00-15:30Lab Session, Chair: Matti Wiegmann.
14:00-14:25Overview: Oppositional Thinking Analysis
Damir Korenčić, Berta Chulvi, Xavier Bonet, Mariona Taulé, Francisco Rangel, and Paolo Rosso
14:25-14:30Best System Award: Oppositional Thinking Analysis
Symanto
14:30-14:45Conspiracy vs critical thinking using an ensemble of transformers with data augmentation techniques
Angelo Maximilian Tulbure and Mariona Coll Ardanuy
14:45-15:00SINAI at PAN 2024 Oppositional Thinking Analysis: Exploring the fine-tuning performance of LLMs
María Estrella Vallecillo-Rodríguez, María Teresa Martín-Valdivia and Arturo Montejo-Ráez
15:00-15:15Towards a Computational Framework for Distinguishing Critical and Conspiratorial Texts by Elaborating on the Context and Argumentation with LLMs
Ariana Sahitaj, Premtim Sahitaj, Salar Mohtaj, Sebastian Möller and Vera Schmitt
15:15-15:30DSVS at PAN 2024: Ensemble Approach of Large Language Models for Analyzing Conspiracy Theories Against Critical Thinking Narratives
Sergio Damian, Brian Herrera-Gonzalez, David Vazquez-Santana, Hiram Calvo, Edgardo Felipe-Riverón and Cornelio Yáñez-Márquez
16:30-18:00Lab Session, Chair: tbd.
16:30-17:00Overview: Multilingual Text Detoxification
Daryna Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinew Ali Ayele, Naquee Rizwan, Florian Schneider, Xintong Wang, Seid Muhie Yimam, Dmitry Ustalov, Elisei Stakovskii, Alisa Smirnova, Ashraf Elnagar, Animesh Mukherjee, Alexander Panchenko
17:00-17:15Linguistic_Hygenist at PAN 2024 TextDetox: HybridDetox - A Combination of Supervised and Unsupervised Methods for Effective Multilingual Text Detoxification
Susmita Gangopadhyay, M.Taimoor Khan, Hajira Jabeen
17:15-17:30A Multilingual Text Detoxification Method Based on Few-shot Learning and CO-STAR Framework
Jiangao Peng, Zhongyuan Han, Huan Zhang, Jingyan Ye, Chang Liu, Biao Liu, Mingcan Guo, Haoyang Chen, Zijie Lin, Yujiao Tang
17:30-17:45SomethingAwful at PAN 2024 TextDetox: Uncensored Llama 3 Helps to Censor Better
Sergey Pletenev
17:45-18:00PAN 2024 Multilingual TextDetox: Exploring Different Regimes For Synthetic Data Training For Multilingual Text Detoxification
Nikita Sushko
18:00-18:10Closing

Participation Modalities

To participate at PAN, first register for your task of choice at CLEF, then follow the instructions below.

Submission

We use TIRA for all submissions to PAN. Please go to tira.io, create an account, and register for the individual tasks you want to participate in. You need to submit your software or your results via TIRA. You can find all submission guides in TIRA's forum.

Data

You can download PAN's datasets here. For details, please check the individual task's website.

Evaluation and Baseline Code

All code used at PAN is published at GitHub. You can find all validators, evaluators, and baselines there.

Software Submissions

PAN promotes reproducible science with software submissions. Please prepare and submit your software as Docker image(s). You can find guides and examples in the resources linked above. Some tasks allow only software submissions and only release the test data after the conference.

Paper Submission and Presentation

PAN is co-located with CLEF 2024 in Grenoble. Every participant is expected to write a notebook paper describing their approach to CLEF (published at CEUR-WS, which is indexed by DBLP). At the CLEF conference, all submissions will be presented as talks or posters. CLEF will be a hybrid conference.

Organizing Committee