ArAIEval

Arabic AI Tasks Evaluation (ArAIEval)

Task 1: Unimodal (Text) Propagandistic Technique Detection

Definition

The unimodel (text) propagandistic technique detection task is defined below:

Given a multigenre text snippet (a news paragraph or a tweet), the task is to detect the propaganda techniques used in the text together with the exact span(s) in which each propaganda technique appears. This a sequence tagging task.

Leaderboard

Datasets

The dataset covers two genres, tweets and paragraphs extracted from news articles. Please check the Task 1 repository.

Evaluation

This is a multilabel sequence tagging task. Participating systems will be evaluated using a modified F_1 measure that accounts for partial matching between the spans across the gold labels and the predictions.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at Task 1 repository

Submission

Guidelines

The process consists of two phases:

  1. System Development Phase: This phase involves working on the dev set.
  2. Final Evaluation Phase (will start on 27 April 2023): This phase involves working on the test set, which will be released during the evaluation cycle. For each phase, please adhere to the following guidelines:
  • Each team should create and maintain a single account for submissions. Please ensure all runs are submitted through this account. Submissions from multiple accounts by the same team could result in your system being not ranked in the overview paper.
  • The most recent file submitted to the leaderboard will be considered your final submission.
  • The output file must be named task1_any_suffix.jsonl, where [1] (for example, task1_team_name.jsonl). Failure to follow this naming convention will result in an error on the leaderboard.
  • You are required to compress the .jsonl file into a .zip file (for example, zip task1.zip task1.jsonl) and submit it via the Codalab page.
  • Please include your team name and a description of your method with each submission.
  • You are permitted to submit a maximum of 200 submissions per day for each subtask.

Submission Site

System Development Phase: Please submit your results on the respective subtask tab through this competition on CodaLab.