ArAIEval

Arabic AI Tasks Evaluation (ArAIEval)

Task 2: Disinformation Detection

Definition

For the disinformation detection task, we offer the two subtasks, which are defined below:

  • Subtask 2A: Given a tweet, categorize whether it is disinformative. This is a binary classification task.
  • Subtask 2B: Given a tweet, detect the fine-grained disinformation class, if any. This is a multiclass classification task. The fine-grained labels include hate-speech, offensive, rumor, and spam

Leaderboard

Subtask 2A

Subtask 2B

Datasets

The dataset consists of tweets annotated in binary and multiclass settings. Please check the Task 2 repository.

Evaluation

To measure the performance of the systems, for all subtasks, we use micro-F1 and macro-F1, where labels are imbalanced. The official evaluation measure for all subtasks is micro-F1.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at Task 2 repository

Submission

Guidelines

The process consists of two phases:

  1. System Development Phase: This phase involves working on the dev set.
  2. Final Evaluation Phase: This phase involves working on the test set, which will be released during the evaluation cycle. For each phase, please adhere to the following guidelines:
  • Each team should create and maintain a single account for submissions. Please ensure all runs are submitted through this account. Submissions from multiple accounts by the same team could result in your system being not ranked in the overview paper.
  • The most recent file submitted to the leaderboard will be considered your final submission.
  • The output file must be named task[1/2][A/B]_any_suffix.tsv, where [1/2] is the task number and A/B is your specific subtask (for example, task1A_team_name.tsv or task2B_team_name.tsv). Failure to follow this naming convention will result in an error on the leaderboard. Subtasks include 1A, 1B, 2A and 2B.
  • You are required to compress the .tsv file into a .zip file (for example, zip task1B.zip task1B.tsv) and submit it via the Codalab page.
  • Please include your team name and a description of your method with each submission.
  • You are permitted to submit a maximum of 200 submissions per day for each subtask.

Submission Site

System Development Phase: Please submit your results on the respective subtask tab through this competition on CodaLab.

Final Evaluation Phase: Please submit your results on the respective subtask tab through this competition on CodaLab.