ArAIEval

Arabic AI Tasks Evaluation (ArAIEval)

Task 2: Multimodal Propagandistic Memes Classification

Definition

For the multimodal propagandistic memes classification task, we offer the three subtasks, which are defined below:

  • Subtask 2A: Given a text extracted from meme, categorize whether it is propagandistic or not.
  • Subtask 2B: Given a meme (text overlayed image), the task is to detect whether the content is propagandistic.
  • Subtask 2C: Given multimodal content (text extracted from meme and the meme itself) the task is to detect whether the content is propagandistic.

All are binary classification tasks.

Leaderboard

TBA

Datasets

The dataset consists of memes annotated as propagandistic vs not-propagandistic, which were collected from different social media (e.g., Facebook, Twitter, Instagram and Pinterest). Please check the Task 2 repository.

Evaluation

To measure the performance of the systems, for all subtasks, we compute accuracy (Acc), weighted precision (P), recall (R), and F1. However, the official evaluation measure for all subtasks is macro-F1.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at Task 2 repository

Submission

Guidelines

The process consists of two phases:

  1. System Development Phase: This phase involves working on the dev set.
  2. Final Evaluation Phase (will start on 27 April 2023): This phase involves working on the test set, which will be released during the evaluation cycle. For each phase, please adhere to the following guidelines:
  • Each team should create and maintain a single account for submissions. Please ensure all runs are submitted through this account. Submissions from multiple accounts by the same team could result in your system being not ranked in the overview paper.
  • The most recent file submitted to the leaderboard will be considered your final submission.
  • The output file must be named task[2][A/B/C]_any_suffix.tsv, where [2] is the task number and A/B/C is your specific subtask (for example, task2A_team_name.tsv or task2B_team_name.tsv). Failure to follow this naming convention will result in an error on the leaderboard. Subtasks include 2A, 2B and 2C.
  • You are required to compress the .tsv file into a .zip file (for example, zip task2A.zip task2A.tsv) and submit it via the Codalab page.
  • Please include your team name and a description of your method with each submission.
  • You are permitted to submit a maximum of 200 submissions per day for each subtask.

Submission Site

System Development Phase: Please submit your results on the respective subtask tab through this competition on CodaLab.