ArAIEval

Arabic AI Tasks Evaluation (ArAIEval)

Welcome to ArAIEval shared task at ArabicNLP 2024!

Propagandistic content spreading online in mainstream and social media is often used to mislead the general audience. Automatic identification of such content is crucial to assist fact-checkers, journalists, and the general audience in challenging the information presented in this content.

ArAIEval offers two shared tasks:

  1. detection of propagandistic textual spans with persuasion techniques identification (unimodal),
  2. distinguishing between propagandistic and non-propagandistic memes (multimodal).

ArAIEval covers both the tweets, news articles and memes. The classification settings includes span identification and binary subtasks.

Tasks

  • Task 1: Unimodal (Text) Propagandistic Technique Detection
  • Task 2: Multimodal Propagandistic Memes Classification

Important Dates

  • 15 Mar 2024: Registration on CodaLab and the start of the development cycle (release of training and development datasets, along with submission for the development phase on CodaLab)
  • 27 April 2023 23:59 AOE: Beginning of the evaluation cycle (test sets release)
  • 1 May 2024 23:59 AOE: End of the evaluation cycle (run submission)
  • 4 May 2024 23:59 AOE: End of the evaluation cycle (run submission)
  • 5 May 2024: Release leaderboard
  • 15 May 2024: Deadline for the submission of shared task papers
  • 17 May 2024: Deadline for the submission of shared task papers
  • 20 May 2024: Deadline for the submission of shared task papers
  • 22 May 2024: Deadline for the submission of shared task papers
  • 17 June 2024: Notification of acceptance of shared task papers
  • 1 July 2024: Deadline for submission of camera-ready papers
  • 16 August 2023: ArabicNLP Conference (colocated with ACL-2024)

Recent Updates

  • 1st March, 2024: Website is up!

Proceedings

Instructions to Prepare Your Paper:

The title of paper should be in the following format: <Team Name> at ArAIEval Shared Task: <Title of the work>

For example: Scholarly at ArAIEval Shared Task: LLMs in Persuasion Technique Detection for Identifying Manipulative Strategies

Cite: Please check the citation section to cite the shared task papers

Templates: Please find the templates at: https://github.com/acl-org/acl-style-files/

Paper Length: Shared task description papers may include up to 4 pages of content, in addition to unlimited references.

More information: For more details, please visit the following link: https://arabicnlp2024.sigarab.org/call-for-papers

Paper submission: https://openreview.net/group?id=SIGARAB.org/ArabicNLP/2024/Shared_Task/ArAIEval

Suggested structure for the system description papers:

Abstract: four/five sentences highlighting your approach and key results.
Introduction: 3/4 a page expanding on the abstract mentioning key background such as why the task is challenging for current modeling techniques and why your approach is interesting/novel.
Related Work: This section provides a review of existing research in the field, with a focus on highlighting how our contribution diverges from and adds value to the current body of work.
Data: review of the data you used to train your system. Be sure to mention the size of the training, validation and test sets that you’ve used, and the label distributions, as well as any tools you used for preprocessing data.
System: a detailed description of how the systems were built and trained. If you’re using a neural network, were there pre-trained embeddings, how was the model trained, what hyperparameters were chosen and experimented with? How long did the model take to train, and on what infrastructure? Linking to source code is valuable here as well, but the description should be able to stand alone as a full description of how to reimplement the system. While other paper styles include background as a separate section, it’s fine to simply include citations to similar systems which inspired your work as you describe your system.
Results: a description of the key results of the paper. If you have done extra error analysis into what types of errors the system makes, this is extremely valuable for the reader. Unofficial results from after the submission deadline can be very useful as well.
Discussion: general discussion of the task and your system. Description of characteristic errors and their frequency over a sample of development data. What would you do if you had another 3 months to work on it?
Conclusion: a restatement of the introduction, highlighting what was learned about the task and how to model it.

Citation

Please cite the following papers.

[1] Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md. Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. “ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content”. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), August 2024, Bangkok. Association for Computational Linguistics.

[2] Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. “ArMeme: Propagandistic Content in Arabic Memes.” Last modified 2024. arXiv preprint arXiv:2406.03916.

[3] Maram Hasanain, Fatema Ahmed, and Firoj Alam. “Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles.” In Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation (LREC-COLING 2024), 2024.

[4] Maram Hasanain and Fatema Ahmed and Firoj Alam. “Large Language Models for Propaganda Span Annotation.” arXiv preprint arXiv:2311.09812 (2023).

[5] Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023. “ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text.” In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), December. Singapore: Association for Computational Linguistics.

[6] Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. “Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic.”, In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), 108–118, December. Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics. https://aclanthology.org/2022.wanlp-1.11.

@inproceedings{araieval:arabicnlp2024-overview,
    title = ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content,
    author = "Hasanain, Maram and Hasan, Md. Arid and Ahmed, Fatema and Suwaileh, Reem  and Biswas, Md. Rafiul and Zaghouani, Wajdi and Alam, Firoj",
    booktitle = "Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024)",
    month = Aug,
    year = "2024",
    address = "Bangkok",
    publisher = "Association for Computational Linguistics",
}

@article{alam2024armeme,
  title={ArMeme: Propagandistic Content in Arabic Memes},
  author={Alam, Firoj and Hasnat, Abul and Ahmed, Fatema and Hasan, Md Arid and Hasanain, Maram},
  year={2024},
  journal={arXiv preprint arXiv:2311.09812},
}

@inproceedings{hasanain2024can,
  title={Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles},
  author={Hasanain, Maram and Ahmed, Fatema and Alam, Firoj},
  booktitle = "Proceedings of the 2024 Joint International Conference On Computational Linguistics, Language Resources And Evaluation",
  series = "LREC-COLING 2024",
  year={2024},
  address={Torino, Italy}
}

@article{hasanain2023large,
  title={Large Language Models for Propaganda Span Annotation},
  author={Hasanain, Maram and Ahmed, Fatema and Alam, Firoj},
  journal={arXiv preprint arXiv:2311.09812},
  year={2023}
}

@inproceedings{araieval:arabicnlp2023-overview,
    title = "ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text",
    author = "Hasanain, Maram and Alam, Firoj and Mubarak, Hamdy, and Abdaljalil, Samir  and Zaghouani, Wajdi and Nakov, Preslav  and Da San Martino, Giovanni and Freihat, Abed Alhakim",
    booktitle = "Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023)",
    month = Dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
}

@inproceedings{alam-etal-2022-overview,
    title = "Overview of the {WANLP} 2022 Shared Task on Propaganda Detection in {A}rabic",
    author = "Alam, Firoj  and
      Mubarak, Hamdy  and
      Zaghouani, Wajdi  and
      Da San Martino, Giovanni  and
      Nakov, Preslav",
    booktitle = "Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.wanlp-1.11",
    pages = "108--118",
}

Contact

Slack: channel
Email: araieval@googlegroups.com

Organizers

  • Firoj Alam, Qatar Computing Research Institute, Qatar
  • Maram Hasanain, Qatar Computing Research Institute, Qatar
  • Reem Suwaileh, HBKU, Qatar
  • Md. Arid Hasan, University of New Brunswick, Canada
  • Fatema Ahmed, Qatar Computing Research Institute, Qatar
  • Md. Rafiul Biswas, HBKU, Qatar
  • Wajdi Zaghouani, HBKU, Qatar

Acknowledgments

The contributions of this work were funded by the NPRP grant 14C-0916-210015, which is provided by the Qatar National Research Fund (a member of Qatar Foundation).

QRDI



QNRF