Lightweight Adaptation of Large Language Models for Toxicity Detection in Low-Resource Languages
Mariano Aloiso
MAS, 2024
MONTUFAR CUARTAS, GUIDO FRANCISCO
Toxicity detection —the process of identifying harmful or offensive content— remains a significant challenge in natural language processing, especially for low-resource languages like Bodo, a Tibeto-Burman language spoken in Assam, India. This study addresses this challenge by adapting a large language model (LLM) that has not been exposed to Bodo text, aiming to enable toxicity detection with minimal data and computational resources. We explore techniques including zero-shot and few-shot learning, cross-lingual transfer learning, and parameter-efficient fine-tuning methods such as adapter and Low-Rank Adaptation (LoRA). Additionally, we employ data augmentation strategies such as backtranslation, noise injection, and transliteration to overcome data scarcity. Initial findings reveal the limitations of multilingual transformers in tasks involving unseen languages, with zero-shot inference using English labels achieving an F1 score of 44.42. By employing targeted adaptation techniques, we achieve substantial performance improvements. Our best-performing approach combines multilingual fine-tuning, transliteration, LoRA, and noise injection, resulting in an F1 score of 78.16. These findings demonstrate the feasibility of lightweight methods for toxicity detection in underserved languages, even with minimal computational resources and datasets.
2024