Supervised and Semi-Supervised Methods for Energy-Efficient Multimodal Hate Detection

The rapid growth of user-generated content on social media has increased the spread of hateful content, posing significant challenges for content moderation systems. Among these forms of abuse, sexism and misogyny have emerged as a particularly pervasive and damaging phenomenon. Detecting misogynistic and sexist content is further complicated by the multimodal nature of online communication, especially memes, as well as by the inherent subjectivity and disagreement among human annotators when labeling such content. Although recent advances in large multimodal models have demonstrated strong performance in this area, but they require substantial computational resources and have a significant environmental impact, limiting their practicality for large-scale deployment. Moreover, prevailing approaches treat annotation disagreement as noise to be suppressed rather than as meaningful signal about content ambiguity. This thesis investigates computationally efficient and environmentally sustainable approaches for multimodal sexism and misogyny detection, that explicitly model annotation disagreement. Grounded in the principles of Green AI, the work explores lightweight methods that leverage pretrained representations without relying on extensive fine-tuning or large labeled datasets. A semi‑supervised constrained clustering method is first introduced to leverage pretrained vision–language embeddings with minimal annotation and computational cost, achieving competitive performance while offering favorable energy–accuracy trade‑offs. However, this method assumes unambiguous ground truth labels, overlooking the contested nature of some content. The second stage develops a supervised contrastive learning framework that detects hate while simultaneously modeling annotator disagreement as a complementary task. Experiments show that the same lightweight architecture performs effectively on both hate and disagreement detection, though the latter proves to be inherently more challenging, reflecting the difficulty of predicting human perceptual variation. This framework is also extended to joint hate-disagreement prediction, enabling a single efficient model to simultaneously detect hateful content and flag ambiguous cases requiring human review. Together, these findings demonstrate that robust, uncertainty-aware content moderation systems can be built without reliance on large, resource-intensive models, and that modeling subjectivity is not only feasible but also compatible with computational efficiency.