Introducing the workshop

20-minute talks by Omar Sanseverio and Hady Elsahar followed by joint Q & A

20 minute keynotes by Lucia Specia and Preethi Jyothi

Meeting spot 2 in GatherTown for the Ask a Wikipedian discussions

Meeting spot 3 in GatherTown for the Ask a Wikipedian discussions

Keynotes by Leila Zia, Andrew Lih, and Caroline Becker

Panel on "Multilinguality in multimodal research and open data" with Preethi Jyothi, Michael Running Wolf, Jason Baldridge, Omar Sanseviero, and Margaret Mitchell Moderated by: Lucie Kaffee

Panel on "How can Wikimedia and CV/ML communities learn from each other?" with Lucia Specia, Caroline Becker, Leila Zia, Marc Najork, Hady Elsahar Moderated by: Andrew Lih

Panel and Q&A with Tom Duerig (Google), Lijuan Wang (Microsoft), Marcus Rohrbach (Meta AI Research). Moderated by Yannis Kalantidis.

20 minutes keynotes by Michael Running Wolf and Margaret Mitchell.

20 minute keynotes by Jason Baldridge

With the increased accessibility of web and online encyclopedias, the amount of data to manage is constantly increasing. In Wikipedia, for example, there are millions of pages written in multiple languages. These pages contain images that often lack the textual context, remaining conceptually floating and therefore harder to find and manage.In this work, we present the system we designed for participating in the Wikipedia Image-Caption Matching challenge on Kaggle, whose objective is to use data associated with images (URLs and visual data) to find the correct caption among a large pool of available ones. A system able to perform this task would improve the accessibility and completeness of multimedia content on large online encyclopedias. Specifically, we propose a cascade of two models, both powered by the recent Transformer model, able to efficiently and effectively infer a relevance score between the query image data and the captions. We verify through extensive experimentation that the proposed two-model approach is an effective way to handle a large pool of images and captions while maintaining bounded the overall computational complexity at inference time.Our approach achieves remarkable results, obtaining a normalized Discounted Cumulative Gain (nDCG) value of 0.53 on the private leaderboard of the Kaggle challenge.

This paper is a technical report to share our experience and findings building a Korean and English bilingual multimodal model. While many of the multimodal datasets focus on English and multilingual multimodal research uses machine-translated texts, employing such machine-translated texts is limited to describing unique expressions, cultural information, and proper noun in languages other than English. In this work, we collect 1.1 billion image-text pairs (708 million Korean and 476 million English) and train a bilingual multimodal model named KELIP. We introduce simple yet effective training schemes, including MAE pre-training and multi-crop augmentation. Extensive experiments demonstrate that a model trained with such training schemes shows competitive performance in both languages. Moreover, we discuss multimodal-related research questions: 1) strong augmentation-based methods can distract the model from learning proper multimodal relations; 2) training multimodal model without cross-lingual relation can learn the relation via visual semantics; 3) our bilingual KELIP can capture cultural differences of visual semantics for the same meaning of words.; 4) a large-scale multimodal model can be used for multimodal feature analogy. We hope that this work will provide helpful experience and findings for future research. We provide an open-source pre-trained KELIP.

English Wikipedia has long been an important data source for much research and natural language machine learning modeling. The growth of non-English language editions of Wikipedia, greater computational resources, and calls for equity in the performance of language and multimodal models have led to the inclusion of many more language editions of Wikipedia in datasets and models. Building better multilingual and multimodal models requires more than just access to expanded datasets; it also requires a better understanding of what is in the data and how this content was generated. This paper seeks to provide some background to help researchers think about what differences might arise between different language editions of Wikipedia and how that might affect their models. It details three major ways in which content differences between language editions arise (local context, community and governance, and technology) and recommendations for good practices when using multilingual and multimodal data for research and modeling.

Closing remarks