Cross-lingual Generalization as Compression in Multilingual Language Models

Abstract

Multilingual Language Models exhibit the surprising ability to generalize across languages, performing tasks in languages different from those seen during training. However, the mechanisms driving this cross-lingual generalization remain largely unknown. In this talk, I will present my hypothesis that cross-lingual generalization arises as an artifact of compression. Specifically, I propose that the model, constrained by limited parameter space, must efficiently encode new pre-training data, leading to the creation of abstractions that serve as memory-efficient representations. I will introduce self-conditioning as an effective method for identifying neurons corresponding to arbitrary concepts and demonstrate how the representations of these concepts become increasingly aligned during pre-training.