Imitating Language with Inverse Reinforcement Learning

The majority of language model training builds on imitation learning.
It covers pretraining, supervised fine-tuning, and affects the starting 
conditions for reinforcement learning from human feedback (RLHF). The 
simplicity and scalability of maximum likelihood estimation (MLE) for 
next token prediction led to its role as predominant paradigm. 
However, the broader field of imitation learning can more effectively 
utilize the sequential structure underlying autoregressive generation.
We focus on investigating the inverse RL perspective to imitation, 
extracting rewards and directly optimizing sequences instead of individual 
token likelihoods and evaluate its benefits for fine-tuning large language 
models. We find clear advantages for IRL-based imitation, in particular 
for retaining diversity while maximizing task performance, rendering IRL 
a strong alternative to MLE even without online data generation. 
We also find that IRL-extracted reward functions can be highly correlated 
with task-specific performance metrics.