Atlas: Few-shot Learning with Retrieval Augmented Language Models
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, Edouard Grave; 24(251):1−43, 2023.
Abstract
Many large language models have demonstrated impressive few-shot learning performance on various tasks. However, tasks that require knowledge, such as question answering and fact checking, often necessitate a high number of parameters to store the required knowledge. Retrieval-augmented models excel at knowledge-intensive tasks without the need for as many parameters, but their effectiveness in few-shot settings remains unclear. In this study, we introduce Atlas, a meticulously designed and pre-trained retrieval-augmented language model capable of learning knowledge-intensive tasks with minimal training examples. We evaluate Atlas on a wide range of tasks, including MMLU, KILT, and Natural Questions, and investigate the impact of the document index’s content, demonstrating its easy updatability. Notably, Atlas achieves over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameter model by 3% despite having 50 times fewer parameters.
[abs]