PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel; 24(240):1−113, 2023.
Abstract
This study explores the impact of scale on few-shot learning by training a 540-billion parameter, densely activated, Transformer language model called Pathways Language Model (PaLM). The model was trained on 6144 TPU v4 chips using Pathways, a new ML system that enables highly efficient training across multiple TPU Pods. The results demonstrate the benefits of scaling, achieving state-of-the-art few-shot learning performance on numerous language understanding and generation benchmarks. PaLM 540B surpasses the finetuned state-of-the-art on multi-step reasoning tasks and even outperforms average human performance on the BIG-bench benchmark. The study also highlights PaLM’s strong capabilities in multilingual tasks and source code generation. Additionally, an analysis on bias and toxicity is provided, along with a study on the extent of training data memorization with respect to model scale. The ethical considerations related to large language models are discussed, along with potential mitigation strategies.
[abs]