BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive…

• AN
  Agence Nationale de la Recherche

  Awards: ANR-19-P3IA-0001, 19-P3IA-0001, ANR-19-CE38-0012, ANR-19
• NO
  Nederlandse Organisatie voor Wetenschappelijk Onderzoek

  Award: 406.DI.19.059
• CN
  Centre National de la Recherche Scientifique
• GÉ
  Grand Équipement National De Calcul Intensif