There are several popular corpora used in training large language models. Some of them are:

Wikipedia : Wikipedia is a widely used corpus for training LLMs. It contains millions of articles covering a wide range of topics, making it a valuable source of general knowledge.

Common Crawl : Common Crawl is a nonprofit organization that provides a large-scale web corpus of billions of pages. It is used to train LLMs for a variety of natural language processing tasks.

BooksCorpus : The BooksCorpus is a dataset of over 11,000 books, containing over 740 million words. It is often used to train LLMs for text generation and language modeling tasks. Google News : The Google News dataset contains articles from over 70,000 news sources in multiple languages. It is often used to train LLMs for text classification and topic modeling tasks.

OpenWebText : OpenWebText is a large and diverse dataset consisting of web pages, books, and other text sources. It is commonly used to train LLMs for text generation and language modeling tasks.

COCO Captions : The COCO Captions dataset contains over 330,000 images with descriptive captions. It is used to train LLMs for image captioning tasks.

Penn Treebank : The Penn Treebank is a dataset of over 4 million words of parsed and annotated text. It is often used to train LLMs for syntactic analysis and language modeling tasks.