Together AI releases RedPajama-7B models

Together AI released RedPajama-7B, a family of three 7-billion-parameter models — a base model, an instruction-tuned variant and a chat variant — trained on the RedPajama dataset, a roughly 5-terabyte, 1-trillion-token corpus the project had published in April 2023 as an open reproduction of the data mix described in Meta’s LLaMA paper. All three models were released under the Apache 2.0 licence, and the base model itself was trained on the 1-trillion-token RedPajama-1T subset. Training used 3,072 Nvidia V100 GPUs on the Summit supercomputer through a US Department of Energy INCITE compute award.

Together AI reported that the instruct variant, fine-tuned on instruction-following data from the P3 and Natural Instructions collections, outperformed Falcon-7B and MPT-7B by 2 to 9 points on the HELM benchmark suite and was, by that measure, the highest-scoring open model at the 7B scale. The plain base model trailed LLaMA-7B by about 4 points on the same metrics — LLaMA’s weights, though also widely circulated by that point, remained under a non-commercial research licence.

The release completed the goal the RedPajama project had set two months earlier: not just an open dataset, but a fully open, apples-to-apples model trained on it, with data, training code and weights all public — a reproducibility standard that closed and semi-open releases of the period, including LLaMA itself, did not meet.