TII releases Falcon 180B

The UAE’s Technology Innovation Institute released Falcon 180B, a 180-billion-parameter language model that, at launch, was the largest set of openly available weights of any language model. Hugging Face, which hosted the release, described it as topping the leaderboard for openly accessible pretrained models.

TII trained the model on 3.5 trillion tokens drawn from its RefinedWeb web-scrape dataset, four times the training data used for Meta’s Llama 2, using roughly 7 million GPU-hours across 4,096 GPUs running concurrently. TII said the model outperformed GPT-3.5 on the MMLU benchmark and was, depending on the test, comparable to Google’s PaLM 2-Large or positioned between GPT-3.5 and GPT-4.

The release terms complicated the “open” label that quickly attached to the model. Falcon 180B’s licence, built on Apache 2.0, permitted free commercial use and modification but carved out an exception: anyone wanting to offer the model as a hosted inference or fine-tuning service to third parties needed separate permission from TII. That restriction — absent from fully permissive licences such as Apache or MIT — put Falcon 180B in the same contested category as Llama 2’s licence, where “open” described weight availability rather than unrestricted commercial freedom.

Falcon 180B’s size made it impractical to run without substantial hardware — Hugging Face noted it required roughly 400GB of memory for full-precision inference — which limited its role mostly to research and to well-resourced deployments rather than the wide hobbyist adoption smaller open releases attracted. It nonetheless served as an early marker that state-funded labs outside the US and China could produce open-weight models competitive with the leading proprietary systems of the time.