GPT-J-6B: 6B JAX-Based Transformer

GPT-J-6B is a 6-billion-parameter autoregressive language model built on JAX-based Mesh Transformer, released in 2021 by Ben Wang and Aran Komatsuzaki. It is a GPT-2-like causal language model trained on the Pile dataset, performing nearly on par with OpenAI's 6.7B GPT-3 (Curie) on various zero-shot downstream tasks. The project also serves as a reference implementation for model parallelism using xmap on JAX and TPUs, demonstrating high training throughput and efficiency. It is openly licensed under Apache-2.0 and hosted via EleutherAI.