OpenAI announced GPT-2 on February 14, 2019, but withheld its release due to concerns about safety and potential abuse. The decision was met with criticism from some researchers who felt it limited opportunities for study and understanding of the model.

GPT-2 is a scaled-up version of GPT-1, featuring more parameters and trained on a larger dataset. OpenAI decided not to release the full model due to concerns about potential malicious applications, opting instead to release a smaller model and a technical paper for research purposes.

GitHub repository angelos-p/llm-from-scratch offers a hands-on workshop for building a GPT training pipeline from scratch. The workshop focuses on reproducing the GPT-2 model with 124 million parameters using PyTorch.

Enabled fp8 training for GPT-2 resulted in a 4.3% reduction in training time, bringing it down to 2.91 hours. Using 8XH100 spot instance prices, the cost to reproduce GPT-2 is approximately $20.