Industry estimates put the training cost of GPT-4 at over $100 million, involving tens of thousands of A100 GPUs, months of continuous computation, and petabyte-scale training data.