Each inference requires loading model weights into GPU memory and generating responses token by token; GPT-4-level model parameter counts are typically on the order of hundreds of billions.