GPT-4V uses a visual encoder based on the ViT (Vision Transformer) architecture to convert images into visual tokens, which are then jointly processed with text tokens through a shared Transformer architecture.