A vision encoder is a necessary component for allowing many leading LLMs to be able to work with images uploaded by users.

It employs a vision transformer encoder alongside a large language model (LLM). The vision encoder converts images into tokens, which an attention-based extractor then aligns with the LLM.

LLaVA 1.5 improves upon the original by connecting the language model and vision encoder through a multi-layer perceptron (MLP), a simple deep learning model where all neurons are fully connected.

Gemma 3 packs an upgraded vision encoder that handles high-res and non-square images with ease. It also includes the ShieldGemma 2 image safety classifier, ...