kosmos-2-patch14-224

Microsoft Kosmos-2 is a multimodal large language model for image-to-text tasks that grounds its outputs to specific regions in an input image. It can generate captions, answer visual questions, and perform referring expression comprehension—all while linking phrases to bounding box coordinates. This allows models to not only describe what is in an image but also indicate where each described entity is located.

Key capabilities

• Multimodal grounding: Phrase grounding and referring expression comprehension, where the model localizes objects mentioned in text.
• Grounded visual question answering (VQA): Answers questions about an image and returns bounding boxes for relevant objects.
• Referring expression generation: Given an object region (bounding box), produces a natural language description of that object.
• Grounded image captioning: Generates brief or detailed captions with spatially aligned entity boxes.

Example output

Given the image above, Kosmos-2 can produce the caption “An image of a snowman warming himself by a fire.” and output bounding boxes for “a snowman” and “a fire”.

Kosmos-2 is designed for applications that require fine-grained visual understanding with spatial grounding, such as interactive image analysis, grounded dialogue systems, and visual reasoning. As a hosted API on gigarouter, the model is available through an OpenAI-compatible endpoint, eliminating the need for local setup or GPU infrastructure.