Model Overview

Description:

The LLaVA-NeXT model was proposed in LLaVA-NeXT: Improved reasoning, OCR, and world knowledge by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVA-NeXT (also called LLaVA-1.6) improves upon LLaVA-1.5 by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning.

LLaVA combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. LLaVA 1.6 improves on LLaVA 1.5 by:

Using Mistral-7B (for this checkpoint) and Nous-Hermes-2-Yi-34B which has better commercial licenses and bilingual support
More diverse and high quality data mixture
Dynamic high resolution

References(s):

Model Architecture:

Architecture Type: Transformer

Network Architecture: Mistral + CLIP

Model version: 7B

Input:

Input Format: Red, Green, Blue (RGB) Image + Text

Input Parameters: temperature, top-p, max output tokens, seed

Output:

Output Format: Text

Output Parameters: None

Software Integration:

Runtime(s): N/A

Supported Hardware Platform(s): Hopper

Supported Operating System(s): Linux

Intended uses & limitations

You can use the raw model for tasks like image captioning, visual question answering, multimodal chatbot use cases.

BibTeX entry and citation info

@misc{liu2023improved,
      title={Improved Baselines with Visual Instruction Tuning}, 
      author={Haotian Liu and Chunyuan Li and Yuheng Li and Yong Jae Lee},
      year={2023},
      eprint={2310.03744},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}