nemoretriever-parse Overview

Description:

nemoretriever-parse is a general purpose text-extraction model, specifically designed to handle documents. Given an image, nemoretriever-parse is able to extract formatted-text, with bounding-boxes and the corresponding semantic class. This has downstream benefits for several tasks such as increasing the availability of training-data for Large Language Models (LLMs), improving the accuracy of retriever systems, and enhancing document understanding pipelines.

This model is for demonstration purposes and it is not for production usage.

License/Terms of Use:

GOVERNING TERMS: The NIM container is governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products. Use of this model is governed by the NVIDIA Community Model License.

Deployment Geography:

Global

Use Case:

nemoretriever-parse will be capable of comprehensive text understanding and document structure understanding.
It will be used in retriever and curator solutions. Its text extraction datasets and capabilities will help with LLM
and VLM training, as well as improve run-time inference accuracy of VLMs.

The nemoretriever-parse model will perform text extraction from PDF and PPT documents.
The nemoretriever-parse can classify the objects (title, section, caption, index, footnote, lists, tables, bibliography, image)
in a given document, and provide bounding boxes with coordinates.

Release Date

March 5th 2025

Reference:

https://huggingface.co/docs/transformers/en/model_doc/mbart

Model Architecture:

Architecture Type: Transformer-based vision-encoder-decoder model

Network Architecture:

• Vision Encoder: ViT-H model (https://huggingface.co/nvidia/C-RADIO)
• Adapter Layer: 1D convolutions & norms to compress dimensionality and sequence length of the latent space (1280 tokens to 320 tokens)
• Decoder: mBart [1] 10 blocks
• Tokenizer: Galactica (https://arxiv.org/abs/2211.09085); same as Nougat tokenizer
  

Input:

Input Type(s): Image

Input Format(s): Red, Green, Blue (RGB)

Input Parameters: 2D

Other Properties Related to Input:

• Max Input Resolution (Width, Height): 1648, 2048
• Min Input Resolution (Width, Height): 1024, 1280
• Channel Count: 3
  

Output:

Output Type(s): Text

Output Format: String

Output Parameters: 1D

Other Properties Related to Output: nemoretriever-parse output format is a string which encodes text content (formatted or not) as well as bounding boxes and class attributes.

Software Integration:

Runtime Engine(s): TensorRT-LLM

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Hopper
  
• NVIDIA Ampere
  
• NVIDIA Turing
  

[Preferred/Supported] Operating System(s):

• Linux
  

Model Version(s):

nemoretriever-parse-v0.2-beta: As part of this first release, we share the set of weights named overjoyed-adder.

Training, Testing, and Evaluation Datasets:

Training Dataset:

nemoretriever-parse is first pre-trained on our internal datasets: human, synthetic, and automated.

Testing and Evaluation:

nemoretriever-parse is evaluated on multiple datasets for robustness, including public and internal dataset.

Inference:

Engine: TensorRT-LLM

Test Hardware:

• NVIDIA H100
  

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here.