Instructions to use nvidia/NVIDIA-Nemotron-Parse-v1.1-TC with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/NVIDIA-Nemotron-Parse-v1.1-TC with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nvidia/NVIDIA-Nemotron-Parse-v1.1-TC", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("nvidia/NVIDIA-Nemotron-Parse-v1.1-TC", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/NVIDIA-Nemotron-Parse-v1.1-TC with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/NVIDIA-Nemotron-Parse-v1.1-TC"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-Parse-v1.1-TC",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/NVIDIA-Nemotron-Parse-v1.1-TC

SGLang

How to use nvidia/NVIDIA-Nemotron-Parse-v1.1-TC with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/NVIDIA-Nemotron-Parse-v1.1-TC" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-Parse-v1.1-TC",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/NVIDIA-Nemotron-Parse-v1.1-TC" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-Parse-v1.1-TC",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nvidia/NVIDIA-Nemotron-Parse-v1.1-TC with Docker Model Runner:
```
docker model run hf.co/nvidia/NVIDIA-Nemotron-Parse-v1.1-TC
```

NVIDIA-Nemotron-Parse-v1.1-TC / README.md

amalad

Update README

a3b2937 verified 10 months ago

10.8 kB

license: other
license_name: nvidia-community-model-license
license_link: >-
  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - nvidia
  - VLM
  - OCR

NVIDIA Nemotron Parse v1.1 Lite Overview

NVIDIA Nemotron Parse v1.1 Lite is designed to understand document semantics and extract text and tables elements with spatial grounding. Given an image, NVIDIA Nemotron Parse v1.1 Lite produces structured annotations, including formatted text, bounding-boxes and the corresponding semantic classes, ordered according to the document's reading flow. It overcomes the shortcomings of traditional OCR technologies that struggle with complex document layouts with structural variability, and helps transform unstructured documents into actionable and machine-usable representations. This has several downstream benefits such as increasing the availability of training-data for Large Language Models (LLMs), improving the accuracy of extractor, curator, retriever and AI agentic applications, and enhancing document understanding pipelines.

This model is ready for commercial use.

License

GOVERNING TERMS: The NIM container is governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products. Use of this model is governed by the NVIDIA Community Model License. Use of the tokenizer included in this model is governed by the CC-BY-4.0 license.

Deployment Geography:

Global

Use Case:

NVIDIA Nemotron Parse 1.1 Lite will be capable of comprehensive text understanding and document structure understanding. It will be used in retriever and curator solutions. Its text extraction datasets and capabilities will help with LLM and VLM training, as well as improve run-time inference accuracy of VLMs. The NVIDIA Nemotron Parse 1.1 Lite model will perform text extraction from PDF and PPT documents. The NVIDIA Nemotron Parse 1.1 Lite can classify the objects (title, section, caption, index, footnote, lists, tables, bibliography, image) in a given document, and provide bounding boxes with coordinates.

Release Date:

November 17, 2025

References

https://huggingface.co/docs/transformers/en/model_doc/mbart

Model Architecture

Architecture Type :

Transformer-based vision-encoder-decoder model

Network Architecture

Vision Encoder: ViT-H model (https://huggingface.co/nvidia/C-RADIO)
Adapter Layer: 1D convolutions & norms to compress dimensionality and sequence length of the latent space (13184 tokens to 833 tokens)
Decoder: mBart [1] 10 blocks
Tokenizer: Use of the tokenizer included in this model is governed by the CC-BY-4.0 license
Number of Parameters: < 1B

Computational Load (For NVIDIA Models Only)

Cumulative Compute: 2.2e+22
Estimated Energy and Emissions for Model Training: Energy Consumption: 7,827.46 kWh
Carbon Emissions: 3.21 tCO2e

Input

Input Type: Image, Text
Input Type(s): Red, Green, Blue (RGB) + Prompt (String)
Input Parameters: 2D, 1D

Other Properties Related to Input:
- Max Input Resolution (Width, Height): 1664, 2048
- Min Input Resolution (Width, Height): 1024, 1280
Channel Count: 3

Output

Output Type: Text
Output Format: String
Output Parameters: 1D

Other Properties Related to Output:
- NVIDIA Nemotron Parse 1.1 Lite output format is a string which encodes text content (formatted or not) as well as bounding boxes and class attributes.
  In the default prompt setting, text content is represented as markdown, and math expressions as LaTeX, enclosed in [..] or (..). If a mathematical expression does not require LaTeX formatting to be represented (e.g., consisting only of characters and subscripts/superscripts), it is represented as markdown. Tables are represented as LaTeX. Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s): TensorRT-LLM

Supported Hardware Microarchitecture Compatibility:
NVIDIA Hopper/NVIDIA Ampere/NVIDIA Turing

Supported Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version:

V1.1-Lite This version offers 20% speed improvement compared to Nemotron-Parse-v1.1. Additionally, unlike Nemotron-Parse-v1.1, it preserves the page order of un-ordered element: Tables, Captions, Pictures, Footnotes.

Quick Start

Install dependencies

pip install -r requirements.txt

Usage example

import torch
from PIL import Image, ImageDraw
from transformers import AutoModel, AutoProcessor, AutoTokenizer, AutoConfig, AutoImageProcessor, GenerationConfig
from postprocessing import extract_classes_bboxes, transform_bbox_to_original, postprocess_text

# Load model and processor
model_path = "nvidia/NVIDIA-Nemotron-Parse-v1.1-Lite"  # Or use a local path
device = "cuda:0"

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16
).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(model_path)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

# Load image
image = Image.open("path/to/your/image.jpg")
task_prompt = "</s><s><predict_bbox><predict_classes><output_markdown>"

# Process image
inputs = processor(images=[image], text=task_prompt, return_tensors="pt").to(device)
prompt_ids = processor.tokenizer.encode(task_prompt, return_tensors="pt", add_special_tokens=False).cuda()


generation_config = GenerationConfig.from_pretrained(model_path, trust_remote_code=True)
# Generate text
outputs = model.generate(**inputs,  generation_config=generation_config)

# Decode the generated text
generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0]

Postprocessing

import torch
from PIL import Image, ImageDraw
from transformers import AutoModel, AutoProcessor, AutoTokenizer, AutoConfig, AutoImageProcessor, GenerationConfig
from postprocessing import extract_classes_bboxes, transform_bbox_to_original, postprocess_text
classes, bboxes, texts = extract_classes_bboxes(generated_text)
bboxes = [transform_bbox_to_original(bbox, image.width, image.height) for bbox in bboxes]

# Specify output formats for postprocessing
table_format = 'latex' # latex | HTML | markdown
text_format = 'markdown' # markdown | plain
blank_text_in_figures = False # remove text inside 'Picture' class
texts = [postprocess_text(text, cls = cls, table_format=table_format, text_format=text_format, blank_text_in_figures=blank_text_in_figures) for text, cls in zip(texts, classes)]

for cl, bb, txt in zip(classes, bboxes, texts):
    print(cl, ': ', txt)

draw = ImageDraw.Draw(image)
for bbox in bboxes:
  draw.rectangle((bbox[0], bbox[1], bbox[2], bbox[3]), outline="red")

Inference with VLLM

Install dependencies

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install "git+https://github.com/amalad/vllm.git@nemotron_parse"
uv pip install timm albumentations

Inference example

from vllm import LLM, SamplingParams
from PIL import Image


sampling_params = SamplingParams(
    temperature=0,
    top_k=1,
    repetition_penalty=1.1,
    max_tokens=9000,
    skip_special_tokens=False,
)

llm = LLM(
    model="nvidia/NVIDIA-Nemotron-Parse-v1.1-Lite",
    max_num_seqs=64,
    limit_mm_per_prompt={"image": 1},
    dtype="bfloat16",
    trust_remote_code=True,
)

image = Image.open("<YOUR-IMAGE-PATH>")

prompts = [
    {  # Implicit prompt
        "prompt": "</s><s><predict_bbox><predict_classes><output_markdown>",
        "multi_modal_data": {
            "image": image
        },
    },
    {  # Explicit encoder/decoder prompt
        "encoder_prompt": {
            "prompt": "",
            "multi_modal_data": {
                "image": image
            },
        },
        "decoder_prompt": "</s><s><predict_bbox><predict_classes><output_markdown>",
    },
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Decoder prompt: {prompt!r}, Generated text: {generated_text!r}")

Training, Testing, and Evaluation Datasets:

Training Dataset

NVIDIA Nemotron Parse 1.1 Lite is first pre-trained on our internal datasets: human, synthetic and automated. Data Modality: *Text *Image
Data Collection Method by Dataset: Hybrid: Human, Synthetic, Automated Labeling Method by Dataset: Hybrid: Human, Synthetic, Automated

Testing and Evaluation Dataset:

NVIDIA Nemotron Parse 1.1 Lite is evaluated on multiple datasets for robustness, including public and internal dataset. Data Collection Method by Dataset: Hybrid: Human, Synthetic, Automated Labeling Method by Dataset: Hybrid: Human, Synthetic, Automated

Inference

Runtime Engine(s): TensorRT-LLM

Test Hardware: NVIDIA H100# Synchronization

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here.

You are responsible for ensuring that your use of NVIDIA AI Models complies with all applicable laws.

Enterprise Support

Get access to knowledge base articles and support cases or submit a ticket.