Note

• Use this model according to the Llama 3.2 Community License (a custom, commercial license agreement).
• Datatype: 4-bit bitsandbytes Model

Introduction

Llama-3.2-11B-Vision-Instruct is part of the Llama 3.2 Vision collection developed by Meta, a multimodal large language model (LMM) family that integrates image and text reasoning. This 11-billion parameter model is designed for a wide array of vision-language tasks such as visual recognition, image reasoning, and captioning. Llama-3.2-Vision models offer superior performance across a variety of image-text benchmarks, leveraging the power of both pretrained and instruction-tuned multimodal learning.

Llama-3.2-11B-Vision-Instruct Model

• Developer: Meta
• Model Size: 11 billion parameters (10.6B)
• Architecture: Llama-3.2-11B-Vision-Instruct is built on the Llama-3.1 architecture, an auto-regressive model based on an optimized transformer structure. It incorporates a vision adapter that enables image reasoning capabilities through cross-attention layers, integrating image encoder representations into the core language model.
• Context Length: 128k
• Knowledge Cutoff: December 2023
• Supported Languages: Officially supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai for text-only tasks, with broader language support available in limited capacities. For image-text applications, only English is supported.

Run Llama-3.2 with an API

Running the API with Clarifai's Python SDK

You can run the Llama-3.2 Model API using Clarifai’s Python SDK.

Export your PAT as an environment variable. Then, import and initialize the API Client.

Find your PAT in your security settings.

export CLARIFAI_PAT={your personal access token}

Predict via Image URL

from clarifai.client.model import Model
from clarifai.client.input import Inputs

prompt = "What time of day is it?"
image_url = "https://samples.clarifai.com/metro-north.jpg"

inference_params = dict(temperature=0.2, max_tokens=100, top_k=50, top_p= 0.9)

model_prediction = Model("https://clarifai.com/meta/Llama-3/models/llama-3_2-11b-vision-instruct").predict(inputs = [Inputs.get_multimodal_input(input_id="",image_url=image_url, raw_text=prompt)],inference_params=inference_params)

print(model_prediction.outputs[0].data.text.raw)

You can also run Llama-3.2 API using other Clarifai Client Libraries like Java, cURL, NodeJS, PHP, etc here.

Predict via local Image

from clarifai.client.model import Model
from clarifai.client.input import Inputs

IMAGE_FILE_LOCATION = 'LOCAL IMAGE PATH'
with open(IMAGE_FILE_LOCATION, "rb") as f:
file_bytes = f.read()

prompt = "What time of day is it?"
inference_params = dict(temperature=0.2, max_tokens=100, top_k=50, top_p= 0.9)

model_prediction = Model("https://clarifai.com/meta/Llama-3/models/llama-3_2-11b-vision-instruct").predict(inputs = [Inputs.get_multimodal_input(input_id="", image_bytes = file_bytes, raw_text=prompt)], inference_params=inference_params)

print(model_prediction.outputs[0].data.text.raw)

Use Cases

Llama-3.2-11B-Vision-Instruct is built for a range of commercial and research applications that combine both visual and textual inputs. Key use cases include:

1. Visual Question Answering (VQA) and Visual Reasoning:

• Users can ask the model questions about images, and the model generates coherent responses based on its understanding of the visual content.

2. Document Visual Question Answering (DocVQA):

• Llama-3.2-11B-Vision-Instruct can analyze complex visual documents like contracts or forms and answer questions regarding their content, taking into account both text and layout.

3. Image Captioning:

• The model excels at producing detailed, coherent captions for images by analyzing visual elements and describing them accurately in natural language.

4. Image-Text Retrieval:

• The model can be used to match images with relevant text descriptions, making it suitable for applications like multimedia search engines.

5. Visual Grounding:

• Llama-3.2-11B-Vision-Instruct can link textual references to specific objects or regions in images, enhancing understanding in tasks that require spatial awareness.

Evaluation and Benchmark Results

Llama-3.2-11B-Vision-Instruct outperforms many existing multimodal models on industry-standard benchmarks, including tasks related to Visual Question Answering (VQA), image captioning, and image-text retrieval.

The following benchmarks reflect the model's strengths:

• VQA: High accuracy in answering questions about a diverse set of images.
• Image Captioning: State-of-the-art performance on COCO Captions and similar datasets.
• Image-Text Retrieval: Consistently strong results in pairing images with their correct textual descriptions.

Dataset

Llama-3.2-11B-Vision-Instruct was trained on a large multimodal dataset of 6 billion image-text pairs. The dataset includes diverse visual content from various domains, allowing the model to generalize well across multiple image reasoning and text generation tasks. The training data is carefully selected to ensure a wide range of real-world image-text scenarios.

Advantages

1. Multimodal Capabilities: Llama-3.2-11B-Vision-Instruct supports both text and image inputs, enabling sophisticated visual reasoning and the generation of human-like responses based on complex images.
2. Superior Visual Reasoning: With its vision adapter, the model excels in tasks requiring a deep understanding of visual content, outperforming competitors in tasks such as VQA and image captioning.
3. Instruction-Tuned: The model has been tuned to follow instructions accurately, making it highly suitable for applications where human-like interaction is required.
4. Large Context Handling: With a 128k token context length, the model is equipped to manage complex, multi-turn conversations and reasoning over large pieces of visual and textual data.

Limitations

1. English-Limited for Multimodal Tasks: While the model supports several languages for text-only tasks, its multimodal (image-text) capabilities are currently limited to English.
2. Bias in Training Data: As with many large models, Llama-3.2-11B-Vision-Instruct inherits biases present in its training data, which may result in skewed or inappropriate outputs in certain scenarios.
3. Knowledge Cutoff: The model’s knowledge is capped at December 2023, meaning it may not be aware of more recent events or developments.