Magistral-Small-2506 Overview

Description

Magistral-Small-2506 is a lightweight, general-purpose language model that generates and understands natural language for tasks like Q&A, summarization, and instruction following. Designed for efficiency, it balances performance with low computational overhead, making it suitable for real-world applications. Building upon Mistral Small 3.1 (2503), with added reasoning capabilities, undergoing SFT from Magistral Medium traces and RL on top, it's a small, efficient reasoning model with 24B parameters.

Magistral Small can be deployed locally, fitting within a single RTX 4090 or a 32GB RAM MacBook once quantized.

This model is for research and development only.

Learn more about Magistral in our blog post.
The model was presented in the paper Magistral.

• Supported Languages: Supports dozens of languages, including English, French, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Nepali, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Turkish, Ukrainian, Vietnamese, Arabic, Bengali, Chinese, and Farsi

Key Features

• Reasoning: Capable of long chains of reasoning traces before providing an answer.

• Multilingual: Supports dozens of languages, including English, French, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Nepali, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Turkish, Ukrainian, Vietnamese, Arabic, Bengali, Chinese, and Farsi.

• Context Window: A 128k context window, but performance might degrade past 40k. Hence we recommend setting the maximum model length to 40k.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to Non-NVIDIA MistralAi/Magistral-Small-2506.

License/Terms of Use

GOVERNING TERMS: This trial service is governed by the NVIDIA API Trial Terms of Service. Use of this model is governed by the NVIDIA Community Model License. Additional Information: Apache 2.0.

Deployment Geography

Global

Use Case

Magistral Small is a reasoning model with multilingual capabilities that can be deployed locally, fitting within a single RTX 4090.

Release Date

Build.NVIDIA.com: 2024-07-09 via link
Hugging Face: 2024-06-25 link

Reference(s)

Technical Paper: Magistral

Model Architecture

Architecture Type: Decoder-only (Transformer-based)

Network Architecture: Transformer-based, decoder-only architecture with rotary positional embeddings, multi-head self-attention, and long-context support (up to 128k tokens). Based on Mistral open-weight foundation models.

Model Architecture:

• Architecture Type: Transformer decoder
  
• Network Architecture: Transformer with FlashAttention
  
• Parameter Count: 24B
  
• Base Model: Mistral-Small-3.1-24B-Base-2503
  

Input

Input Type(s): Text

Input Format(s): String

Input Parameters: One-Dimensional (1D)

Other Properties Related to Input: Context Length: 128k

Output

Output Type(s): Text

Output Format(s): String

Output Parameters: One-Dimensional (1D)

Other Properties Related to Output: Max theoretical context length: 128,000 tokens, Practical limit: ≈ 40k tokens (40960)

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.**

Software Integration

Runtime Engine: vLLM

Supported Hardware Microarchitecture Compatibility

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper
• NVIDIA Lovelace

Preferred Operating System(s)

Linux

Model Version(s)

Magistral-Small-2506 1.0

Training, Testing, and Evaluation Datasets

Training Dataset

• Data Collection Method by dataset: Hybrid: Automated, Human
• Labeling Method by dataset: Hybrid: Automated, Human
• Properties: Trained on ~2 trillion tokens from web, books, code, and conversations; no sensors used (not applicable for text-based data).

Testing Dataset

• Data Collection Method by dataset: Hybrid: Automated, Human
• Labeling Method by dataset: Hybrid: Automated, Human
• Properties: Undisclosed

Evaluation Dataset

• Data Collection Method by dataset: Hybrid: Automated, Human

• Labeling Method by dataset: Hybrid: Automated, Human

• Properties: The benchmarks noted in the following session were used for evaluation.

Benchmark Results

Inference

Acceleration Engine: vLLM
Test Hardware:

• L40s x2

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here.