API Reference
API Reference
API Catalog

ibm / granite-34b-code-instruct

Model Overview

Description

Granite-34B-Code-Instruct generates, explains, and translates code from a natural language prompt. It is a 34B parameter model fine tuned from Granite-34B-Code-Base on a combination of permissively licensed instruction data to enhance instruction following capabilities including logical reasoning and problem-solving skills. This model is ready for commercial use.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party’s requirements for this application and use case; see link to the Granite-34B-Code-Instruct Model Card.

License and Terms of use

GOVERNING TERMS: Your use of this API is governed by the NVIDIA API Trial Service Terms of Use; and the use of this model is governed by the NVIDIA AI Foundation Models Community License and Apache 2.0 License.

Model Developer: IBM Research

Model Release Date: May 6th, 2024

Model Architecture

  • Architecture Type: Transformer
  • Network Architecture: GPT Big Code

Input

  • Input Type: Text
  • Input Format: String
  • Input Parameters: max_tokens, temperature, top_p, stop, frequency_penalty, presence_penalty, seed

Output

  • Output Type: Text
  • Output Format: String

Software Integration:

  • Supported Hardware Platform(s): NVIDIA Hopper

[Preferred/Supported] Operating System(s):

  • Linux

Training Data

Granite Code Instruct models are trained on the following types of data.

  • Code Commits Datasets: we sourced code commits data from the CommitPackFT dataset, a filtered version of the full CommitPack dataset. From CommitPackFT dataset, we only consider data for 92 programming languages. Our inclusion criteria boils down to selecting programming languages common across CommitPackFT and the 116 languages that we considered to pretrain the code-base model (Granite-8B-Code-Base).
  • Math Datasets: We consider two high-quality math datasets, MathInstruct and MetaMathQA. Due to license issues, we filtered out GSM8K-RFT and Camel-Math from MathInstruct dataset.
  • Code Instruction Datasets: We use Glaive-Code-Assistant-v3, Glaive-Function-Calling-v2, NL2SQL11 and a small collection of synthetic API calling datasets.
  • Language Instruction Datasets: We include high-quality datasets such as HelpSteer and an open license-filtered version of Platypus. We also include a collection of hardcoded prompts to ensure our model generates correct outputs given inquiries about its name or developers.

Inference

Engine: Triton + TensorRT-LLM

Test Hardware: H100

Ethical Considerations and Limitations

Granite code instruct models are primarily finetuned using instruction-response pairs across a specific set of programming languages. Thus, their performance may be limited with out-of-domain programming languages. In this situation, it is beneficial providing few-shot examples to steer the model's output. Moreover, developers should perform safety testing and target-specific tuning before deploying these models on critical applications. The model also inherits ethical considerations and limitations from its base model. For more information, please refer to Granite-34B-Code-Base model card.