APIActive

ONNX Runtime GenAI C API

C API for ONNX Runtime GenAI, enabling generative AI model loading, tokenization, and generation.

Field note

What it does

The C API for ONNX Runtime GenAI provides functions to create models, manage configurations, tokenize text, and generate tokens. It includes APIs for model creation, tokenizer, sequences, generator parameters, generator, adapters, tensors, images/audios, named tensors, and utility functions.

Capabilities

Available capabilities

Chat Template Tensor Handling Advanced Analysis/Tokenization Tensor Operations Text generation with constraints Multimodal Processing Configuration Multimodal Support Model Management Model Loading

Configure Providers

Clear, append, and set options for execution providers.

Decode Batch

Decode a batch of sequences.

Encode Batch

Encode a batch of strings.

Set Generation Parameters

Set search options like max_length and do_sample for generation.

Generator Parameters

Graph Capture

Enable graph capture mode with a maximum batch size.

Create Model

Create a model from a configuration directory with optional runtime settings.

Model Creation

Create models from directory, runtime settings, or config.

Generate

Generate output sequences using generator parameters.

Multimodal Inputs

Set Active Adapter

Set the active adapter on a generator instance.

Set Model Input

Set an additional model input.

Token Streaming

Incrementally decode tokens using a tokenizer stream.

Encode and Decode

Encode strings to token sequences and decode tokens back to strings.

Tokenizer Encode/Decode

Encode and decode text, apply chat templates, and stream decode.

Tokenizer Stream

Create a tokenizer stream for incremental decoding.