Field note
What it does
Run LLM inference at maximum throughput This example demonstrates some techniques for running LLM inference at the highest possible throughput on Modal. For more on other aspects of maximizing the performance of LLM inference, see our guide. For a simpler introduction to LLM serving, see this example. As our sample application, we use an LLM to summarize thousands of filings with the U.S. federal government's Securities and Exchange Commission (SEC), made available to the public for free in daily data dumps via the SEC's Electronic Data Gathering, Analysis, and Retrieval System (EDGAR). We like to check out the Form 4s, which detail (legal) insider trading. Using the Qwen 3 8B parameter LLM on this task, which has inputs that average a few thousand tokens and outputs that average a few hundred tokens, we observe processing speeds of 30,000 input tok/s and 2,000 output tok/s per H100 GPU, as in the sample Modal Dashboard screenshot below. Note the 100% GPU utilization, indicating the absence of host overhead, and the high GPU power utilization, further indicating we are close to the hardware's physical limits.
Capabilities
Available capabilities
Tags
Tags
No tags filed yet.
Ways to use it
Ways to use it
No integrations filed yet.