NVIDIA TensorRT-LLMNVIDIA TensorRT-LLM
TensorRT-LLM 是 NVIDIA 的生产级 LLM 推理栈：先把模型编译成 TensorRT 引擎，再用 trtllm-serve 提供 OpenAI 兼容端点。 它面向 NVIDIA GPU 集群，不是单机开箱工具。
跨大类对照：LangGraph。
NVIDIA's production LLM inference stack: compile the model into a TensorRT engine, then serve it through trtllm-serve with an OpenAI-compatible endpoint. Built for NVIDIA GPU clusters, not single-machine download-and-run.
适合与不适合When to use it, when not to
适合：NVIDIA GPU 集群、要生产吞吐与低延迟、已有容器与编译流水线。 不适合：单卡消费级开箱、AMD/CPU 部署、想直接下 GGUF 就跑。
Fits NVIDIA GPU clusters that need production throughput and low latency and already have container/compile pipelines. Does not fit single-card consumer setups, AMD/CPU deployment, or download-and-run GGUF workflows.
中文生态China availability 国产 / 国内可直接用 / 备案状态 · 快照domestic / direct access / CAC filing · snapshot 2026-11-03
- 出品属地Origin
- 国外出品foreign-made NVIDIA（美国）。NVIDIA (US).
- 国内使用Direct access
- 直连不可达direct blocked docs.nvidia.com 大陆直连未核验；NVIDIA 文档与 NGC 通常需网络环境。Direct reachability from mainland not verified; NVIDIA docs and NGC usually need a network path.
- 备案状态CAC filing
- 未见于备案名单not in filing list：未见于网信办公示名单（2026-11-03 检索）；开发工具未面向大陆公众提供服务，备案不适用。Not in the CAC public registry (searched 2026-11-03); the developer tool does not serve the mainland public, so filing does not apply.
备案信息以网信办《生成式人工智能服务已备案信息》公示名单为准（按批次滚动更新，本站只抄编号不解读）；「大陆可直连」是本站从大陆网络直连实测的结果，不是厂商承诺。GitHub 类站点当日实测有波动，口径写在依据里。Filing status is quoted from the CAC public registry (updated batch by batch; this site copies the number and does not interpret it). "Direct access" is this site's own reachability measurement from a mainland network, not a vendor promise. GitHub-hosted sites fluctuated on the measurement day — the note says so.
固定坐标系Fixed coordinate system
8 个维度dimensions, 与同分区其他对象逐项可比each comparable item by item against others in the same section.
- 模型与开放条件Model & openness
面向 NVIDIA GPU 的 LLM 推理工具包：Python API 定义模型，trtllm-serve 提供服务端点； 支持模型清单以官方文档为准，本站未逐条清点。
An LLM inference toolkit for NVIDIA GPUs: model definitions go through a Python API and trtllm-serve exposes a serving endpoint; the supported-model list follows official docs and is not enumerated here.
- 运行位置Where it runs
编译 + 服务化：把模型编译成 TensorRT 引擎，再通过 trtllm-serve 暴露 OpenAI 兼容接口； 官方定位是数据中心/云端的大规模生产部署。
Compile then serve: the model is compiled into a TensorRT engine and exposed through trtllm-serve with an OpenAI-compatible interface; the official positioning is large-scale data-centre/cloud deployment.
- 本地文件Local files
自托管，权重与引擎在自有环境；NVIDIA GPU 限定，CPU/AMD 加速不在官方支持范围。
Self-hosted, with weights and engines in your own environment; NVIDIA GPUs only, with CPU/AMD acceleration outside the official support scope.
- 关机后的任务Tasks after shutdown
常驻服务形态，适合批处理与高并发；没有「关机后续跑」的官方声明， 任务队列与重试由部署方负责，进程退出后需重新加载引擎。
A long-running service suited to batching and high concurrency; no official statement of continuing after shutdown.
- 工具与扩展Tools & extensions
trtllm-serve 提供 API；工具调用与 MCP 支持状态未逐条核验。
trtllm-serve provides the API; tool-calling and MCP support not verified item by item.
- 上下文与记忆Context & memory
长上下文按模型与显存配置；KV cache 优化在官方文档线，细节未逐条核验。
Long context depends on model and VRAM; KV-cache optimisation lives in official docs, not verified item by item.
- 权限与限制Permissions & limits
自托管无账号体系；鉴权、配额与多租户隔离由部署方自配，官方不提供内置租户模型， GPU 资源配额需外部调度。
Self-hosted with no account system; authentication, quotas and multi-tenant isolation are configured by the deployer.
- 适合什么任务Best for
适合：NVIDIA GPU 集群、要生产级吞吐与低延迟、已有容器/编译流水线。 不适合：单机消费级显卡开箱、AMD/CPU 部署、想要 GGUF 即下即跑。
Fits NVIDIA GPU clusters that need production throughput and low latency and already have container/compile pipelines. Does not fit single-card consumer setups, AMD/CPU deployment, or download-and-run GGUF workflows.
头号误解Top misconception
- 把 TensorRT-LLM 当成 llama.cpp 的替代——它需要编译引擎，部署链更重Treating TensorRT-LLM as a drop-in replacement for llama.cpp — it compiles engines and the deployment chain is heavier
- 以为 AMD/CPU 也能跑——官方定位是 NVIDIA GPUAssuming AMD/CPU also run — the official positioning is NVIDIA GPUs
- 把官方性能数字当本站实测——本站不跑吞吐基准Taking official performance numbers as this site’s measurements — no throughput benchmarks are run here
未知项清单Known unknowns
- 当前支持模型清单（官方文档为准，未逐条清点）
- 工具调用与 MCP 支持状态
- 各版本编译时间与引擎体积
- Current supported-model list (official docs are authoritative; not enumerated here)
- Tool-calling and MCP support status
- Compile time and engine size per release
证据来源Evidence sources
判断可回到以下一手源复核。本站核验日Every judgement can be rechecked against the primary sources below. Verified on 2026-11-03，内容更新日content updated on 2026-11-03。
| 类型Type | 名称Name | 链接Link |
|---|---|---|
| docs | NVIDIA · TensorRT-LLM 官方文档（Python API / trtllm-serve / 生产定位；2026-11-03 复核官方页）NVIDIA · TensorRT-LLM official docs (Python API / trtllm-serve / production positioning) | https://docs.nvidia.com/tensorrt-llm/ |
| docs | NVIDIA · TensorRT Product Family（LLM 推理工具包定位）NVIDIA · TensorRT Product Family (LLM inference toolkit positioning) | https://docs.nvidia.com/tensorrt/index.html |
| repo | NVIDIA · TensorRT-LLM GitHub（代码与 release notes）NVIDIA · TensorRT-LLM GitHub (code and release notes) | https://github.com/NVIDIA/TensorRT-LLM |
| changelog | TensorRT-LLM · Releases（官方 changelog）TensorRT-LLM · Releases (official changelog) | https://github.com/NVIDIA/TensorRT-LLM/releases |
实测记录Measured runs
本档未跑吞吐与显存实测；结论只覆盖运行时能力与许可。
This site has not completed throughput or VRAM tests; the conclusions cover runtime capability and licensing only.
相关条目Related entries
- vLLMvLLM同为服务化推理；vLLM 偏通用 GPU，TensorRT-LLM 偏 NVIDIA 编译优化Both are serving engines; vLLM targets general GPUs while TensorRT-LLM focuses on NVIDIA compilation.
- SGLangSGLang另一条高吞吐服务化路线，对照读Another high-throughput serving route to read in contrast.
- LangGraphLangGraph另一层 · 框架与运行时Other layer · Frameworks & runtimes
本页由This page is generated from the ai-handbook 数据层生成（CC BY 4.0）。data layer (CC BY 4.0).
方法论与坐标系定义见仓库内For the methodology and coordinate definitions, see METHODOLOGY.md.