ExLlamaV3ExLlamaV3
ExLlamaV3 是 ExLlamaV2 的接续线：用 EXL3 量化格式在 NVIDIA GPU 上做低延迟本地推理。 它是引擎本体，不是 GGUF 工具，也不是 API server。
跨大类对照：LlamaIndex。
The successor to ExLlamaV2: low-latency local inference on NVIDIA GPUs using the EXL3 quantisation format. It is an engine, not a GGUF tool and not an API server.
适合与不适合When to use it, when not to
适合：NVIDIA 单机/多卡、追 EXL3 量化、愿意自己配 CUDA 环境。 不适合：AMD/CPU、想要开箱 GUI、依赖 GGUF 生态。
Fits NVIDIA single/multi-GPU users chasing EXL3 quantisation and low-latency inference who are willing to set up CUDA. Does not fit AMD/CPU, an out-of-the-box GUI, or the GGUF ecosystem.
中文生态China availability 国产 / 国内可直接用 / 备案状态 · 快照domestic / direct access / CAC filing · snapshot 2026-11-03
- 出品属地Origin
- 国外出品foreign-made turboderp-org（开源社区项目）。turboderp-org (open-source community project).
- 国内使用Direct access
- 大陆可直连direct OK github.com 大陆直连不稳定；PyPI 可经镜像获取。github.com direct access from mainland is unstable; PyPI is available through mirrors.
- 备案状态CAC filing
- 未见于备案名单not in filing list：未见于网信办公示名单（2026-11-03 检索）；开源项目未面向大陆公众提供服务，备案不适用。Not in the CAC public registry (searched 2026-11-03); the open-source project does not serve the mainland public, so filing does not apply.
备案信息以网信办《生成式人工智能服务已备案信息》公示名单为准（按批次滚动更新，本站只抄编号不解读）；「大陆可直连」是本站从大陆网络直连实测的结果，不是厂商承诺。GitHub 类站点当日实测有波动，口径写在依据里。Filing status is quoted from the CAC public registry (updated batch by batch; this site copies the number and does not interpret it). "Direct access" is this site's own reachability measurement from a mainland network, not a vendor promise. GitHub-hosted sites fluctuated on the measurement day — the note says so.
固定坐标系Fixed coordinate system
8 个维度dimensions, 与同分区其他对象逐项可比each comparable item by item against others in the same section.
- 模型与开放条件Model & openness
EXL3 量化格式（.exl3 checkpoint）与部分 HF safetensors；模型支持以官方仓与文档为准， 本站未逐条清点。它是 ExLlamaV2 的接续线。
Uses EXL3 quantised checkpoints (.exl3) and some HF safetensors; supported models follow the official repository and docs and are not enumerated here. It is the successor line to ExLlamaV2.
- 运行位置Where it runs
CUDA 推理引擎，面向本地/单机多卡；PyPI 包 exllamav3（1.6.5，2026-10-25 发布）。 需要 CUDA 环境，具体版本以官方仓为准。
A CUDA inference engine for local/single-machine multi-GPU use; the PyPI package is exllamav3 (1.6.5, released 2026-10-25). A CUDA environment is required; exact versions follow the official repo.
- 本地文件Local files
自托管，权重与 .exl3 checkpoint 在本地；没有官方托管云。
Self-hosted, with weights and .exl3 checkpoints on the local machine; there is no official hosted cloud.
- 关机后的任务Tasks after shutdown
本地推理进程，不是常驻服务；要服务化通常接 TabbyAPI 等外层。
A local inference process, not a long-running service; serving usually goes through an outer layer such as TabbyAPI.
- 工具与扩展Tools & extensions
以推理执行为主；工具调用与 MCP 不在核心面，未逐条核验， 需要这些能力时通常在外层应用里补。
Primarily inference execution; tool calling and MCP are not core and are not verified item by item.
- 上下文与记忆Context & memory
长上下文按显存与配置；KV cache 细节在官方文档线，未逐条核验。
Long context depends on VRAM and configuration; KV-cache details live in official docs and are not verified item by item.
- 权限与限制Permissions & limits
本地进程无账号体系；权限由运行环境决定，对外提供服务时需要自己加鉴权与访问控制。
A local process with no account system; permissions come from the runtime environment.
- 适合什么任务Best for
适合：NVIDIA 单机/多卡、追 EXL3 量化与低延迟推理、愿意自己配 CUDA 环境。 不适合：AMD/CPU、想要开箱 GUI、想要 GGUF 生态。
Fits NVIDIA single/multi-GPU users chasing EXL3 quantisation and low-latency inference who are willing to set up CUDA. Does not fit AMD/CPU, an out-of-the-box GUI, or the GGUF ecosystem.
头号误解Top misconception
- 把 ExLlamaV3 当成 llama.cpp 的替代——它走 EXL3/CUDA 路线，不是 GGUFTreating ExLlamaV3 as a replacement for llama.cpp — it uses EXL3/CUDA, not GGUF
- 以为装了包就能跑——还需要 CUDA 与对应模型 checkpointAssuming installing the package is enough — you still need CUDA and a matching checkpoint
- 把 TabbyAPI 当成 ExLlama 本体——TabbyAPI 是外层 API serverTreating TabbyAPI as ExLlama itself — TabbyAPI is the outer API server
未知项清单Known unknowns
- 当前支持的模型架构清单（官方仓为准，未逐条清点）
- 各版本 CUDA 要求与显存下限
- 与 TabbyAPI 的兼容矩阵
- Current supported model-architecture list (official repo is authoritative; not enumerated here)
- CUDA requirements and VRAM floors per release
- Compatibility matrix with TabbyAPI
证据来源Evidence sources
判断可回到以下一手源复核。本站核验日Every judgement can be rechecked against the primary sources below. Verified on 2026-11-03，内容更新日content updated on 2026-11-03。
| 类型Type | 名称Name | 链接Link |
|---|---|---|
| repo | turboderp-org · ExLlamaV3 官方仓库（README / docs；2026-11-03 复核官方页）turboderp-org · ExLlamaV3 official repository (README / docs) | https://github.com/turboderp-org/exllamav3 |
| docs | PyPI · exllamav3（1.6.5，2026-10-25；2026-11-03 复核）PyPI · exllamav3 (1.6.5, 2026-10-25; rechecked 2026-11-03) | https://pypi.org/project/exllamav3/ |
| docs | ExLlamaV3 · convert 文档（EXL3 checkpoint 转换）ExLlamaV3 · convert docs (EXL3 checkpoint conversion) | https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md |
| changelog | ExLlamaV3 · Releases（官方 changelog）ExLlamaV3 · Releases (official changelog) | https://github.com/turboderp-org/exllamav3/releases |
实测记录Measured runs
本档未跑吞吐与显存实测；结论只覆盖运行时能力与许可。
This site has not completed throughput or VRAM tests; the conclusions cover runtime capability and licensing only.
相关条目Related entries
- TabbyAPITabbyAPIExLlama 的外层 API server，二者常一起用The outer API server for ExLlama; the two are commonly used together.
- llama.cppllama.cppGGUF 路线，与 EXL3/CUDA 路线对照The GGUF route, contrasting with the EXL3/CUDA route.
- LlamaIndex FrameworkLlamaIndex Framework另一层 · 框架与运行时Other layer · Frameworks & runtimes
本页由This page is generated from the ai-handbook 数据层生成（CC BY 4.0）。data layer (CC BY 4.0).
方法论与坐标系定义见仓库内For the methodology and coordinate definitions, see METHODOLOGY.md.