Skip to content

Commit 08e4907

Browse files
eugenehpeugenehp
committed
0.6.0
1 parent a171830 commit 08e4907

67 files changed

Lines changed: 4729 additions & 447 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.cargo/config.toml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Let directly executed binaries find the llama.cpp shared libraries.
2+
#
3+
# With the default `dynamic-link` feature, llama-cpp-sys-4 places the llama/ggml
4+
# shared libraries next to the binaries cargo builds. Finding them at *runtime*
5+
# differs by platform:
6+
#
7+
# - macOS: handled entirely by llama-cpp-sys-4's build script, which rewrites
8+
# the dylib install names to `@loader_path/…`. Nothing is needed here.
9+
# - Windows: the loader already searches the directory of the `.exe`.
10+
# - Linux/BSD: ELF needs the *final executable* to carry an rpath, and a
11+
# dependency's build script cannot inject link args into a dependent
12+
# crate's binary — so it has to be set here.
13+
#
14+
# Downstream users who link dynamically need the same stanza in their own
15+
# project (or `RUSTFLAGS`); see the "Dynamic linking" section of the README.
16+
#
17+
# Note: the `RUSTFLAGS` environment variable *replaces* these flags rather than
18+
# appending, so any CI job that sets `RUSTFLAGS` must repeat the rpath itself.
19+
20+
[target.'cfg(target_os = "linux")']
21+
rustflags = ["-C", "link-arg=-Wl,-rpath,$ORIGIN"]
22+
23+
[target.'cfg(target_os = "freebsd")']
24+
rustflags = ["-C", "link-arg=-Wl,-rpath,$ORIGIN"]

‎.github/workflows/llama-cpp-rs-check.yml‎

Lines changed: 73 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,10 @@ jobs:
1919
env:
2020
RUSTC_WRAPPER: sccache
2121
SCCACHE_GHA_ENABLED: "true"
22-
RUSTFLAGS: -C link-arg=-fuse-ld=mold
22+
# `RUSTFLAGS` replaces .cargo/config.toml rustflags rather than appending,
23+
# so the $ORIGIN rpath that lets directly executed binaries find the
24+
# llama.cpp shared libraries must be repeated here.
25+
RUSTFLAGS: '-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,$ORIGIN'
2326
steps:
2427
- name: Checkout
2528
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
@@ -36,9 +39,11 @@ jobs:
3639
with:
3740
components: clippy, rustfmt
3841
- name: Clippy
39-
run: cargo clippy
42+
run: cargo clippy --workspace --all-targets -- -D warnings
43+
# `cargo fmt` on its own rewrites files in the runner and always exits 0,
44+
# so it never gated anything. `--check` reports and fails instead.
4045
- name: Fmt
41-
run: cargo fmt
46+
run: cargo fmt --all -- --check
4247
- name: Test
4348
run: cargo test --workspace --exclude openai-server --exclude llama-jni
4449
- name: sccache stats
@@ -49,7 +54,10 @@ jobs:
4954
env:
5055
RUSTC_WRAPPER: sccache
5156
SCCACHE_GHA_ENABLED: "true"
52-
RUSTFLAGS: -C link-arg=-fuse-ld=mold
57+
# `RUSTFLAGS` replaces .cargo/config.toml rustflags rather than appending,
58+
# so the $ORIGIN rpath that lets directly executed binaries find the
59+
# llama.cpp shared libraries must be repeated here.
60+
RUSTFLAGS: '-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,$ORIGIN'
5361
steps:
5462
- name: checkout
5563
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
@@ -66,6 +74,63 @@ jobs:
6674
uses: dtolnay/rust-toolchain@stable
6775
- name: Build
6876
run: cargo build --workspace --all-targets --exclude llama-jni
77+
# `cargo run` and `cargo test` set LD_LIBRARY_PATH to the target directory,
78+
# so they cannot catch a missing rpath — only running the binary directly
79+
# exercises how a user (or a shipped artifact) actually loads the
80+
# llama.cpp shared libraries.
81+
- name: Verify a dynamically linked binary runs without cargo
82+
run: |
83+
output=$(./target/debug/simple --help 2>&1)
84+
echo "$output" | head -5
85+
if echo "$output" | grep -qiE 'error while loading shared libraries|cannot open shared object|no such file or directory'; then
86+
echo "::error::binary could not locate the llama.cpp shared libraries — is the \$ORIGIN rpath still applied?"
87+
exit 1
88+
fi
89+
- name: sccache stats
90+
run: sccache --show-stats || true
91+
feature-combos:
92+
# Every other job builds the default feature set
93+
# (openmp,mtmd,dynamic-link), so a break under any other combination stays
94+
# hidden until a release tag fires prebuilt-llama.yml. These two cover the
95+
# feature-gated bindgen sections (mtmd, rpc) and static linkage.
96+
name: Check non-default feature combos (${{ matrix.label }})
97+
runs-on: ubuntu-latest
98+
strategy:
99+
fail-fast: false
100+
matrix:
101+
include:
102+
- label: static + mtmd
103+
features: mtmd
104+
- label: rpc
105+
features: rpc
106+
env:
107+
RUSTC_WRAPPER: sccache
108+
SCCACHE_GHA_ENABLED: "true"
109+
# `RUSTFLAGS` replaces .cargo/config.toml rustflags rather than appending,
110+
# so the $ORIGIN rpath that lets directly executed binaries find the
111+
# llama.cpp shared libraries must be repeated here.
112+
RUSTFLAGS: '-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,$ORIGIN'
113+
steps:
114+
- name: checkout
115+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
116+
with:
117+
submodules: recursive
118+
- name: Install Compile Deps
119+
env:
120+
DEBIAN_FRONTEND: noninteractive
121+
run:
122+
sudo apt-get update && sudo apt-get install -y build-essential curl libssl-dev libclang-dev pkg-config cmake git mold
123+
- name: Setup sccache
124+
uses: mozilla-actions/sccache-action@v0.0.11
125+
- name: Setup Rust
126+
uses: dtolnay/rust-toolchain@stable
127+
- name: Build
128+
run: cargo build -p llama-cpp-4 --no-default-features --features ${{ matrix.features }}
129+
# examples/rpc is its own workspace (see its Cargo.toml), so no other job
130+
# ever compiles it — which is how it silently went stale.
131+
- name: Build rpc example
132+
if: matrix.features == 'rpc'
133+
run: cargo build --manifest-path examples/rpc/Cargo.toml
69134
- name: sccache stats
70135
run: sccache --show-stats || true
71136
mac:
@@ -129,7 +194,10 @@ jobs:
129194
env:
130195
RUSTC_WRAPPER: sccache
131196
SCCACHE_GHA_ENABLED: "true"
132-
RUSTFLAGS: -C link-arg=-fuse-ld=mold
197+
# `RUSTFLAGS` replaces .cargo/config.toml rustflags rather than appending,
198+
# so the $ORIGIN rpath that lets directly executed binaries find the
199+
# llama.cpp shared libraries must be repeated here.
200+
RUSTFLAGS: '-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,$ORIGIN'
133201
steps:
134202
- name: Checkout
135203
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1

‎CHANGELOG.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,155 @@
22

33
## Unreleased
44

5+
## [0.6.0] - 2026-08-17
6+
7+
### Changed
8+
9+
- **llama.cpp**: vendored submodule updated to `34af94cd9` (tag `b10470`, also
10+
tagged `v0.1.1` — upstream adopted [semantic versioning](https://github.com/ggml-org/llama.cpp/blob/master/docs/release.md)
11+
in this window and `v0.1.1` is its newest release) from `221f0f635` (b10235),
12+
pulling in 235 upstream commits — multi-output backend
13+
sampling ([#25532](https://github.com/ggml-org/llama.cpp/pull/25532)),
14+
speculative-type auto-detection from draft-GGUF metadata
15+
([#26814](https://github.com/ggml-org/llama.cpp/pull/26814),
16+
[#27005](https://github.com/ggml-org/llama.cpp/pull/27005)), backend sampling
17+
for dflash + dspark ([#26958](https://github.com/ggml-org/llama.cpp/pull/26958)),
18+
the mtmd audio-generation API, and new model architectures (BailingMoE3,
19+
MiniMax, Granite-Switch, Muse Glimmer, GLM-4.7-Flash MTP).
20+
- **BREAKING** — `LlamaSampler::penalties` / `LlamaSampler::penalties_simple` take
21+
a leading `n_vocab: i32` argument. Upstream moved `n_vocab` out of
22+
`llama_sampler_data` into the penalty sampler itself
23+
([#26520](https://github.com/ggml-org/llama.cpp/pull/26520)), so
24+
`llama_sampler_init_penalties` now needs it explicitly. Pass
25+
[`LlamaModel::n_vocab`], as `mirostat` already required. `penalty_last_n = -1`
26+
("context size") is gone upstream; only `0` disables the penalty.
27+
- **BREAKING** — `LlamaSampler::dry` no longer takes `n_ctx_train`: upstream
28+
dropped the parameter from `llama_sampler_init_dry`.
29+
- **BREAKING** — `common_sampler_params` (`llama-cpp-sys-4`) resynced with
30+
upstream `common_params_sampling`, which it hand-mirrors and had drifted far
31+
from. Removed `tfs_z` and `penalize_nl` (both deleted upstream); added
32+
`top_n_sigma`, `adaptive_target`, `adaptive_decay`, and `timing_per_token`;
33+
`dry_penalty_last_n` now defaults to `64` rather than `-1` (which is no longer
34+
a valid "context size" sentinel). The default `samplers` chain now matches
35+
upstream — it previously still listed the **removed TFS-Z sampler**.
36+
Correspondingly `COMMON_SAMPLER_TYPE_TFS_Z` is gone and
37+
`COMMON_SAMPLER_TYPE_PENALTIES`, `_TOP_N_SIGMA`, and `_ADAPTIVE_P` are added.
38+
- **BREAKING** — `LlamaLoadMode` gained an `Auto` variant (`LLAMA_LOAD_MODE_AUTO`,
39+
`-1`) and is now `#[repr(i32)]` on every target rather than `#[repr(u32)]`,
40+
because the enum is signed everywhere once it carries a negative discriminant.
41+
`Auto` is llama.cpp's new default: it memory-maps unless one of the backend
42+
devices lacks mmap support. `LlamaModelParams::use_mmap()` reports `true` for
43+
`Auto`, so the default-parameters behaviour is unchanged.
44+
- **Patch `0003` (exact speculative state) rebased**: upstream removed the
45+
`need_embd()` / `need_embd_nextn()` virtuals from `common_speculative_impl`
46+
([#26904](https://github.com/ggml-org/llama.cpp/pull/26904)), which those hunks
47+
used as context. Patches `0004` and `0005` apply unchanged.
48+
- `mtp_shim` no longer calls the deleted `common_speculative_need_embd` /
49+
`common_speculative_need_embd_nextn`. `MtpSession::need_embd` /
50+
`Eagle3Session::need_embd` still report `false` and `need_embd_pre_norm` still
51+
reports `true` only for MTP sessions, matching what upstream returned.
52+
53+
### Added
54+
55+
- **`LlamaContextParams::with_n_outputs_max_per_seq()` / `n_outputs_max_per_seq()`**
56+
(`llama-cpp-4`), wrapping the new `llama_context_params.n_outputs_max_per_seq`.
57+
Backend samplers are initialized for this many outputs per sequence, so
58+
multi-output backend sampling must raise it above its default of `1`.
59+
- **`llama_version()`** (`llama-cpp-4`): the vendored llama.cpp version string.
60+
Now that upstream ships semver releases this is the direct way to report which
61+
upstream a binary carries, since the crate and llama.cpp versions move
62+
independently.
63+
- **Tests for the APIs this release changes**, which previously had none:
64+
`LlamaLoadMode` (the `Auto` default, round-tripping every variant, and that the
65+
negative discriminant survives — the exact failure a `#[repr(u32)]` would
66+
reintroduce), `penalties` / `penalties_simple` construction and behaviour,
67+
`dry`, and `n_outputs_max_per_seq`.
68+
- **CI job `feature-combos`**: PRs previously only ever built the default
69+
`openmp,mtmd,dynamic-link` set, so a break under any other feature stayed
70+
hidden until a release tag fired `prebuilt-llama.yml`. The new job builds
71+
static `mtmd` (no `dynamic-link`) and `rpc` — the two feature-gated bindgen
72+
paths. The `mtmd` bindgen collision fixed in this release was caught only
73+
because `mtmd` happens to be on by default; adding this job immediately
74+
surfaced that `--features rpc` had not compiled for some time (see below).
75+
- `GGML_RPC_*` constants are now in the `rpc` bindgen allowlist, so
76+
`GGML_RPC_MAX_SERVERS` is available to validate device lists.
77+
78+
### Fixed
79+
80+
- **`mtmd` bindings no longer fail to compile**: upstream's new
81+
`namespace mtmd_helper` C++ RAII wrappers flatten to the same `mtmd_helper_*`
82+
names as the C API under bindgen without cxx-namespaces — `mtmd_helper::gen_audio`
83+
collided with the opaque C `struct mtmd_helper_gen_audio`. The `mtmd_helper::`
84+
namespace is now blocklisted; it was never usable from Rust.
85+
- **Dynamically linked binaries now run when executed directly.** With the
86+
default `dynamic-link` feature, CMake stamped every llama/ggml dylib with an
87+
`@rpath/…` install name, which rustc recorded in the final binary — but Cargo
88+
never adds an `LC_RPATH`, so running a built binary yourself died with
89+
`Library not loaded: @rpath/libggml-base.0.dylib … no LC_RPATH's found`. This
90+
affected **every** binary built from this crate, not just the examples; it was
91+
masked because `cargo run` and `cargo test` set
92+
`DYLD_FALLBACK_LIBRARY_PATH` / `LD_LIBRARY_PATH` to the target directory.
93+
The build script now rewrites the dylib install names and their sibling
94+
references to `@loader_path/…`, which needs no rpath and works for downstream
95+
consumers automatically. ELF cannot be fixed the same way — the rpath must sit
96+
on the final executable, which a dependency's build script cannot inject — so
97+
Linux/BSD get `-Wl,-rpath,$ORIGIN` via a new [`.cargo/config.toml`], with the
98+
requirement documented for downstream users in the README. Windows was already
99+
fine. Note `RUSTFLAGS` *replaces* config rustflags, so the CI jobs that set it
100+
now repeat the rpath. The rewrite is applied on the prebuilt-archive path too,
101+
not just the CMake build — prebuilt libraries carry the same `@rpath/…` names.
102+
CI now executes a built binary directly, since `cargo run` / `cargo test`
103+
inject the library path and therefore can never catch this class of bug.
104+
105+
[`.cargo/config.toml`]: .cargo/config.toml
106+
107+
- **CI `Fmt` and `Clippy` steps now actually gate.** `Fmt` ran bare `cargo fmt`,
108+
which rewrites files inside the runner and always exits `0`, so formatting was
109+
never enforced and the tree had accumulated drift; it is now
110+
`cargo fmt --all -- --check`. `Clippy` ran bare `cargo clippy`, which skipped
111+
tests, benches, and examples and treated findings as non-fatal; it is now
112+
`cargo clippy --workspace --all-targets -- -D warnings`. The workspace was
113+
formatted and its clippy findings resolved to make both steps pass.
114+
- **`examples/common.rs`** documented output regenerated from an actual run; it
115+
still advertised `tfs_z`, `penalize_nl`, and `dry_penalty_last_n: -1`.
116+
117+
- **The `rpc` feature compiles again.** `llama-cpp-4`'s `rpc` module had drifted
118+
out of sync with `ggml-rpc.h` and failed to build with five errors; nothing in
119+
CI ever compiled it, and `--features rpc` is not reachable from a default
120+
build, so it went unnoticed. Note `ggml-rpc.h` is **unchanged** in this
121+
llama.cpp bump — the breakage predates it. Upstream's RPC API is device-aware
122+
(one endpoint can expose several devices), so:
123+
- `RpcBackend::init` takes a `device: u32`, and `buffer_type` /
124+
`get_device_memory` use it. Added `RpcBackend::device()`, and `as_ptr()` is
125+
now public so the handle is actually usable.
126+
- `RpcServer` is replaced by a blocking `rpc::serve(endpoint, cache_dir,
127+
n_threads, devices)` matching `ggml_backend_rpc_start_server`. The old
128+
`RpcServer::start(backend, endpoint, free_mem, total_mem)` did not
129+
correspond to any current upstream signature, and returned a handle for a
130+
call that never returns.
131+
- `add_rpc_device` becomes `add_rpc_server`, returning a
132+
`ggml_backend_reg` — upstream renamed it and changed its return type from a
133+
device to a registration.
134+
- **`examples/rpc` builds again**, and is now the working consumer that proves
135+
the API above. It was previously listed in neither `workspace.members` nor
136+
`workspace.exclude`, so cargo rejected it standalone ("current package
137+
believes it's in a workspace when it's not") and the workspace never compiled
138+
it — which is how it and the `rpc` module drifted unnoticed. It now declares
139+
its own `[workspace]`: making it a root member would unify the `rpc` feature
140+
into every workspace build, forcing all contributors to compile llama.cpp with
141+
RPC support. It gained `--device`, `--cache-dir`, and `--threads` flags, and
142+
enumerates devices from ggml's backend registry. Verified by actually running
143+
it: the server reports `device 0: CPU` and blocks serving.
144+
145+
### Documented
146+
147+
- **`LlamaSampler::name()`** now explains llama.cpp's `?`-prefix convention:
148+
handed parameters that make it a no-op (`temp(1.0)`, `penalties` with
149+
`penalty_last_n = 0`, …), llama.cpp silently substitutes an identity sampler
150+
named `"?temp"` / `"?penalties"`. Construction still succeeds, so the name is
151+
the only signal the sampler does nothing. Long-standing upstream behaviour
152+
(unchanged in this bump), but previously undocumented here.
153+
5154
## [0.5.1] - 2026-08-03
6155

7156
### Added

0 commit comments

Comments
 (0)