← All tasks
cppggml-org/llama.cpp #29138Not a task: already works

Misc. bug: ggml-rpc-server crashes on vulkan when a duplicate copy of libggml-vulkan is dlopened

envgap__ggml-org__llama.cpp-29138

01 / FAILURE SIGNATURE

As reported upstream

No identifying execution failure has been captured.
Not a benchmark task.
  • The project already builds and runs before the fix, so there is nothing to repair.

02 / ENVIRONMENT RECIPE

Base commit
fb34fc262c1b43f1832c7472429fb2247d650493
Manifest
ggml/src/ggml-vulkan/CMakeLists.txt
Reproduce
Awaiting issue-specific recipe
Run under trace
Awaiting a meaningful runtime command

03 / ORIGINAL ISSUE TEXT

ggml-org/llama.cpp #29138 · read the original issue
### Name and Version

$ llama-server --version
version: 0.4.1-dev (build 11051, commit 5b59b83f4)
built with GNU 14.2.0 for Linux x86_64

### Operating systems

Linux

### Which llama.cpp modules do you know to be affected?

Other (Please specify in the next section)

### Command line

```shell
ggml-rpc-server --host 0.0.0.0 --port 50052 -c
```

### Problem description & steps to reproduce

Starting the RPC server results in an immediate crash.

Since f172be756 ("vulkan: split buffers and debug code into separate files, add shared headers", #28732) the Vulkan backend's internal cross-TU symbols (`vk_instance`, `ggml_vk_init`, `ggml_vk_instance_init`, ...) are exported from  `libggml-vulkan.so` with default visibility (they used to be `static`).        

This interacts badly with `ggml_backend_load_all()`:
1/ `ggml_backend_load_best("vulkan", ...)` scans the executable directory and dlopens `libggml-vulkan.so` found there.
2. In a non-GGML_BACKEND_DL build that library has no `ggml_backend_score` / `ggml_backend_init` symbols, so `load_backend()` returns and the handle is immediately **dlclosed**. This has always happened and used to be harmless.
3. If `libggml-vulkan.so` in the executable directory is a *regular file copy* rather than a symlink to the already-loaded `libggml-vulkan.so.0` (different inode — e.g. after installing with `cp`/`scp`, which dereference symlinks), glibc loads it as a **second copy** of the library.                         
4. The duplicate's dynamic relocations bind to the first copy's exported  globals (normal ELF interposition). At dlclose, the duplicate's static destructors therefore run against the **live** library's objects: `vk_instance_t::~vk_instance_t()` destroys the real `vk_instance`, leaving `device_indices` dangling.                                                                       
 
Any subsequent Vulkan device use then fails on an "impossible" assert; the exact site varies per run because the vector is dangling, e.g.:      

```
ggml/src/ggml-vulkan/ggml-vulkan.cpp:14735: GGML_ASSERT(device < (int) vk_instance.device_indices.size()) failed                                                                         
ggml/src/ggml-vulkan/ggml-vulkan.cpp:5173:  GGML_ASSERT(idx < vk_instance.device_indices.size()) failed                                                                                                
```

even though `ggml_vulkan: Found 1 Vulkan devices` was printed at startup.

### First Bad Commit

f172be756

### Relevant log output

<details>
<summary>Logs</summary>
<!-- Copy-pasted short logs go into the "console" area here -->

```console
Sep 19 12:13:14 rood ggml-rpc-server[514539]: ggml_vulkan: Found 1 Vulkan devices:
Sep 19 12:13:14 rood ggml-rpc-server[514539]: ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Sep 19 12:13:14 rood ggml-rpc-server[514539]: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: WARNING: Host ('0.0.0.0') is != '127.0.0.1'
Sep 19 12:13:14 rood ggml-rpc-server[514539]: Never expose the RPC server to an open network!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: This is an experimental feature and is not secure!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Sep 19 12:13:14 rood ggml-rpc-server[514539]: Starting RPC server v7.0.0
Sep 19 12:13:14 rood ggml-rpc-server[514539]: endpoint : 0.0.0.0:50052
Sep 19 12:13:14 rood ggml-rpc-server[514539]: local cache : /var/lib/llama-rpc/cache/rpc/
Sep 19 12:13:14 rood ggml-rpc-server[514539]: Devices:
Sep 19 12:13:14 rood ggml-rpc-server[514539]: /home/erik/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:5173: GGML_ASSERT(idx < vk_instance.device_indices.size()) failed
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-base.so.0(+0x187e5) [0x7fe0bbfad7e5]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-base.so.0(ggml_print_backtrace+0x1df) [0x7fe0bbfadbbf]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-base.so.0(ggml_abort+0x11e) [0x7fe0bbfadd4e]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-vulkan.so.0(_Z12ggml_vk_initP23ggml_backend_vk_contextm+0x80b) [0x7fe0b8f6a18b]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-vulkan.so.0(ggml_backend_vk_init+0x1f9) [0x7fe0b8f6a399]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/libggml-rpc.so.0(ggml_backend_rpc_start_server+0x181) [0x7fe0bbf53b31]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/ggml-rpc-server(+0x75b8) [0x5643c66475b8]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7fe0bba35ca8]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fe0bba35d65]
Sep 19 12:13:14 rood ggml-rpc-server[514540]: /usr/local/bin/ggml-rpc-server(+0x7df1) [0x5643c6647df1]
```
</details>

<!-- Long logs that you upload as files go here, outside the "console" area -->
Continue on GitHub ↗

04 / LABELS

Labels from the report text only; not yet run

No supported category has been assigned.

Label rules and the text that matched
[]