Source builds silently drop the NAX kernels when MACOSX_DEPLOYMENT_TARGET < 26.2 — no configure-time warning
envgap__ml-explore__mlx-3821
01 / FAILURE SIGNATURE
As reported upstream
No identifying execution failure has been captured.
Not a benchmark task.
- In a clean container the reported failure did not reproduce, or the known fix did not make the project run.
02 / ENVIRONMENT RECIPE
- Base commit
57c66cac7cb3e5b1eb350488a61f1506b40d39f8- Manifest
mlx/backend/metal/kernels/CMakeLists.txt- Reproduce
Awaiting issue-specific recipe- Run under trace
Awaiting a meaningful runtime command
03 / ORIGINAL ISSUE TEXT
ml-explore/mlx #3821 · read the original issue
Since #3622, `mlx/backend/metal/kernels/CMakeLists.txt` only builds the `*_nax` kernels when **`CMAKE_OSX_DEPLOYMENT_TARGET >= 26.2`** in addition to `MACOS_SDK_VERSION >= 26.2`; otherwise it defines `MLX_METAL_NO_NAX`, which makes `is_nax_available()` a hard `false`:
```cmake
if(MLX_METAL_VERSION GREATER_EQUAL 400
AND MACOS_SDK_VERSION VERSION_GREATER_EQUAL 26.2
AND CMAKE_OSX_DEPLOYMENT_TARGET VERSION_GREATER_EQUAL 26.2)
... build *_nax kernels ...
else()
target_compile_definitions(mlx PRIVATE MLX_METAL_NO_NAX)
endif()
```
The problem: a plain `pip install .` (or `uv pip install .`) on macOS inherits `MACOSX_DEPLOYMENT_TARGET` from CPython's build config (typically **11.0** for python.org/uv interpreters), so on M5-class hardware with a current SDK the resulting build **silently drops every NAX kernel**. There is no warning — the only trace is a `message(STATUS ...)` line swallowed by pip's captured output.
The perf cliff is large, and only on GEMM-shaped work, which makes it easy to misdiagnose as a kernel regression (we spent a morning bisecting exactly that):
| kernel (M5 Max, macOS 26.5, main @ e9463bb) | target ≥ 26.2 | target < 26.2 (NO_NAX) |
|---|---|---|
| bf16 GEMM 4096×4096×4096 | 2.5–3.8 ms | 9.2 ms (~2.7×) |
| affine 4-bit qmm, M=32768, 2880→11520 | 38 ms | 158 ms (~4×) |
| SDPA causal L=32k, 32h/8kv, d64 | 70 ms | 355 ms (~5×) |
| end-to-end 32k one-shot MoE prefill (gpt-oss-20b via mlx-lm) | 13.7 s | 45.7 s |
| M=1 decode-shape kernels (qmv etc.) | unchanged | unchanged |
Two aggravating details:
1. **`26.0` also trips it** (verified with `MACOSX_DEPLOYMENT_TARGET=26.0 pip install .`). On v0.31.2 the gate was SDK-only, so the same build recipe that produced a NAX-enabled 0.31.2 source build now produces a NAX-less build of main — the "regression" appears exactly when upgrading the source checkout, which strengthens the wrong conclusion.
2. Decode-shape ops are untouched, so casual `mlx_lm.generate` smoke tests look fine; only prefill/batch workloads show it.
### Suggestion
Any (or all) of:
- `message(WARNING "NAX kernels disabled: CMAKE_OSX_DEPLOYMENT_TARGET=${CMAKE_OSX_DEPLOYMENT_TARGET} < 26.2 (SDK ${MACOS_SDK_VERSION} supports them). Set MACOSX_DEPLOYMENT_TARGET=26.2 to enable.")` in the `else()` branch when the SDK/Metal-version conditions would otherwise qualify;
- defaulting the kernel deployment target to the host macOS version when the env var comes from the Python toolchain rather than the user (the top-level `CMakeLists.txt` already falls back to `MACOS_VERSION` when the variable is empty — it just never is under pip);
- a line in the build docs next to the source-install instructions.
Happy to send a PR for the warning if that's the preferred shape.
### Environment
- M5 Max, macOS 26.5, Xcode 26.6 (Metal toolchain component installed)
- mlx main @ e9463bb, Python 3.12.13
- verified both directions: `MACOSX_DEPLOYMENT_TARGET=26.0` → `MLX_METAL_NO_NAX` present in `flags.make`, GEMM 9.18 ms; `=26.2` → flag absent, GEMM 2.5 ms, same source, same machine
04 / LABELS
Labels from the report text only; not yet run
No supported category has been assigned.
Label rules and the text that matched
[]