← All tasks
cppml-explore/mlx #3821Not a task: not reproduced

Source builds silently drop the NAX kernels when MACOSX_DEPLOYMENT_TARGET < 26.2 — no configure-time warning

envgap__ml-explore__mlx-3821

01 / FAILURE SIGNATURE

As reported upstream

No identifying execution failure has been captured.
Not a benchmark task.
  • In a clean container the reported failure did not reproduce, or the known fix did not make the project run.

02 / ENVIRONMENT RECIPE

Base commit
57c66cac7cb3e5b1eb350488a61f1506b40d39f8
Manifest
mlx/backend/metal/kernels/CMakeLists.txt
Reproduce
Awaiting issue-specific recipe
Run under trace
Awaiting a meaningful runtime command

03 / ORIGINAL ISSUE TEXT

ml-explore/mlx #3821 · read the original issue
Since #3622, `mlx/backend/metal/kernels/CMakeLists.txt` only builds the `*_nax` kernels when **`CMAKE_OSX_DEPLOYMENT_TARGET >= 26.2`** in addition to `MACOS_SDK_VERSION >= 26.2`; otherwise it defines `MLX_METAL_NO_NAX`, which makes `is_nax_available()` a hard `false`:

```cmake
if(MLX_METAL_VERSION GREATER_EQUAL 400
   AND MACOS_SDK_VERSION VERSION_GREATER_EQUAL 26.2
   AND CMAKE_OSX_DEPLOYMENT_TARGET VERSION_GREATER_EQUAL 26.2)
  ... build *_nax kernels ...
else()
  target_compile_definitions(mlx PRIVATE MLX_METAL_NO_NAX)
endif()
```

The problem: a plain `pip install .` (or `uv pip install .`) on macOS inherits `MACOSX_DEPLOYMENT_TARGET` from CPython's build config (typically **11.0** for python.org/uv interpreters), so on M5-class hardware with a current SDK the resulting build **silently drops every NAX kernel**. There is no warning — the only trace is a `message(STATUS ...)` line swallowed by pip's captured output.

The perf cliff is large, and only on GEMM-shaped work, which makes it easy to misdiagnose as a kernel regression (we spent a morning bisecting exactly that):

| kernel (M5 Max, macOS 26.5, main @ e9463bb) | target ≥ 26.2 | target < 26.2 (NO_NAX) |
|---|---|---|
| bf16 GEMM 4096×4096×4096 | 2.5–3.8 ms | 9.2 ms (~2.7×) |
| affine 4-bit qmm, M=32768, 2880→11520 | 38 ms | 158 ms (~4×) |
| SDPA causal L=32k, 32h/8kv, d64 | 70 ms | 355 ms (~5×) |
| end-to-end 32k one-shot MoE prefill (gpt-oss-20b via mlx-lm) | 13.7 s | 45.7 s |
| M=1 decode-shape kernels (qmv etc.) | unchanged | unchanged |

Two aggravating details:

1. **`26.0` also trips it** (verified with `MACOSX_DEPLOYMENT_TARGET=26.0 pip install .`). On v0.31.2 the gate was SDK-only, so the same build recipe that produced a NAX-enabled 0.31.2 source build now produces a NAX-less build of main — the "regression" appears exactly when upgrading the source checkout, which strengthens the wrong conclusion.
2. Decode-shape ops are untouched, so casual `mlx_lm.generate` smoke tests look fine; only prefill/batch workloads show it.

### Suggestion

Any (or all) of:

- `message(WARNING "NAX kernels disabled: CMAKE_OSX_DEPLOYMENT_TARGET=${CMAKE_OSX_DEPLOYMENT_TARGET} < 26.2 (SDK ${MACOS_SDK_VERSION} supports them). Set MACOSX_DEPLOYMENT_TARGET=26.2 to enable.")` in the `else()` branch when the SDK/Metal-version conditions would otherwise qualify;
- defaulting the kernel deployment target to the host macOS version when the env var comes from the Python toolchain rather than the user (the top-level `CMakeLists.txt` already falls back to `MACOS_VERSION` when the variable is empty — it just never is under pip);
- a line in the build docs next to the source-install instructions.

Happy to send a PR for the warning if that's the preferred shape.

### Environment

- M5 Max, macOS 26.5, Xcode 26.6 (Metal toolchain component installed)
- mlx main @ e9463bb, Python 3.12.13
- verified both directions: `MACOSX_DEPLOYMENT_TARGET=26.0` → `MLX_METAL_NO_NAX` present in `flags.make`, GEMM 9.18 ms; `=26.2` → flag absent, GEMM 2.5 ms, same source, same machine
Continue on GitHub ↗

04 / LABELS

Labels from the report text only; not yet run

No supported category has been assigned.

Label rules and the text that matched
[]