← All tasks
pythonDataDog/dd-trace-py #15906Not a task: not reproduced

[BUG] dd-lib-python-init:v4 amd64 image contains ARM64 psutil binaries causing ImportError on x86_64 nodes

envgap__DataDog__dd-trace-py-15906

01 / FAILURE SIGNATURE

As reported upstream

ImportError: /opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.12-manylinux2014/ddtrace/vendor/psutil/_psutil_linux.abi3.so: cannot open shared object file: No such file or directory
Not a benchmark task.
  • In a clean container the reported failure did not reproduce, or the known fix did not make the project run.

02 / ENVIRONMENT RECIPE

Base commit
ab1a48689346cfc69fbf483a1f2345148d28a95f
Manifest
ddtrace/vendor/psutil/setup.py
Reproduce
Awaiting issue-specific recipe
Run under trace
Awaiting a meaningful runtime command

03 / ORIGINAL ISSUE TEXT

DataDog/dd-trace-py #15906 · read the original issue
## Bug Description

The `dd-lib-python-init:v4` amd64 image contains **ARM64 binaries** for psutil's native extensions instead of x86-64 binaries. This causes the `PSUtilRuntimeMetricCollector` to fail when the init container copies libraries to x86_64 Kubernetes nodes.

## Environment

- **Image**: `eu.gcr.io/datadoghq/dd-lib-python-init:v4`
- **Affected digest (amd64)**: `sha256:697b7522e3c1a2cfe1adeda17181535e5c1149589940c1f174c03a0482a556d2`
- **Kubernetes nodes**: amd64/x86_64 architecture
- **Python version**: 3.12 (affects all Python versions)
- **Deployment**: Kubernetes with Datadog Admission Controller auto-instrumentation

## Error Message

```
ImportError: /opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.12-manylinux2014/ddtrace/vendor/psutil/_psutil_linux.abi3.so: cannot open shared object file: No such file or directory
Could not import module "ddtrace.vendor.psutil" for <PSUtilRuntimeMetricCollector(enabled=False,periodic=True,required_modules=['ddtrace.vendor.psutil'])>. Disabling collector.
```

## Root Cause

The amd64 variant of `dd-lib-python-init:v4` was built with ARM64 binaries for psutil:

### Evidence

**1. Container runs on x86_64 node:**
```bash
$ uname -m
x86_64
```

**2. Binary architecture check of psutil .so file copied by init container:**
```python
import struct
with open('/opt/datadog/.../ddtrace/vendor/psutil/_psutil_linux.abi3.so', 'rb') as f:
    f.read(18)
    e_machine = struct.unpack('<H', f.read(2))[0]
    print(f'Architecture: {e_machine}')  # 183 = ARM64 (aarch64)

# Output: Architecture: 183  (ARM64/aarch64, should be 62 for x86-64)
```

**3. Verification with Docker:**

AMD64 image contains ARM64 psutil binaries:
```bash
$ docker run --rm --platform linux/amd64 \
  --entrypoint sh \
  eu.gcr.io/datadoghq/dd-lib-python-init@sha256:697b7522e3c1a2cfe1adeda17181535e5c1149589940c1f174c03a0482a556d2 \
  -c "cat /datadog-init/package/ddtrace_pkgs/site-packages-ddtrace-py3.12-manylinux2014/ddtrace/vendor/psutil/_psutil_linux.abi3.so" \
  | python3 -c "import struct, sys; f=sys.stdin.buffer; f.read(18); print({62:'x86-64',183:'ARM64'}[struct.unpack('<H',f.read(2))[0]])"

# Output: ARM64  ❌ (should be x86-64 for amd64 image)
```

Other binaries in the same image are correctly x86-64:
```bash
# _stacktrace.cpython-312-x86_64-linux-gnu.so: x86-64 ✓
```

This suggests the issue is specific to psutil's `.abi3.so` files, likely due to the build process mixing architectures when creating the megawheel.

## Impact

- ✅ **Application still works**: ddtrace catches the exception and disables the collector
- ❌ **Lost functionality**: `PSUtilRuntimeMetricCollector` is disabled, losing CPU/memory runtime metrics
- ⚠️ **Confusion**: Error appears in every pod startup log, misleading debugging efforts

## Why Node.js doesn't have this problem

The `dd-lib-js-init` container includes **all architectures** and Node.js selects the correct one at runtime:
```
/opt/datadog/.../prebuilds/
  ├── linuxglibc-arm64/*.node
  └── linuxglibc-x64/*.node     ← Selected at runtime
```

Python's init container only includes one architecture per platform-specific directory, so architecture mismatches cause hard failures.

## Related Issues

This appears similar to #5565 where the `dl_megawheel.py` script was mixing files from different wheels. That was fixed for `bytecode`, but the issue seems to have reoccurred with `psutil` in v4.

## Expected Behavior

The amd64 variant of `dd-lib-python-init:v4` should contain x86-64 binaries:
- `e_machine` value should be `62` (x86-64), not `183` (ARM64)
- psutil should load successfully on x86_64 nodes
- `PSUtilRuntimeMetricCollector` should be enabled

## Additional Context

- ARM64 image digest (`sha256:8fc441331a...`) correctly contains ARM64 binaries ✓
- Multi-arch manifest exists and is correctly tagged
- The issue is in the **build process** of the amd64 image, not manifest selection

## Suggested Fix

Audit the build process for `dd-lib-python-init:v4` to ensure architecture-specific wheels are extracted to architecture-specific directories without cross-contamination, similar to the fix applied in #5576.

---

**Priority**: Medium - This is a packaging defect affecting all amd64 users of v4, causing loss of runtime metrics. While not service-breaking (app continues to work), it degrades observability for a significant user base and is a regression of a previously fixed issue.
Continue on GitHub ↗

04 / LABELS

Labels from the report text only; not yet run

No supported category has been assigned.

Label rules and the text that matched
[]