[BUG] dd-lib-python-init:v4 amd64 image contains ARM64 psutil binaries causing ImportError on x86_64 nodes
envgap__DataDog__dd-trace-py-15906
01 / FAILURE SIGNATURE
As reported upstream
ImportError: /opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.12-manylinux2014/ddtrace/vendor/psutil/_psutil_linux.abi3.so: cannot open shared object file: No such file or directory
Not a benchmark task.
- In a clean container the reported failure did not reproduce, or the known fix did not make the project run.
02 / ENVIRONMENT RECIPE
- Base commit
ab1a48689346cfc69fbf483a1f2345148d28a95f- Manifest
ddtrace/vendor/psutil/setup.py- Reproduce
Awaiting issue-specific recipe- Run under trace
Awaiting a meaningful runtime command
03 / ORIGINAL ISSUE TEXT
DataDog/dd-trace-py #15906 · read the original issue
## Bug Description
The `dd-lib-python-init:v4` amd64 image contains **ARM64 binaries** for psutil's native extensions instead of x86-64 binaries. This causes the `PSUtilRuntimeMetricCollector` to fail when the init container copies libraries to x86_64 Kubernetes nodes.
## Environment
- **Image**: `eu.gcr.io/datadoghq/dd-lib-python-init:v4`
- **Affected digest (amd64)**: `sha256:697b7522e3c1a2cfe1adeda17181535e5c1149589940c1f174c03a0482a556d2`
- **Kubernetes nodes**: amd64/x86_64 architecture
- **Python version**: 3.12 (affects all Python versions)
- **Deployment**: Kubernetes with Datadog Admission Controller auto-instrumentation
## Error Message
```
ImportError: /opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.12-manylinux2014/ddtrace/vendor/psutil/_psutil_linux.abi3.so: cannot open shared object file: No such file or directory
Could not import module "ddtrace.vendor.psutil" for <PSUtilRuntimeMetricCollector(enabled=False,periodic=True,required_modules=['ddtrace.vendor.psutil'])>. Disabling collector.
```
## Root Cause
The amd64 variant of `dd-lib-python-init:v4` was built with ARM64 binaries for psutil:
### Evidence
**1. Container runs on x86_64 node:**
```bash
$ uname -m
x86_64
```
**2. Binary architecture check of psutil .so file copied by init container:**
```python
import struct
with open('/opt/datadog/.../ddtrace/vendor/psutil/_psutil_linux.abi3.so', 'rb') as f:
f.read(18)
e_machine = struct.unpack('<H', f.read(2))[0]
print(f'Architecture: {e_machine}') # 183 = ARM64 (aarch64)
# Output: Architecture: 183 (ARM64/aarch64, should be 62 for x86-64)
```
**3. Verification with Docker:**
AMD64 image contains ARM64 psutil binaries:
```bash
$ docker run --rm --platform linux/amd64 \
--entrypoint sh \
eu.gcr.io/datadoghq/dd-lib-python-init@sha256:697b7522e3c1a2cfe1adeda17181535e5c1149589940c1f174c03a0482a556d2 \
-c "cat /datadog-init/package/ddtrace_pkgs/site-packages-ddtrace-py3.12-manylinux2014/ddtrace/vendor/psutil/_psutil_linux.abi3.so" \
| python3 -c "import struct, sys; f=sys.stdin.buffer; f.read(18); print({62:'x86-64',183:'ARM64'}[struct.unpack('<H',f.read(2))[0]])"
# Output: ARM64 ❌ (should be x86-64 for amd64 image)
```
Other binaries in the same image are correctly x86-64:
```bash
# _stacktrace.cpython-312-x86_64-linux-gnu.so: x86-64 ✓
```
This suggests the issue is specific to psutil's `.abi3.so` files, likely due to the build process mixing architectures when creating the megawheel.
## Impact
- ✅ **Application still works**: ddtrace catches the exception and disables the collector
- ❌ **Lost functionality**: `PSUtilRuntimeMetricCollector` is disabled, losing CPU/memory runtime metrics
- ⚠️ **Confusion**: Error appears in every pod startup log, misleading debugging efforts
## Why Node.js doesn't have this problem
The `dd-lib-js-init` container includes **all architectures** and Node.js selects the correct one at runtime:
```
/opt/datadog/.../prebuilds/
├── linuxglibc-arm64/*.node
└── linuxglibc-x64/*.node ← Selected at runtime
```
Python's init container only includes one architecture per platform-specific directory, so architecture mismatches cause hard failures.
## Related Issues
This appears similar to #5565 where the `dl_megawheel.py` script was mixing files from different wheels. That was fixed for `bytecode`, but the issue seems to have reoccurred with `psutil` in v4.
## Expected Behavior
The amd64 variant of `dd-lib-python-init:v4` should contain x86-64 binaries:
- `e_machine` value should be `62` (x86-64), not `183` (ARM64)
- psutil should load successfully on x86_64 nodes
- `PSUtilRuntimeMetricCollector` should be enabled
## Additional Context
- ARM64 image digest (`sha256:8fc441331a...`) correctly contains ARM64 binaries ✓
- Multi-arch manifest exists and is correctly tagged
- The issue is in the **build process** of the amd64 image, not manifest selection
## Suggested Fix
Audit the build process for `dd-lib-python-init:v4` to ensure architecture-specific wheels are extracted to architecture-specific directories without cross-contamination, similar to the fix applied in #5576.
---
**Priority**: Medium - This is a packaging defect affecting all amd64 users of v4, causing loss of runtime metrics. While not service-breaking (app continues to work), it degrades observability for a significant user base and is a regression of a previously fixed issue.04 / LABELS
Labels from the report text only; not yet run
No supported category has been assigned.
Label rules and the text that matched
[]