Source and editable installs fail in some locales
envgap__gitpython-developers__GitPython-1747
01 / FAILURE SIGNATURE
As reported upstream
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 473: illegal multibyte sequence
Not a benchmark task.
- The project already builds and runs before the fix, so there is nothing to repair.
02 / ENVIRONMENT RECIPE
- Base commit
5c6a4f4068739a85873f3eb14f18292d5c323fd8- Manifest
setup.py- Reproduce
Awaiting issue-specific recipe- Run under trace
Awaiting a meaningful runtime command
03 / ORIGINAL ISSUE TEXT
gitpython-developers/GitPython #1747 · read the original issue
Methods of installing GitPython that run code from `setup.py` fail in some locales. This does not affect installing from a wheel, but it does affect installing from an sdist, or installing from a local directory, including the editable install procedure recommended for development in the readme. The same problem happens when building GitPython. Building or installing using the old method of running `setup.py` directly is also affected. The error is of this form, though the codec will not always be `gbk`:
```text
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 473: illegal multibyte sequence
```
I am unsure if this ever happens in practice on Unix-like systems, whose locales are usually UTF-8. However, it happens on Windows systems in which `README.md` cannot be decoded using the system's active ANSI code page. This is a rarely-changed systemwide setting, so changing the user preferred languages, display language, or input method are not workarounds. I discovered this on a Simplified Chinese (zh-CN) build of Windows Server 2022 while using it to test some WSL-related test helper logic in #1745. Such a system uses ANSI code page 936. `README.md` is UTF-8, but it currently happens that it can be decoded with code page 1252, which Windows builds for Western European languages use as their ANSI code page. I expect encodings other than cp936 to fail as well.
With the PyPI sdist for GitPython 3.1.40 (on Python 3.12 x86-64, though I expect all supported versions to be affected):
```text
(.venv) C:\Users\Administrator\gptest> pip install --no-binary GitPython GitPython
Collecting GitPython
Using cached GitPython-3.1.40.tar.gz (200 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
Traceback (most recent call last):
File "C:\Users\Administrator\gptest\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
main()
File "C:\Users\Administrator\gptest\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\gptest\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\AppData\Local\Temp\pip-build-env-8j1uxmuv\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\AppData\Local\Temp\pip-build-env-8j1uxmuv\overlay\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "C:\Users\Administrator\AppData\Local\Temp\pip-build-env-8j1uxmuv\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 20, in <module>
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 473: illegal multibyte sequence
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
```
A workaround is to pass `-X utf8` to `python`. For example, `python -X utf8 -m pip install ...` can be used for installation.
The fix should be straightforward. Importing `setup.py` confirms that the specific cause is reading `README.md`:
```text
(.venv) C:\Users\Administrator\repos\GitPython [main ≡]> python -c 'import setup'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Administrator\repos\GitPython\setup.py", line 20, in <module>
long_description = rm_file.read()
^^^^^^^^^^^^^^
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa6 in position 477: illegal multibyte sequence
```
It can be fixed by passing `encoding="utf-8"`. **I have proposed this change in #1748.**
gitdb and smmap are unaffected, because [gitdb does not open files to read those data in `setup.py`](https://github.com/gitpython-developers/gitdb/blob/ec58b7e55c8fec2479290a22faa293799f6805fc/setup.py#L24), while [smmap does open `README.md` but passes `encoding="utf-8"`](https://github.com/gitpython-developers/smmap/blob/f1ace75be355fdec927793e462b9b12bf6ec9520/setup.py#L14).04 / LABELS
Labels from the report text only; not yet run
No supported category has been assigned.
Label rules and the text that matched
[]