[Bug]: TypeError: Unknown color '178'
envgap__wandb__wandb-10396
01 / FAILURE SIGNATURE
As reported upstream
KeyError: 178
Not a benchmark task.
- In a clean container the reported failure did not reproduce, or the known fix did not make the project run.
02 / ENVIRONMENT RECIPE
- Base commit
d55d2321daed43318e7e58ff22d73b54ff741917- Manifest
pyproject.toml- Reproduce
Awaiting issue-specific recipe- Run under trace
Awaiting a meaningful runtime command
03 / ORIGINAL ISSUE TEXT
wandb/wandb #10396 · read the original issue
### Describe the bug
I am using the `wandb` integration in `transformers.Trainer` to log my training `report_to=["wandb"]`. Sometimes, but not always, I get this error:
```
Traceback (most recent call last):
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/click/termui.py", line 498, in style
bits.append("\033[{}m".format(_ansi_colors[fg]))
~~~~~~~~~~~~^^^^
KeyError: 178
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/vol1/bram/CommonCrawl-CreativeCommons/scripts/post_processing/train_topic_classifier.py", line 587, in <module>
main()
File "/vol1/bram/CommonCrawl-CreativeCommons/scripts/post_processing/train_topic_classifier.py", line 527, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 2238, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 2488, in _inner_training_loop
self.control = self.callback_handler.on_train_begin(args, self.state, self.control)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer_callback.py", line 506, in on_train_begin
return self.call_event("on_train_begin", args, state, control)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer_callback.py", line 556, in call_event
result = getattr(callback, event)(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 965, in on_train_begin
self.setup(args, state, model, **kwargs)
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 892, in setup
self._wandb.init(
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 1595, in init
wandb._sentry.reraise(e)
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/analytics/sentry.py", line 162, in reraise
raise exc.with_traceback(sys.exc_info()[2])
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 1581, in init
return wi.init(run_settings, run_config, run_printer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 996, in init
result = wait_with_progress(
^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 24, in wait_with_progress
return wait_all_with_progress(
^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 87, in wait_all_with_progress
return asyncio_compat.run(progress_loop_with_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 30, in run
return future.result()
^^^^^^^^^^^^^^^
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 74, in run
return asyncio.run(self._run_or_cancel(fn))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 98, in _run_or_cancel
return fn_task.result()
^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 75, in progress_loop_with_timeout
with asyncio_compat.cancel_on_exit(display_progress()):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 211, in cancel_on_exit
raise exception
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 990, in display_init_message
await progress.loop_printing_operation_stats(
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 56, in loop_printing_operation_stats
async with asyncio_compat.open_task_group() as task_group:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 217, in __aexit__
await anext(self.gen)
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 193, in open_task_group
await task_group._wait_all()
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 162, in _wait_all
raise exc
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 40, in loop_update_screen
progress.update(stats)
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 106, in update
self._update_operation_stats([progress])
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 124, in _update_operation_stats
loading_symbol=self._printer.loading_symbol(self._tick),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/printer.py", line 406, in loading_symbol
return click.style(
^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/click/termui.py", line 500, in style
raise TypeError("Unknown color '{}'".format(fg))
TypeError: Unknown color '178'
```
As far as I can tell wandb is requesting the color inside `click` which leads to issues.
- python 3.12
- transformers==4.55.4
- wandb==0.21.1
- click==7.1.2
---
Note: although your pyproject.toml does not allo click>=8, upgrading from click 7 to 8 does seem to solve the issue. Not sure if this is a transformers+wandb+click compatibility issue?04 / LABELS
Labels from the report text only; not yet run
No supported category has been assigned.
Label rules and the text that matched
[]