← All tasks
pythonwandb/wandb #10396Not a task: not reproduced

[Bug]: TypeError: Unknown color '178'

envgap__wandb__wandb-10396

01 / FAILURE SIGNATURE

As reported upstream

KeyError: 178
Not a benchmark task.
  • In a clean container the reported failure did not reproduce, or the known fix did not make the project run.

02 / ENVIRONMENT RECIPE

Base commit
d55d2321daed43318e7e58ff22d73b54ff741917
Manifest
pyproject.toml
Reproduce
Awaiting issue-specific recipe
Run under trace
Awaiting a meaningful runtime command

03 / ORIGINAL ISSUE TEXT

wandb/wandb #10396 · read the original issue
### Describe the bug

I am using the `wandb` integration in `transformers.Trainer` to log my training `report_to=["wandb"]`. Sometimes, but not always, I get this error:

```
Traceback (most recent call last):
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/click/termui.py", line 498, in style
    bits.append("\033[{}m".format(_ansi_colors[fg]))
                                  ~~~~~~~~~~~~^^^^
KeyError: 178

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/vol1/bram/CommonCrawl-CreativeCommons/scripts/post_processing/train_topic_classifier.py", line 587, in <module>
    main()
  File "/vol1/bram/CommonCrawl-CreativeCommons/scripts/post_processing/train_topic_classifier.py", line 527, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 2238, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 2488, in _inner_training_loop
    self.control = self.callback_handler.on_train_begin(args, self.state, self.control)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer_callback.py", line 506, in on_train_begin
    return self.call_event("on_train_begin", args, state, control)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/trainer_callback.py", line 556, in call_event
    result = getattr(callback, event)(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 965, in on_train_begin
    self.setup(args, state, model, **kwargs)
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/transformers/integrations/integration_utils.py", line 892, in setup
    self._wandb.init(
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 1595, in init
    wandb._sentry.reraise(e)
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/analytics/sentry.py", line 162, in reraise
    raise exc.with_traceback(sys.exc_info()[2])
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 1581, in init
    return wi.init(run_settings, run_config, run_printer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 996, in init
    result = wait_with_progress(
             ^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 24, in wait_with_progress
    return wait_all_with_progress(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 87, in wait_all_with_progress
    return asyncio_compat.run(progress_loop_with_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 30, in run
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 74, in run
    return asyncio.run(self._run_or_cancel(fn))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 98, in _run_or_cancel
    return fn_task.result()
           ^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 75, in progress_loop_with_timeout
    with asyncio_compat.cancel_on_exit(display_progress()):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 211, in cancel_on_exit
    raise exception
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_init.py", line 990, in display_init_message
    await progress.loop_printing_operation_stats(
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 56, in loop_printing_operation_stats
    async with asyncio_compat.open_task_group() as task_group:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bram/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 217, in __aexit__
    await anext(self.gen)
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 193, in open_task_group
    await task_group._wait_all()
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 162, in _wait_all
    raise exc
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 40, in loop_update_screen
    progress.update(stats)
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 106, in update
    self._update_operation_stats([progress])
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/progress.py", line 124, in _update_operation_stats
    loading_symbol=self._printer.loading_symbol(self._tick),
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/wandb/sdk/lib/printer.py", line 406, in loading_symbol
    return click.style(
           ^^^^^^^^^^^^
  File "/vol1/bram/CommonCrawl-CreativeCommons/.venv/lib/python3.12/site-packages/click/termui.py", line 500, in style
    raise TypeError("Unknown color '{}'".format(fg))
TypeError: Unknown color '178'
```

As far as I can tell wandb is requesting the color inside `click` which leads to issues.

- python 3.12
- transformers==4.55.4
- wandb==0.21.1
- click==7.1.2

---

Note: although your pyproject.toml does not allo click>=8, upgrading from click 7 to 8 does seem to solve the issue. Not sure if this is a transformers+wandb+click compatibility issue?
Continue on GitHub ↗

04 / LABELS

Labels from the report text only; not yet run

No supported category has been assigned.

Label rules and the text that matched
[]