Data and AI struggles

Trouble with Hugging Face tokenizer

MLOps2 posts from 2 people2d active+00 posts in the last 7 days, 0 the 7 days before (fading)Unverified

Posts per day

Posts per day2 posts, Sep 9 to Oct 21

The posts behind it

2, newest first
GitHub issuesOct 6 2 weeks agoLoading a tokenizer imports torch and scikit-learn when they are installed (about 2 s of a 3 s import)### System Info transformers 5.19.0 (also 5.16.1), torch 2.14.1, scikit-learn 1.9.1, tokenizers 0.23, Python 3.12, Linux x86_64. ### Reproduction With torch and scikit-learn installed, the import takes 2.5-3.0 s with a warm page cache and imports both. With them uninstalled it...huggingface/transformers · AshishKumar44 repliesOpen ↗GitHub issuesSep 18 4 weeks agoT5 sentencepiece tokenizer without precompiled_charsmap cannot be loaded### System Info - `transformers` version: 5.18.0.dev0 - Platform: Linux-7.1.10-arch1-1-x86_64-with-glibc2.43 - Python version: 3.14.4 - Huggingface_hub version: 1.31.0 - Safetensors version: 0.8.0 - Accelerate version: 1.15.0 - Accelerate config: not found - DeepSpeed version:...huggingface/transformers · DRKV3336 repliesOpen ↗

Companies and products named

Named2 postsHugging FaceNamed1 postPyTorchNamed1 postNVIDIANamed1 postGitHub
About this problem

Evidence

2 posts from 2 people in 1 place, about 1 a week over 19 days. Mostly on huggingface/transformers. Tools named alongside: PyTorch, NVIDIA, GitHub.

Frustration Frustration 0 of 3· Seen on GitHub issues

How it was grouped

Posts about Hugging Face with no known issue, sharing the word "tokenizer". First post Sep 18, 2026, latest Oct 6, 2026. Unverified: fewer than 3 posts, or one person or place only. Method

History

CorroborationOct 20, 2026corroborated to unverified
MomentumOct 13, 2026steady to fading
AddedOct 11, 2026Trouble with Hugging Face tokenizer (3 posts)