Lightweight piece tokenization library
No known vulnerabilities in the latest version
Based on latest version 2.0.0. If you're running an older version, check OSV for your specific version.
[](/packages/curated-tokenizers)
<a href="/packages/curated-tokenizers"><img src="/api/badges/curated-tokenizers?period=month" alt="PyPI Stats"></a>