Plain-AyI
← All projects
Developer building blocks

Tokenizers

A fast code library that splits text into the small pieces, called tokens, that AI language models read.

Widely usedNew this week

What it does

AI language models don't read words directly. They read tokens, small chunks of text turned into numbers. Tokenizers is Hugging Face's library for doing this conversion quickly, and turning numbers back into text.

It can load the tokenizers used by many popular models. It is written in Rust for speed and can be used from Python and Node.js.

The README says the project is moving to a new major release, and some older features are still being brought back.

Who it's for

Programmers and researchers building or running AI language models.

Can I use it without coding?

Developers only. It is a code library for programmers. New to this? Start here.

Can I run it?

Cost
FreeNothing to pay or sign up for.
Computer
An ordinary computer is fine
Privacy
Runs on your own device: your data stays with you
API key
Not needed

Try it

Using Claude Code, Cursor or another AI coding assistant? Paste the prompt and it will walk you through the setup.New to this?

Before you install

  • Only download from the official GitHub page. Copies with similar names exist for many popular projects.
  • License (Apache-2.0): Permissive: free to use, change and share, including for business.
  • More on staying safe

Where people are talking about it