Tokenizers

NLP

Fast tokenization library.

Installation
pip install tokenizers
Documentation
Official Docs

Chapters Using Tokenizers

Chapter 14: Pre-trained Language Models (Modern Architectures and Large Language Models)