Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
nlp machine-learning natural-language-processing ai tokens nlp-machine-learning bert tokenization wordpiece bert-embeddings wordpiece-tokenization llm
-
Updated
Mar 3, 2025 - C#