Parser Building Toolkit for JavaScript

Online playground for OpenAPI tokenizers

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

VSCode extension to highlight nested code blocks

Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.

Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.

Tokenize CSS according to the CSS Syntax

Promise based streaming tokenizer

An extension for the Tiptap editor, enabling the annotation of text. Comes with support for overlapping annotations, useful for e.g. NLP tokenization.

Web tool to count LLM tokens (GPT, Claude, Llama, ...)

GPT4 Tokenizer Visualizer

📐 GPT token estimation and context size utilities without a full tokenizer

🐉 JavaScript expression tokenizer and evaluator

Language APIs and support features from Textmate tokenization in Visual Studio Code.

⚙️ A simple tokenizer for deno

Play with OpenAI API's using your own API Key. Your API Key is stored and used only from your browser.

sentencepiece port to webassembly with browser compatibility

A primitive token library used to read from, and to write a node Buffer

A simple lexer written entirely in JavaScript, with no dependencies!

Lexer / tokenizer