feat: Add Rust implementation of Tokenizer

This commit adds the Rust implementation of the Tokenizer class. The Tokenizer struct is defined in the `node-napi/src/lib.rs` file. It includes methods for encoding and decoding text using the tokenizer. The implementation is based on the `RustTokenizer` struct from the `tokenizers` crate. This change enables the use of the Tokenizer in Rust-based projects.
flexpilot-ai · Oct 4, 2024 · 71b2ef8 · 71b2ef8
1 parent 557fde7
commit 71b2ef8
Show file tree

Hide file tree

Showing 36 changed files with 96,946 additions and 951 deletions.
diff --git a/.github/conda/bld.bat b/.github/conda/bld.bat
diff --git a/.github/conda/build.sh b/.github/conda/build.sh
diff --git a/.github/conda/meta.yaml b/.github/conda/meta.yaml
diff --git a/.github/stale.yml b/.github/stale.yml