llm-serving

Here are 3 public repositories matching this topic...

AICI: Prompts as (Wasm) Programs

rust ai wasm inference transformer language-model model-serving wasmtime llm llmops llm-serving llm-inference llm-framework

A Lazy, high throughput and blazing fast structured text generation backend.

performance ai transformer llama model-serving llm llmops llm-serving

Lightweight and extensible LLM Inference serving benchmark tool written in Rust.

benchmarking llm-serving llm-inference

Add a description, image, and links to the llm-serving topic page so that developers can more easily learn about it.

To associate your repository with the llm-serving topic, visit your repo's landing page and select "manage topics."