TensorFold

TensorFold is an LLM inference engine based on the Zig language, supporting running on Apple Silicon and NVIDIA GPU. It can directly read checkpoints, tokenize requests, and run Metal or CUDA kernels, thereby achieving efficient model serving.

The installation and use of TensorFold are very simple, supporting installation via Homebrew on Mac, or deployment via NVIDIA GPU on Linux. It also provides various command-line options, such as pulling models, starting services, and sending requests, to meet the needs of different users.