开发者发布了一个名为TurboFieldfare的开源推理引擎，用Swift和Metal编写，能够在任何M系Mac上运行4-bit量化的Gemma 4 26B模型，内存占用仅约2GB1。该引擎通过在RAM中保持模型共享部分和KV缓存，同时从SSD流式传输路由专家的方式，有效解决了内存限制问题1。模型的4-bit量化权重占用约14GB1，首次运行将从Hugging Face下载15GB的权重1。
性能表现方面，在8GB M2 MacBook Air上，该引擎可实现5–6 tok/s的生成速度1；在M5 MacBook Pro上则可达到31–35 tok/s1。该引擎支持OpenAI兼容的本地服务器，具备流式传输和工具调用功能1。开发者为实现这一方案进行了超过100次实验1。
Developers have released TurboFieldfare, an open-source inference engine designed to run the 4-bit quantized Gemma 4 26B model on any M-series Mac while consuming approximately 2 GB of memory 1. Built using Swift and Metal, the engine overcomes memory constraints by keeping model shared components and key-value caches in RAM while streaming routed experts from the SSD 1.
The engine delivers varying performance depending on hardware capabilities 1. On an M2 MacBook Air with 8 GB of RAM, it achieves generation speeds of 5–6 tokens per second, while an M5 MacBook Pro reaches 31–35 tokens per second 1. The 4-bit quantized model weights total approximately 14 GB, and users will need to download 15 GB of weights from Hugging Face during their first run 1. The tool supports an OpenAI-compatible local server with streaming and tool-calling capabilities 1, and the developers report conducting over 100 experiments during development 1.
评论
还没有评论，欢迎留下第一条。