一位开发者在Hacker News上分享了在M4 Pro Mac mini上搭建本地大语言模型服务器的完整技术方案。该系统采用48GB RAM的硬件配置1，部署耗时约30分钟1，通过oMLX推理服务器1运行Qwen 3.6-35B和Gemma-4等多个模型1。其中Qwen 3.6-35B-A3B-OptiQ-4bit模型占用约20GB存储空间1，Gemma-4-E4B-it-OptiQ-4bit模型占用约2.4GB1。作者采用Tailscale mesh网络方案1实现多设备对本地模型的访问。
相比云端API服务，本地部署具有明显的经济优势。作者指出，云端月均订阅成本约400美元1，而本地方案仅需硬件一次性投资加电费支出1。在模型精度方面，采用4-bit量化版本相比BF16基准精度仅损失1-2个百分点1。该方案特别适用于MoE（混合专家）架构模型，以Qwen 3.6-35B为例，虽然总参数达35B，但每个token仅激活其中3B参数1，有效降低了推理成本。本地部署还提供了数据隐私保护和离线可用等优势1。
A developer has detailed their experience deploying a local large language model server on an M4 Pro Mac mini, offering a practical alternative to cloud-based AI services.1 The system features 48GB of RAM and runs models including Qwen 3.6-35B and Gemma-4, with inference powered by the oMLX server and network access managed through Tailscale mesh networking.1 The entire setup was completed in approximately 30 minutes.1
The local deployment leverages quantized model versions to optimize performance and resource usage. The primary model, Qwen 3.6-35B-A3B-OptiQ-4bit, occupies roughly 20GB of storage, while the secondary Gemma-4-E4B-it-OptiQ-4bit model requires approximately 2.4GB.1 The 4-bit quantized versions show only a 1-2 percentage point accuracy loss compared to their BF16 baseline counterparts.1 Notably, despite Qwen 3.6-35B containing 35 billion total parameters, the mixture-of-experts architecture activates only 3 billion parameters per token.1
The developer highlighted significant cost advantages of this local approach over cloud subscriptions. While cloud API services typically cost around $400 per month, the local setup requires only a one-time hardware investment and ongoing electricity expenses.1 This solution also provides benefits including data privacy and offline availability.1
评论
还没有评论，欢迎留下第一条。