一名开发者成功在价格约8美元的ESP32-S3微控制器上运行了2890万参数的语言模型[1]。这一突破相比之前在同类芯片上运行的26万参数模型，参数量增加了约100倍[1]。该模型在芯片本地运行，无需连接服务器[1]，文本生成速度约为每秒9tokens[1]。
开发者通过采用Google Gemma模型中的Per-Layer Embeddings技术实现了这一成果[1]，该技术将大部分模型参数存储在慢速闪存中，充分利用了芯片512KB的SRAM容量[1]。所用的语言模型基于TinyStories训练数据集[1]，展示了边缘设备上本地文本生成的可行性。
A developer has successfully deployed a 28.9 million parameter language model on an ESP32-S3 microcontroller, a device that costs approximately $8.[1] The model runs entirely on the chip itself without requiring a connection to external servers, generating text at a rate of approximately 9 tokens per second.[1] This represents a roughly 100-fold increase in model size compared to the 260,000 parameter models previously demonstrated on similar hardware.[1]
The achievement was made possible through the use of Per-Layer Embeddings technology, a technique derived from Google's Gemma models, which allows most of the model parameters to be stored in the chip's slower flash memory rather than its limited 512KB of SRAM.[1] The underlying model was trained on the TinyStories dataset.[1]