一位AI研究者在NVIDIA 5090 GPU上用1.5小时、成本仅67美分的情况下，采用Transformer模型在ARC-AGI-1基准测试中取得44%的成绩1。这一结果与递归模型（TRM/HRM）的表现相当，但所需计算成本大幅降低1。该研究者已将代码开源，并鼓励其他贡献者在现有基础上进一步改进，目标是达到65%的准确率1。
在模型架构和训练方法上，研究采取了多项技术优化1。主要改进包括用SwiGlu替代GELU激活函数、用RMSNorm替代LayerNorm、增加网络层数至8层、采用3D RoPE位置编码，以及在任务级别添加加性嵌入1。在监督学习方面，研究者移除了输入token的训练，仅对输出token进行优化，这一改变使得成绩从40%提升至44%1。此外，研究使用NorMuon优化器替代传统的AdamW、应用Flash Attention和变长训练等技术手段1。
训练数据来自ARC-1、ARC-2和ConceptARC的综合数据集，研究者仔细排除了773个重叠任务以防止数据泄露1。研究者强调，样本效率是AI领域最重要的问题，降低计算成本的核心目标是加快迭代速度1。
A researcher has demonstrated significant cost efficiency in artificial intelligence model training by achieving a 44% score on the ARC-AGI-1 benchmark using only 67 cents worth of computational resources.1 The work was completed in 1.5 hours on an NVIDIA 5090 GPU, matching the performance of recursive models while dramatically reducing expenses.1
The approach employed a Transformer-based architecture enhanced through multiple technical optimizations.1 Key improvements included replacing GELU with SwiGlu activation functions, substituting LayerNorm with RMSNorm, increasing the network depth to 8 layers instead of 4, implementing 3D RoPE positional encoding, and incorporating task-level additive embeddings.1 The training methodology was refined by removing input token supervision and focusing exclusively on output token learning, which improved results from 40% to 44%.1 The researcher also adopted the NorMuon optimizer in place of AdamW and integrated flash attention with variable-length training techniques.1
The training dataset incorporated samples from ARC-1, ARC-2, and ConceptARC sources, with careful exclusion of 773 overlapping tasks to prevent data leakage.1 The researcher achieved 7% on ARC-2 using the same approach.1 The code has been made publicly available, with the author inviting community contributions toward reaching 65% accuracy.1 The research underscores a focus on sample efficiency as a critical frontier in AI development, emphasizing the importance of reducing computational costs to accelerate iteration cycles.1
评论
还没有评论，欢迎留下第一条。