Cloudflare推出了新的AI流量管理工具，使网站所有者能够对不同类型的AI爬虫进行精细化控制。[1]该系统将AI使用场景分为三类：Search（用于索引内容以回答问题）、Agent（代表用户实时执行任务）和Training（用于训练或微调模型）。[1]
根据新规则，Cloudflare将于2026年9月15日设置新的默认值，其中Training和Agent爬虫在显示广告的页面上将被默认阻止，而Search爬虫保持允许状态。[1]同时，Cloudflare引入了内容使用信号机制，网站所有者可在robots.txt中声明三种偏好：immediate（不存储重用）、reference（索引摘录并链接回）以及full（总结和重现）。[1]
为加强对爬虫的可见性和管理，Cloudflare推出了BotBase功能，企业Bot管理客户可以查看所有已知爬虫的完整目录及其分类。[1]此外，Cloudflare更新了Verified标签的定义，使其不再意味着默认允许，而是表示该爬虫在其相关类别中是可允许的。[1]技术层面上，该系统采用RFC 7239 Forwarded头实现传递信任机制，允许通过中介层追踪真实运营者身份。[1]
Cloudflare has introduced enhanced controls for managing artificial intelligence traffic, enabling website operators to make granular decisions about how different categories of AI bots interact with their content [1]. The new system categorizes AI activity into three distinct use cases: Search, which indexes content to answer questions; Agent, which performs real-time tasks on behalf of users; and Training, which gathers data for model development or refinement [1].
Starting September 15, 2026, Cloudflare will implement updated default policies for these traffic types [1]. On pages displaying advertisements, Training and Agent bots will be blocked by default, while Search activity will remain permitted [1]. Website owners will also gain the ability to specify content usage signals through their robots.txt files, choosing from three options: immediate (data not stored for reuse), reference (indexed excerpts with links back to the source), and full (content summarized or reproduced) [1].
The company has also rolled out BotBase, a feature for enterprise Bot Management customers that provides a complete directory of known bots along with their classifications [1]. Additionally, Cloudflare has refined the meaning of its Verified label, which now indicates that a bot is permitted within its relevant category rather than automatically allowing all verified bots [1]. The platform now implements transitive trust using RFC 7239 Forwarded headers, enabling website owners to trace the actual operators behind bot traffic passing through intermediary layers [1].