AI Crawlers Running Wild: Meta, Google, & OpenAI Dominate the Scene with Unprecedented Traffic
AI web crawlers, primarily from Meta, Google, and OpenAI, are generating massive traffic on websites, causing significant strain on web infrastructure. Meta leads this surge with 52% of the activity, followed by Google's 23% and OpenAI's 20%, together dominating the scene. OpenAI's bots, in particular, are responsible for intense traffic bursts, hitting 39,000 requests per minute at times. This demand is triggering concerns about scalability, cloud infrastructure load, and sustainability as AI adoption continues to surge.
Aug 21