I think this article underestimates the proportion of crawler traffic. Big companies like Google, Apple, and some grey area companies like Ahrefs, Bytedance, and Yandex are easy to count. But there's a ton of probably automated GET requests that use user agents typical of human driven browsers. A real browser will ask for images, JavaScript, CSS files after a GET of an HTML file, these probably automated GETs don't have those linked to GETs.
I strongly believe that we f you could count the probably automated requests accurately, human driven requests would be a very low proportion of all requests, less than 30%.
I think this article underestimates the proportion of crawler traffic. Big companies like Google, Apple, and some grey area companies like Ahrefs, Bytedance, and Yandex are easy to count. But there's a ton of probably automated GET requests that use user agents typical of human driven browsers. A real browser will ask for images, JavaScript, CSS files after a GET of an HTML file, these probably automated GETs don't have those linked to GETs.
I strongly believe that we f you could count the probably automated requests accurately, human driven requests would be a very low proportion of all requests, less than 30%.