Браузеры, поисковые боты, HTTP-клиенты — анализ и классификация
Twitterbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http:/…
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari…
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http:/…
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http:/…
facebookexternalhit/1.1; kakaotalk-scrap/1.0; +https://devtalk.kakao.com/t/scrap/3398…
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.3…
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://w…
Googlebot-Video/1.0
facebookexternalhit/1.1
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Googlebot-News
Mozilla/5.0 (compatible; bingbot/2.0 http://www.bing.com/bingbot.htm)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http:/…
Mozilla/5.0 (compatible; SemrushBot; +http://www.semrush.com/bot.html)
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)
Twitterbot/1.0
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://w…
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, …
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, …
Googlebot-Image/1.0
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http:/…
TelegramBot (like TwitterBot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://w…
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, …
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)