The challenges to OSS developers are numerous. First of all, many popular sites on the internet block crawlers other than Google and Bing, because only those ones seem to matter to their business, and any small upstart would be assumed to be a dodgy bot. Secondly, Google amasses the database it has only with vast data centers, incredible amounts of bandwidth, and power requirements unavailable to a startup.
We should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.
This helps to verify that a bot that announces itself as google bot is indeed a google bot. It doesn’t help identify a bot that pretends to be a user/browser.
