> They have every right to do whatever the hell they want, within law, to make their business as successful as possible. If you don't like it, go use DuckDuckGo, or start your own engine. There's a public domain page index available now, have at it.
Google has built its business on other people's content, is that simple. But then it started to gradually "steal" more and more of said people work, until it became the giant corporation that you mention.
And before people start down-voting me, just imagine a giant like IBM (let's say) building a search engine that steals other people's content (because that's what things like Google Places have become) and then making tons of money on the back of it. Hell will break loose.
I agree that Places and other similar parts of google's business are highly ethically suspect, far more so than the user profiling and data retention that are more often talked about.
I don't understand how it's related to this particular issue though.
> I don't understand how it's related to this particular issue though.
Because when it first started I think there was a mutual, non-spoken agreement between web content-owners and Google, sort of "I give you our content for free to use it on your search engine and you bring me back visitors by trying to be impartial in your SERPs; but I don't expect you to compete against me, not now, not ever".
That didn't last long, to say the least, but said content owners are now prisoners of Google (yes, it's a big word, but that's the fact). If my personal experience counts, I let Googlebot crawl my website like crazy, at speed rates for which I would certainly ban any other potential search engine bots, and that's because Google brings in 98% of my search engine traffic. And I'm sure I'm not the only one banning strange-looking bots, which could belong to potential new, wonderful search engines, but which at this rate will never see the light of day.
And, unfortunately, things like DuckDuckGo are sub-optimal outside of the English sphere of influence, at least that's what I could see in my case.
How are we blocking Googlebot? If we're using robots.txt then they can simply ignore it. Googlebot can begin to identify itself differently. There are a million ways to get around a Googlebot ban and I wouldn't be the guy who thinks he's smarter than Google. You'll lose that one. They'll find a way.
But anyway, this isnt really relevant. Can you tie it in for us?
Blocking is not done via robots.txt. It would more likely be IP-based.
Impossible to block Google? Probably true. But only because they have been allowed to grow so large as to be indefeasible. And the reason they've been able to do that is because of what the commenter said: websites allowed them to crawl, fast and hard, year after year. This is not true for all bots.
I'm not sure how this is relevant to the article and the specific issue. I don't disgree with what Google is doing in this case. And I understand why and where Google is headed.
The issue of who is allowed to crawl and who is not is something the commenter raised. It's a huge issue that people take for granted, in my opinion.
Hmm, how is this any worse than reddit or stumbleupon or delicious "profiting" from sharing everyone else's content? It's just reorganizing the information that is already available?
I think we are reaching a point in history where giant corporations are reaching a state-like status where corporate policy begins to have the weight of law in the sense of how many people are affected by it outside the company; corporate public relations and agenda becomes diplomacy with several different states, concerning different interests for the parties involved.
Not that these things weren't true when the corporations started, but as they scale, so does the influence and responsbaility.
It reminds me of the old Cyberpunk 2020 rpg book where corporations have they own law, police, etc.
Google has built its business on other people's content, is that simple. But then it started to gradually "steal" more and more of said people work, until it became the giant corporation that you mention.
And before people start down-voting me, just imagine a giant like IBM (let's say) building a search engine that steals other people's content (because that's what things like Google Places have become) and then making tons of money on the back of it. Hell will break loose.