Absolutely a new standard is needed. This is the first time any major player has suggested that data from a users clickstream should be covered by robots.txt or anything like it.
The comparison to a web directory is perfect - in the absence of a robots.txt Google and other crawlers think the data is absolutely fair game. There isn't yet a corresponding analogy for clickstream data so its fair game. Are you sure all of Google's tools respect robots.txt for passive analysis of user initiated actions on other websites?
I was using "standard" in two different ways there. Let me fix that:
>You're addressing the letter of the "law" not the spirit. Do we really need a new formal standard to indicate that this is unethical? To me it's pretty plain from the ethical standards the web has already agreed upon.
The comparison to a web directory is perfect - in the absence of a robots.txt Google and other crawlers think the data is absolutely fair game. There isn't yet a corresponding analogy for clickstream data so its fair game. Are you sure all of Google's tools respect robots.txt for passive analysis of user initiated actions on other websites?