To be fair, even ignoring the Robots.txt is illegal in most western countries. I was a technical witness a while back, for a case about a bot ignoring the robots.txt. I said it was akin to a peeping tom ignoring a "no trespassing" sign, creeping into someones backyard, and looking through their window. Yes, they actually did bypass security controls, and therefore illegally "hacked" the site by ignoring it.