Sneaky Little Robots
So I read an article in Wired News today about the ‘Covert Crawler’. I found this very interesting due to the AI associated with the said crawler. Basically, the crawler looks (and even acts) like a real person visiting a website. This will make it harder to weed out unwanted traffic (or band-with consumption), because it won’t be so easy to spot the robots from the humans browsing the site.
Provided this takes off, it could shift a lot of the way we handle security on our websites. Also, if landed in the wrong hands this could be very hard to track down the in-cognito robot.
Specifically, with blogs, there are many techniques that make sure spam comments are ignored. But could this change the way that robots interact with forms as well? How will this affect our traffic statistics? How will this affect our logs?
Thoughts?