Quote:
Originally Posted by Ignitionnet
Understood.
I have asked and understand firstly that this is an extremely unlikely course of events, given previous experience from anti-malware that does spider and stroll the internet and secondly that the system has countermeasures to complicate this. No specifics were given sadly, but to describe this as being completely pointless as a service would strike me as harsh.
On the one hand an argument has been made that this service is different from other anti-malware because they come from well known IP ranges and are easy therefore to identify but on the other it's described as useless because malware can be tailored to hide from it. This is a rather contradictory series of statements.
I was not aware of spidering but it was mentioned as a possibility in this thread. If there is no spidering done this service doesn't have a requirement to read robots.txt as far as I'm aware. This is to be read by bots when they follow a link to another site - robots.txt is the first file they should read to ensure they behave themselves.
Given that ignorance of robots.txt is a charge levelled at the system that it would appear to not have functionality necessitating reading robots.txt is relevant.
|
several people have been told (as far as i am aware) that it does honour robots.txt
but evidence did not support this