Quote:
Originally Posted by bluecar1
using various scripts a website can serve a different page depending on what the source IP of the request is, i know of several sites that do this (no to avoid malware detection but to provide a polite notice to users from certian ISP networks)
the current information is it only follows talk talk users after the event, it does no proactive spidering as far as i am aware, unless you can provide information to the contrary
|
Understood.
I have asked and understand firstly that this is an extremely unlikely course of events, given previous experience from anti-malware that does spider and stroll the internet and secondly that the system has countermeasures to complicate this. No specifics were given sadly, but to describe this as being completely pointless as a service would strike me as harsh.
On the one hand an argument has been made that this service is different from other anti-malware because they come from well known IP ranges and are easy therefore to identify but on the other it's described as useless because malware can be tailored to hide from it. This is a rather contradictory series of statements.
I was not aware of spidering but it was mentioned as a possibility in this thread. If there is no spidering done this service doesn't have a requirement to read robots.txt as far as I'm aware. This is to be read by bots when they follow a link to another site - robots.txt is the first file they should read to ensure they behave themselves.
Given that ignorance of robots.txt is a charge levelled at the system that it would appear to not have functionality necessitating reading robots.txt is relevant.