Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation
Guide That Contains This Content
[+] Expand All
[-] Collapse All

    Honeypot Processors: Robots Processor

    The Robots.txt proxy processor is responsible for catching malicious spiders that do not behave in accordance with established standards for spidering. Hackers often utilize the extra information sites expose to spiders, and then use that information to access resources normally not linked from the public site. Because this activity is effectively breaking established standards for spidering, this processor will also identify hackers who are using the information maliciously.

    Table 1: Robots Processor Configuration Parameters

    Parameter

    Type

    Default Value

    Description

    Basic

    Processor Enabled Boolean

    Boolean

    True

    Whether traffic should be passed through this processor.

    Advanced

    Fake Disallowed Directories

    String

    Random

    The path to a fake directory to add to the disallow rules in the robots.txt file. This path should be completely fake and not overlap with actual directories.

    Incident: Malicious Spider Activity

    Boolean

    True

    The user requested a resource which was restricted in the spider rules file, indicating this user is not a good spider, but is spidering the site anyway.

    Published: 2015-02-04