Honeypot Processors: Robots Processor

The Robots.txt proxy processor is responsible for catching malicious spiders that do not behave in accordance with established standards for spidering. Hackers often utilize the extra information sites expose to spiders, and then use that information to access resources normally not linked from the public site. Because this activity is effectively breaking established standards for spidering, this processor will also identify hackers who are using the information maliciously.

Table 20: Robots Processor Configuration Parameters

Parameter

Type

Default Value

Description

Basic

Processor Enabled Boolean

Boolean

True

Whether traffic should be passed through this processor.

Advanced

Fake Disallowed Directories

String

Random

The path to a fake directory to add to the disallow rules in the robots.txt file. This path should be completely fake and not overlap with actual directories.

Incident: Malicious Spider Activity

Boolean

True

The user requested a resource which was restricted in the spider rules file, indicating this user is not a good spider, but is spidering the site anyway.