HTTP Robot Mitigation

HTTP Robot Mitigation

Synopsis

The Robot Mitigation module rewrites the Roboo module in C language.
The Robot Mitigation module provides defense against HTTP robot attacks, mainly against malicious scans and DDoS attacks. Robot Mitigation uses a "challenge" verification method by sending to clients specific responses that can be interpreted by browsers. If the client is a browser, then the original request will be re-generated containing a specific cookie value. Robot Mitigation will determine whether to allow this request to pass according to the inserted cookie.

Robot Mitigation has the following enhancements added to the original Roboo module:

• More efficient due to C language. Detects robots before the request reaches "content handler".
• Simplified configuration, easier to use.
• Calling sequence and actions are handled by NetEye security layer along with other security modules.
• During javascript challenges, Robot Mitigation returns a javascript code randomly taken from user-defined javascript code set instead of returning the same javascript code all the time.
• Support of the Dynamic blacklist module.
• Supports returning notification messages in html pages to clients when HTTP requests are blocked or added to the blacklist.

Directives

robot_mitigation

Enables or disables Robot Mitigation.

robot_mitigation_cookie_name

set cookie name of robot mitigation
Example:

robot_mitigation_cookie_name robot_cookie;

robot_mitigation_mode

Specify challenge type, including JavaScript and Flash:

• If js is selected, Robot Mitigation will return a javascript code to the client. If javascript is enabled on the client browser, it will initiate a new request.
• If swf is selected, Robot Mitigation will return a predefined flash file to the client. If the Flash plugin is installed on the client browser, it will run this flash file and initiate a new request.

robot_mitigation_blacklist

Set the threshold of faling to pass robot check. If the client over this threshold, it will be added into dynamic blacklist.
Example:

robot_mitigation_blacklist 3;

So this means if a client has failed for 4 times, this client will be added into blacklist.

robot_mitigation_timeout

Sets the time for the next challenge to be initiated, in seconds.

robot_mitigation_challenge_ajax

This directive sets if challenge for ajax requests based on XMLRequest header.

robot_mitigation_global_whitelist

This directives specifies the IP whitelist and User-Agent whitelist which are globally defined. The IP whitelist is provided by nginx's geo module.
Example:

#define an ip whitelist
geo $ip_wl {
    ranges;
    default 0;

    127.0.0.1-127.0.0.1 1;
    3.0.0.1-3.2.1.254 1;
}

#define an UA whitelist
whitelist_ua $ua_wl {
    "autotest" ".*\.test\.com";
}

server {
    location {
         robot_mitigation_global_whitelist ua_var_name=ua_wl ip_var_name=ip_wl ip_var_value=1;
    }
}

robot_mitigation_force_update_blacklist (Pro)

This directive increases dynamic blacklist counter without standard robot mitigation verification methods based on js/swf/cookie. With this directive it is possible to create anti ddos and anti bots rules based on the various own conditions.
Such a policy also is effective protection against the advanced bots who can pass standard robot mitigation checks.
Anti ddos example based on IP Behavior variables:

error_page 555 =403 /555.html;
location = /555.html {
	internal;
	root /home/site/www/errors;
	robot_mitigation on;
	robot_mitigation_blacklist 3;
	robot_mitigation_force_update_blacklist on;
}

...

location /sens {
	ip_behavior_sensitive;
	ifall ($insensitive_percent >= 0) ($insensitive_percent <= 5) {
		return 555;
	}
	...
}

Statistics

Use SEnginx Statistics module to see robot mitigation statistics.