Русский

HTTP Robot Mitigation

Contents

Synopsis

The Robot Mitigation module rewrites the Roboo module in C language.
The Robot Mitigation module provides defense against HTTP robot attacks, mainly against malicious scans and DDoS attacks. Robot Mitigation uses a "challenge" verification method by sending to clients specific responses that can be interpreted by browsers. If the client is a browser, then the original request will be re-generated containing a specific cookie value. Robot Mitigation will determine whether to allow this request to pass according to the inserted cookie.

Robot Mitigation has the following enhancements added to the original Roboo module:

  • More efficient due to C language. Detects robots before the request reaches "content handler".
  • Simplified configuration, easier to use.
  • Calling sequence and actions are handled by NetEye security layer along with other security modules.
  • During javascript challenges, Robot Mitigation returns a javascript code randomly taken from user-defined javascript code set instead of returning the same javascript code all the time.
  • Support of the IP Blacklist module.
  • Supports returning notification messages in html pages to clients when HTTP requests are blocked or added to the blacklist.


Directives

robot_mitigation

Syntax robot_mitigation on | off;
Default off
Context Location

Enables or disables Robot Mitigation.

robot_mitigation_cookie_name

Syntax robot_mitigation_cookie_name cookie_name;
Default SENGINX-ROBOT-MITIGATION
Context Location

set cookie name of robot mitigation
Example: 

robot_mitigation_cookie_name robot_cookie;


robot_mitigation_mode

Syntax robot_mitigation_mode js | swf;
Default js
Context Location

Specify challenge type, including JavaScript and Flash:

  • If js is selected, Robot Mitigation will return a javascript code to the client. If javascript is enabled on the client browser, it will initiate a new request.
  • If swf is selected, Robot Mitigation will return a predefined flash file to the client. If the Flash plugin is installed on the client browser, it will run this flash file and initiate a new request.

See also robot_mitigation_mode (Pro).


robot_mitigation_blacklist

Syntax robot_mitigation_blacklist failed_count;
Default
Context Location

Set the threshold of faling to pass robot check. If the client over this threshold, it will be added into IP Blacklist.
Example: 

robot_mitigation_blacklist 3;

So this means if a client has failed for 4 times, this client will be added into blacklist.


robot_mitigation_timeout

Syntax robot_mitigation_timeout timeout;
Default 60
Context Location

Sets the time for the next challenge to be initiated, in seconds.

robot_mitigation_challenge_ajax

Syntax robot_mitigation_challenge_ajax on | off;
Default off
Context Location
Version Since 1.5.5

This directive sets if challenge for ajax requests based on XMLRequest header.

robot_mitigation_global_whitelist

Syntax robot_mitigation_global_whitelist ua_var_name=UA whitlist ip_var_name=IP whitelist ip_var_value=value;
Default
Context Location
Version Since 1.5.11

This directives specifies the IP whitelist and User-Agent whitelist which are globally defined. The IP whitelist is provided by nginx's geo module.
Example:

#define an ip whitelist
geo $ip_wl {
    ranges;
    default 0;

    127.0.0.1-127.0.0.1 1;
    3.0.0.1-3.2.1.254 1;
}

#define an UA whitelist
whitelist_ua $ua_wl {
    "autotest" ".*\.test\.com";
}

server {
    location {
         robot_mitigation_global_whitelist ua_var_name=ua_wl ip_var_name=ip_wl ip_var_value=1;
    }
}


robot_mitigation_force_update_blacklist (Pro)

Syntax robot_mitigation_force_update_blacklist on | off;
Default off
Context Location
Version Since SEnginx Pro 1.10.0

This directive increases IP Blacklist counter without standard robot mitigation verification methods based on js/swf/captcha. This directive allows you to create DDoS and bot protection rules based on various custom conditions.
This policy also provides effective protection against advanced bots that can pass standard robot mitigation checks.
Anti ddos example based on IP Behavior variables: 

error_page 555 =403 /555.html;
location = /555.html {
	internal;
	root /home/site/www/errors;
	robot_mitigation on;
	robot_mitigation_blacklist 3;
	robot_mitigation_force_update_blacklist on;
}

...

location /sens {
	ip_behavior_sensitive;
	ifall ($insensitive_percent >= 0) ($insensitive_percent <= 5) {
		return 555;
	}
	...
}

robot_mitigation_mode (Pro)

Syntax robot_mitigation_mode js | swf | captcha;
Default js
Context Location
Version since SEnginx Pro 1.10.4

This option is the same as robot_mitigation_mode, but "captcha" challenge type was added.
Specify challenge type:

  • If js is selected, Robot Mitigation will return a javascript code to the client. If javascript is enabled on the client browser, it will initiate a new request.
  • If swf is selected, Robot Mitigation will return a predefined flash file to the client. If the Flash plugin is installed on the client browser, it will run this flash file and initiate a new request.
  • If captcha is selected, Robot Mitigation will return a javascript captcha code to the client. If javascript is enabled on the client browser, it will show a captcha. If captcha resolving is successfull, the script initiate a new request.


Statistics

Use SEnginx Statistics module to see robot mitigation statistics.