If you want to apply this globally, edit /etc/httpd/conf/httpd.conf (Centos). Otherwise, just modify the related vhost config file, adding the following:
|
1 2 3 4 5 6 7 |
ServerSignature Off SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot <Location /> Order allow,deny Allow from all Deny from env=bad_bot </Location> |
How to test:
Check if the site responds:
|
1 2 3 4 5 6 7 8 9 10 |
$ curl -I http://www.mywebsite.com HTTP/1.1 200 OK Date: Thu, 13 Nov 2014 17:12:16 GMT Server: Apache Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT ETag: "1fa5dc-9dd5-507bf9cd66ec0" Accept-Ranges: bytes Content-Length: 40405 Vary: Accept-Encoding,Cookie Content-Type: text/html; charset=UTF-8 |
Check if the agent libwww-perl is allowed or forbidden:
|
1 2 3 4 5 6 |
$ curl -A "libwww-perl" -I http://www.mywebsite.com HTTP/1.1 403 Forbidden Date: Thu, 13 Nov 2014 17:12:51 GMT Server: Apache Vary: Accept-Encoding Content-Type: text/html; charset=iso-8859-1 |
Check if using Mozilla as agent works or not:
|
1 2 3 4 5 6 7 8 9 10 |
$ curl -A "Mozilla/5.0 (iPhone; U; CPU PerformanceOptimized iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -I http://www.mywebsite.com HTTP/1.1 200 OK Date: Thu, 13 Nov 2014 17:13:27 GMT Server: Apache Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT ETag: "1fa5dc-9dd5-507bf9cd66ec0" Accept-Ranges: bytes Content-Length: 40405 Vary: Accept-Encoding,Cookie Content-Type: text/html; charset=UTF-8 |