If you want to apply this globally, edit /etc/httpd/conf/httpd.conf (Centos). Otherwise, just modify the related vhost config file, adding the following:
ServerSignature Off SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot <Location /> Order allow,deny Allow from all Deny from env=bad_bot </Location>
How to test:
Check if the site responds:
$ curl -I http://www.mywebsite.com HTTP/1.1 200 OK Date: Thu, 13 Nov 2014 17:12:16 GMT Server: Apache Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT ETag: "1fa5dc-9dd5-507bf9cd66ec0" Accept-Ranges: bytes Content-Length: 40405 Vary: Accept-Encoding,Cookie Content-Type: text/html; charset=UTF-8
Check if the agent libwww-perl is allowed or forbidden:
$ curl -A "libwww-perl" -I http://www.mywebsite.com HTTP/1.1 403 Forbidden Date: Thu, 13 Nov 2014 17:12:51 GMT Server: Apache Vary: Accept-Encoding Content-Type: text/html; charset=iso-8859-1
Check if using Mozilla as agent works or not:
$ curl -A "Mozilla/5.0 (iPhone; U; CPU PerformanceOptimized iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -I http://www.mywebsite.com HTTP/1.1 200 OK Date: Thu, 13 Nov 2014 17:13:27 GMT Server: Apache Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT ETag: "1fa5dc-9dd5-507bf9cd66ec0" Accept-Ranges: bytes Content-Length: 40405 Vary: Accept-Encoding,Cookie Content-Type: text/html; charset=UTF-8