Block UserAgent libwww-perl and test

If you want to apply this globally, edit /etc/httpd/conf/httpd.conf (Centos). Otherwise, just modify the related vhost config file, adding the following:

ServerSignature Off
SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot
<Location />
Order allow,deny
Allow from all
Deny from env=bad_bot
</Location>

How to test:

Check if the site responds:

$ curl -I http://www.mywebsite.com
HTTP/1.1 200 OK
Date: Thu, 13 Nov 2014 17:12:16 GMT
Server: Apache
Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT
ETag: "1fa5dc-9dd5-507bf9cd66ec0"
Accept-Ranges: bytes
Content-Length: 40405
Vary: Accept-Encoding,Cookie
Content-Type: text/html; charset=UTF-8

 

Check if the agent libwww-perl is allowed or forbidden:

$ curl -A "libwww-perl" -I http://www.mywebsite.com
HTTP/1.1 403 Forbidden
Date: Thu, 13 Nov 2014 17:12:51 GMT
Server: Apache
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

 

Check if using Mozilla as agent works or not:

$ curl -A "Mozilla/5.0 (iPhone; U; CPU PerformanceOptimized iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -I http://www.mywebsite.com
HTTP/1.1 200 OK
Date: Thu, 13 Nov 2014 17:13:27 GMT
Server: Apache
Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT
ETag: "1fa5dc-9dd5-507bf9cd66ec0"
Accept-Ranges: bytes
Content-Length: 40405
Vary: Accept-Encoding,Cookie
Content-Type: text/html; charset=UTF-8