Tag Archives: apache

Virtualhost and Letsencrypt

Quick guideline about how to install multiple sites on a single server using Virtualhosting, and have the SSL certificate installed and automatically renewed using Letsencrypt.

There are plenty of how to online, but I wanted to have a quick reference page for myself 🙂

Firstly, this has been tested on Debian 12, but it should work on previous Debian versions and Ubuntu too.

Apache setup and virtualhosts

Firstly, install Apache and other packages that you will mostly likely need, especially if you run WordPress or any php based framework:

apt-get install apache2 php php-mysql libapache2-mod-php php-gd php-curl net-tools telnet dos2unix

Now, you should create the folder structure to host your sites. I used /var/www/virtualhosts/<site>/public_html

I made sure permissions were set correctly too:

chown -R www-data:www-data /var/www/
find /var/www -type -d -exec chmod 775 {} \;

Now, create a virtualhost file for each site. In the following example we are going to create the conf file for site1.

Create /etc/apache2/sites-available/site1.conf

<VirtualHost *:80>
    ServerName site1.com
    ServerAlias www.site1.com
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/virtualhosts/site1/public_html

    <Directory /var/www/virtualhosts/site1/public_html>
        Options -Indexes +FollowSymLinks
        AllowOverride All
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/site1-error.log
    CustomLog ${APACHE_LOG_DIR}/site1-access.log combined
</VirtualHost>

Do the same for all the sites you have.

Once done, upload the content of your sites in public_html folder.

Disable all the default Apache sites and enable the ones you have created. You can use the commands a2dissite and a2ensite or manually create symbolic links into /etc/apache2/sites-enabled/

Check that all the virtualhosts are properly loaded:

source /etc/apache2/envvars
apache2 -S

You should see all your sites under *80 section.
Right now we have enabled only Apache on port 80 to return the sites we have hosted. No 443 yet.

Now, you can use curl to do some tests to see if the virtual hosts are responding correctly.

~ curl -IH'Host: site1.com' http://<server_IP>  # to get the header of site1.com
~ curl -H'Host: site1.com' http://<server_IP>  # to get the full page of site1.com

Hopefully all works (if not, troubleshoot it heheh), let’s point our DNS to our server, and test directly using the domain names.

All good? Cool!

Make sure now that your firewall allows port 80 and port 443. Even if you’re considering to serve your site ONLY over SSL (port 443), the certbot tool that does the auto-renewal of the certificate needs port 80 open.

Installation and configuration of certbot – Letsencrypt

As root, issue the below commands:

apt-get install snapd
snap install core
snap refresh core
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot

You have now the certbot tool installed.

Following the above example of site1.com, we are going now to get the SSL certificate for that site (even the www.site1.com one), and let the tool install and configure everything automatically.

certbot --apache -d site1.com -d www.site1.com

Hopefully all goes well 🙂 Repeat for each of your sites accordingly.

Once done with all the sites, just to make sure the auto-renewal works, you can also issue a dry-run check:

certbot renew --dry-run

Letsencrypt certificates last 90 days (afaik), but the certbot tool installed in this way does the auto-renewal in an automatic fashion.
If you’re curios where this is written (you might think about cron but unable to find anything – like it happend to me).
If this is the case, you can try to run this command, and you may find the certbot listed:

systemctl list-timers

More information are available on the official website at this address.

You can now test using curl again, but hitting https instead of http:

~ curl -IH'Host: site1.com' https://<server_IP>  # to get the header of site1.com
~ curl -H'Host: site1.com' https://<server_IP>  # to get the full page of site1.com

Oh, one note.
By default, at least at the time when I’m writing this article, once you install the certificate, the *80 virtualhost of your site will be modified, adding the following lines, which force a 302 redirect from http to https.

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.site1.com [OR]
RewriteCond %{SERVER_NAME} =site1.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

If it’s what you want – cool.
If you still want to serve your site on http AND https, comment out (or delete) those new lines.

Happy virtualhosting and ssl’ing! 🙂

Disable execution of script in a specific path within a vhost

Add this into your vhost (making sure to match the Directory directive with the correct path):

<Directory /var/www/vhosts/mydomain.com/uploads>
SetHandler none
SetHandler default-handler
Options -ExecCGI
php_flag engine off
RemoveHandler .cgi .php .php3 .php4 .php5 .phtml .pl .py .pyc .pyo .sh
</Directory>

Which PHP script is eating my memory?

For mod_php, you can just add this to your Apache LogFormat:

%{mod_php_memory_usage}n

Might be best to add it right at the end, so as not to break any log parsing.
Credit: http://tech.superhappykittymeow.com/?p=220

For PHP-FPM (i.e. anyone with NginX or anyone with one of our optimised Magento setups.), put the following into your FPM pool config file, probably here:
– /etc/php-fpm.d/website.conf (RHEL/CentOS)
– /etc/php5-fpm/pools.d/website.conf (Ubuntu)

access.log = /var/log/php-fpm/domain.com-access.log
access.format = "%p %{HTTP_X_FORWARDED_FOR}e - %u %t \"%m %{REQUEST_URI}e\" %s %f %{mili}d %{kilo}M %C%% \"%{HTTP_USER_AGENT}e\""

Notes:
• Permissions matter. Check that the User and/or Group (of THIS fpm pool) can write to /var/log/php-fpm (or /php5-fpm, whatever)
• %{HTTP_X_FORWARDED_FOR}e is there because I was behind a Load Balancer, Varnish and/or other reverse proxy.
• %{REQUEST_URI}e\ is there because this CMS (like most, now), rewrite everything to index.php. I want to know the original request, not just the script name.
• %{kilo}M %C – These are the kickers: Memory usage and CPU PerformanceOptimized usage per request. Ker-pow. Pick your favourite awk/sort one-liner to weed out the heavy hitters.

You can log pretty much anything: any arbitrary header, (like REQUEST_URI), and you should find all the options in the comments under the default “www.conf” packaged config file.
Given that this starts with the PID, it should also help track down any segfaults you see in /var/log/messages / kern.log.
Oh, and while you’re at it, PHP-FPM has a slow log you can enable.
Oh, and don’t forget your old friend logrotate.

Counting the average PHP-FPM process memory usage

Looking at the ‘ps’ output isn’t always accurate because of shared memory. Here’s a one-liner to count up my FPM pool, where “www” is in name of the FPM pool:

for pid in $(ps aux | grep fpm | grep "pool www" | awk '{print $2}'); do pmap -d $pid | tail -1 ; done | sed 's/K//' | awk '{sum+=$4} END {print sum/NR/1024}’

Thanks to Dan Farmer for his original Apache memory script, which made me think to do this.

Divide and conquer

Do make sure that multiple websites on the same server are using their own FPM pools, that way it should be much easier to see what’s what.

But you can also separate by URL. I’ve done this with the M-word, but the theory should stand for any CMS where the tasks performed under the admin section will be more intensive than normal front-end user traffic.
The following should work with WordPress, assuming the path is /wp-admin . Magento users can (and should) change their admin path from the default, so watch out for that (usually configured in local.xml)

Or anything really – maybe that “importvideo” script is especially memory-intensive and you don’t want to allow it to bloat up all your processes.

1. Set up a separate PHP-FPM pool.
– give it a different name, different log file names, and listen on a different socket/port.
– Perhaps with a much higher memory_limit
– Perhaps with many fewer pm.max_children (ask customer how many humans actually use the “backend”). You might only need 5 or so.
– Perhaps it needs a longer max_execution_time …. you get the idea.

2. Send your “admin” traffic there.
Here’s how you might go about it. These are excerpts and ideas, rather than solid copy-paste config.

Nginx: something like this:

upstream backend {
server unix:/var/run/php5-domain.com.sock;
}
upstream backend-admin {
server unix:/var/run/php5-domain.com-admin.sock;
}

map "URL:$request_uri." $fcgi_pass {
default backend;
~URL:.*admin.* backend-admin;
}

… then in your ‘server{}’ bit, use the variable for fastcgi_pass:

fastcgi_pass $fcgi_pass;

Apache: something like this…

# Normal backend alias, corresponds with FastCGIExgternalServer 
Alias /php.fcgi /dev/shm/domain-php.fcgi
<Location ~ admin>
# Override Action for “admin” URLs
Action application/x-httpd-php /domain-admin.fcgi
</Location>
Alias /domain-admin.fcgi /dev/shm/domain-admin-php.fcgi

3. You can probably now LOWER the memory_limit for your “main” pool
– if the rest of the website doesn’t use much memory. Now bask in the memory you just saved for the whole application server.

4. Bonus: NewRelic app separation
Don’t let all that heavy Admin work interfere with your nice / renice appdex statistics – we know (and expect) your backend dashboard to be slower.
Put this in the FPM pool config:

php_value[newrelic.appname] = "www.domain.com Admin"

Credits: https://willparsons.tech/

What to do with a down Magento site

1. Application level logs – First place to look.

If you are seeing the very-default-looking Magento page saying “There has been an error processing your request”, then look in here:

ls -lart <DOCROOT>/var/report/ | tail

The stack trace will be in the latest file (there might be a lot), and should highlight what broke.
Maybe the error was in a database library, or a Redis library…see next step if that’s the case.

General errors, often non-fatal, are in <DOCROOT>/var/log/exception.log
Other module-specific logs will be in the same log/ directory, for example SagePay.

NB: check /tmp/magento/var/ .
If the directories in the DocumentRoot are not writable (or weren’t in the past), Magento will use /tmp/magento/var and you’ll find the logs/reports/cache in there.

2. Backend services – Magento fails hard if something is inaccessible

First, find the local.xml. It should be under <DOCROOT>/app/etc/local.xml or possibly a subdirectory like <DOCROOT>/store/app/etc/local.xml

From that, take note of the database credentials, the <session_save>, and the <cache><backend>. If there’s no <cache> section, then you are using filesystem so it won’t be memcache or redis.

– Can you connect to that database from this server? authenticate? or is it at max-connections?
– To test memcache, “telnet host 11211” and type “STATS“.
– To test Redis, “telnet host 6379” and type “INFO”.
You could also use:

redis-cli -s /tmp/redis.sock -a PasswordIfThereIsOne info

If you can’t connect to those from the web server, check that the relevant services are started, pay close attention to the port numbers, and make sure any firewalls allow the connection.
If the memcache/redis info shows evictions > 0, then it’s probably filled up at some point and restarting that service might get you out of the water.

ls -la /etc/init.d/mem* /etc/init.d/redis*

3. Check the normal places – sometimes it’s nothing to do with Magento!

– PHP-FPM logs – good place for PHP fatal errors. usually in var/log/php[5]-fpm/

– Apache or nginx logs
– Is Apache just at MaxClients?
– PHP-FPM max_children?

ps aux | grep fpm | grep -v root | awk '{print $11, $12, $13}' | sort | uniq -c

– Is your error really just a timeout, because the server’s too busy?
– Did OOM-killer break something?

grep oom /var/log/messages /var/log/kern.log

– Has a developer been caught out by apc.stat=0 (or opcache.validate_timestamp=0) ?

Credits: https://willparsons.tech/

Fail2ban notes

General notes about Fail2ban

### Fail2Ban ###

Best practise:
- do NOT edit /etc/fail2ban/jail.conf BUT create a new /etc/fail2ban/jail.local file

=============================================================
# Test fail2ban regex:
example: fail2ban-regex /var/log/secure /etc/fail2ban/filter.d/sshd.conf
example2: fail2ban-regex --print-all-matched/var/log/secure /etc/fail2ban/filter.d/sshd.conf

=============================================================
# Remove email notifications:

comment out 'sendmail-whois' from action in [ssh-iptables]
FYI, comment with # at the BEGINNING of the line like this or it won't work!!!

[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
#           sendmail-whois[name=SSH, dest=root, [email protected], sendername="Fail2Ban"]
logpath  = /var/log/secure
maxretry = 5


=============================================================
# Wordpress wp-login - block POST attacks

/etc/fail2ban/jail.local

[apache-wp-login]
enabled = true
port = http,https
filter = apache-wp-login
logpath = /var/log/httpd/blog.tian.it-access.log
maxretry = 3
bantime = 604800 ; 1 week
findtime = 120

----------------------------------------------------------------------------------------------------------------------

/etc/fail2ban/filter.d/apache-wp-login.conf
[Definition]
failregex = <HOST>.*POST.*wp-login.php HTTP/1.1
ignoreregex =

=============================================================

# Manually ban an IP:
fail2ban-client -vvv set <CHAIN> banip <IP>

# Check status of sshd chain
fail2ban-client status sshd

How to “SSH” brute force

If you want to make safer your remote server, it is good practise to use a good combination of sshd setup and fail2ban.

Firstly, you should setup your server to allow only key auth, and no passwords. This will drastically reduce the risk. This means anyway that you need to keep your ssh key safe and you won’t be able to access your server unless you have this key. Most of the time is something possible 🙂

For this reason, I’m explaining here how I configured my server.

SSHD

/etc/ssh/sshd_config

Have these settings in the config file (NOTE: the verbosity is for Fail2ban)

LogLevel VERBOSE

PasswordAuthentication no

(restart sshd)

/etc/fail2ban/jail.local

[DEFAULT]
# Ban hosts for 
# one hour:
#bantime = 3600

# one day:
bantime = 86400

# A host is banned if it has generated "maxretry" during the last "findtime"
# # seconds.
findtime  = 30

# # "maxretry" is the number of failures before a host get banned.
maxretry = 5

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true
filter = sshd-aggressive
port     = ssh
logpath  = /var/log/secure
maxretry = 3
findtime = 30
bantime  = 86400

/etc/fail2ban/filter.d/sshd.conf

Add a custom section after the ddos one:

custom = ^%(__prefix_line_sl)sDisconnected from <HOST> port [0-9]+ \[preauth\]$

This line matches whoever tries to connect without a proper ssh key.

Add this line to include custom to the sshd-aggressive setup:

aggressive = %(normal)s
             %(ddos)s
             %(custom)s

Apache ProxyPass for WordPress master-slave setup

Simple way

Ensure certain traffic goes to a certain server (master), you can use this:

<LocationMatch "^/wordpress/wp-admin/?.*>
ProxyPreserveHost On
ProxyPass http://ip.of.master.server/
</LocationMatch>

For a better setup with Variables, just follow the… following steps 🙂

Step One: Configure Environment

We need to setup some environment variables to get this to work correctly.
Add the following to your environment on the slave server(s):

RHEL/CentOS: /etc/sysconfig/httpdi

OPTIONS="-DSLAVE"
export MASTER_SERVER="SERVERIP HERE"

Ubuntu: /etc/apache2/envvars

export APACHE_ARGUMENTS="-DSLAVE"
export MASTER_SERVER="SERVERIP HERE"

Step Two: Configure your VirtualHost

In your VirtualHost configuration do something like the following.

<IfDefine SLAVE>
RewriteEngine On
ProxyPreserveHost On
ProxyPass /wp-admin/http://${MASTER_SERVER}/wp-admin/
ProxyPassReverse /wp-admin/http://${MASTER_SERVER}/wp-admin/

RewriteCond %{REQUEST_METHOD} =POST
RewriteRule . http://${MASTER_SERVER}%{REQUEST_URI} [P]
</IfDefine>

Block UserAgent libwww-perl and test

If you want to apply this globally, edit /etc/httpd/conf/httpd.conf (Centos). Otherwise, just modify the related vhost config file, adding the following:

ServerSignature Off
SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot
<Location />
Order allow,deny
Allow from all
Deny from env=bad_bot
</Location>

How to test:

Check if the site responds:

$ curl -I http://www.mywebsite.com
HTTP/1.1 200 OK
Date: Thu, 13 Nov 2014 17:12:16 GMT
Server: Apache
Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT
ETag: "1fa5dc-9dd5-507bf9cd66ec0"
Accept-Ranges: bytes
Content-Length: 40405
Vary: Accept-Encoding,Cookie
Content-Type: text/html; charset=UTF-8

Check if the agent libwww-perl is allowed or forbidden:

$ curl -A "libwww-perl" -I http://www.mywebsite.com
HTTP/1.1 403 Forbidden
Date: Thu, 13 Nov 2014 17:12:51 GMT
Server: Apache
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

Check if using Mozilla as agent works or not:

$ curl -A "Mozilla/5.0 (iPhone; U; CPU PerformanceOptimized iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" -I http://www.mywebsite.com
HTTP/1.1 200 OK
Date: Thu, 13 Nov 2014 17:13:27 GMT
Server: Apache
Last-Modified: Thu, 13 Nov 2014 16:00:35 GMT
ETag: "1fa5dc-9dd5-507bf9cd66ec0"
Accept-Ranges: bytes
Content-Length: 40405
Vary: Accept-Encoding,Cookie
Content-Type: text/html; charset=UTF-8

How to bypass Varnish

Test loading speed ignoring Varnish

>> adding “Cookie: NO_CACHE=1” will make Varnish to pass the request directly to Apache

time curl -I http://example.com/pagetotest -H "Cookie: NO_CACHE=1"

Source IP in Apache logs

If your server is under a load balancer, you might see HTTP requests coming from the Load Balancer’s IP instead of the actual visitor. Generally, load balancers are “recording” the original source IP in the X-Forwarded-For header. This means that this is the header that we need to log in our Apache logs to get the information that we want.

Here how to make this happen:

Default rule:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Modified rule that includes the X-Forwarded-For definition:

LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined


==============================================

>> Best way:


On httpd.conf:
=====================
SetEnvIfNoCase X-Forwarded-For "." from_proxy=1

LogFormat "%{CF-Connecting-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
=======================

On vhost.conf
============================
CustomLog /var/log/httpd/vhost-access.log forwarded env=from_proxy
CustomLog /var/log/httpd/vhost-access.log combined env=!from_proxy
============================

Source: https://community.rackspace.com/products/f/25/t/211

Apache Rewrite rules

Rewrite rules examples:

This can be added in vhost configuration OR in .htaccess file

How to rewrite all web request on my site without www to www.domain.com

RewriteCond %{HTTP_HOST} !^www\.example\.com$ [NC]
RewriteRule (.*) http://www.example.com$1 [R,L]

How to redirect all web requests on port 80 (or HTTP) to port 443 (HTTPS)

RewriteCond %{SERVER_PORT} !^443$
RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [L,R]

How to disable TRACE and TRACK methods on Apache

RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

How to exclude mod_status from being rewritten by existing rules (placed before the problem rule)

RewriteCond %{REQUEST_URI} !=/server-status

How do I redirect all web requests on www.mysite.net/web to www.mysite.net/sect1/web

RewriteCond %{http_host} ^[www\.]*example\.com
RewriteRule ^web(/)?$ /sect1/web [R=301,L]

Rewrite all non-www to www

RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

Force all URLs to be lowercase

RewriteEngine On
RewriteMap lc int:tolower
RewriteCond %{REQUEST_URI} [A-Z]
RewriteRule (.*) ${lc:$1} [R=301,L]

My notepad

Chris' IT notes…