Author Archives: thtieig

PHP-FPM pool sizes explanation

If you see these settings in a PHP-FPM config. file (real life example):

pm.max_children = 100
pm.start_servers = 30
pm.min_spare_servers = 30
pm.max_spare_servers = 100

What does it actually mean?

1) There will never be less than 30 processes (because it starts with 30, and minimum spare is 30)

2) There will always be at least 30 idle processes, except when more than 70 are in use.

3) Idle processes will never be killed off, unless they hit pm.max_requests (because the number of processes cannot possibly exceed the maximum spare)

Too many idle processes is bad, because they use up virtual memory. It is especially bad on a server with no swap, because then they can’t be shunted out of RAM to make room for active processes.
Also, remember that the settings are per-pool. On this particular server, there were three pools with identical settings. The result was that there were never less than 90 PHP processes, even if all were idle.

So please be mindful of the law of unintended consequences, especially when creating additional PHP FPM pools.

 

Credits to my ex colleague Danny 🙂

Set X-Forward in Nginx logs

http://wp.sgrosshome.com/2014/03/03/how-to-configure-nginx-logs-to-show-real-ip-when-behind-a-reverse-proxy/

log_format LB_log '$remote_addr forwarded for $http_x_realip - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
access_log /path/to/you/log/directory/your-log-file-name.log LB_log;

 

After that, Nginx needs to be restarted/reloaded, of course 😉

InnoDB log file size

Question 1: Is my innodb_log_buffer_size too small?

Answer:
Since MySQL 5.0 there is a status called Innodb_log_waits. This status shows the number of times that the log buffer was too small. A wait is required for it to be flushed before continuing.

SHOW GLOBAL STATUS 
LIKE 'innodb_log_waits';

+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| Innodb_log_waits | 0 |
+------------------+-------+

 

If this value is 0 or near innodb_log_buffer_size is defined well. If it is high and continuously growing, increase it or reduce the size of your transactions.

Question 2: What would be a better value?
Answer: https://www.percona.com/blog/2008/11/21/how-to-calculate-a-good-innodb-log-file-size/

Postfix – blacklist domain

Few notes about how to block a specific domain to send out emails

To help in cutting down the number of spam mails currently getting through a specific domain

# vi /etc/postfix/blacklisted_domains

# cat /etc/postfix/blacklisted_domains
mydomain.com	REJECT

# postmap /etc/postfix/blacklisted_domains
# postconf -e 'smtpd_recipient_restrictions = check_recipient_access hash:/etc/postfix/blacklisted_domains, permit'

 

Compromised Email troubleshooting notes

Here some notes about how to troubleshoot a server that got compromised by a php script.

Check email queue

  • Qmail -> qmHandle
  • Postfix -> pmHandle / postqueue
# qmHandle -s
Total messages: 7357
Messages with local recipients: 0
Messages with remote recipients: 7357
Messages with bounces: 0
Messages in preprocess: 0

Get some email IDs

# qmHandle -l | head
1348989 (16, 16/1348989)
Return-path: #@[]
From: [email protected]
To: [email protected]
Subject: failure notice
Date: 30 Jun 2015 07:42:59 +0100
Size: 5093 bytes
less
42240113 (15, 15/42240113)
Return-path: [email protected]

Check for X-PHP header in the mail message
Look for the UID and script that sent the message

# qmHandle -m1348989 | grep X-PHP
X-PHP-Originating-Script: 48:wp-content.php(1) : eval()'d code

Find the script and UID

# grep 48 /etc/passwd => this was Apache ==> this means that the code was injected via Apache

=> permissions issue??

# locate wp-content.php
/var/www/vhosts/example.com/wp-content.php

Move away the file(s) and chown 000
!! if the file starts with – , you need to user chown — 000 filename

Disable execution php following this how to

Delete all the messages containing that header

# qmHandle -h'X-PHP-Originating-Script: 48:wp-content.php'
Calling system script to terminate qmail...
Stopping : Looking for messages with headers matching X-PHP-Originating-Script: 48:wp-content.php
Message 1345933 slotted for deletion.
Message 42240608 slotted for deletion.
Message 1346796 slotted for deletion.
Message 42240391 slotted for deletion.
Message 42241954 slotted for deletion.
[...]
Deleted 113 messages from queue
Restarting qmail... Starting qmail: [ OK ]
done (hopefully).

Extra notes:

Check the queue:

postqueue -p

See the content of a message:

postcat -q <ID from postqueue output>

Check for “X-PHP-Originating-Script” header, which generally gives you the name of the script that generate the email

If they are sent to a specific domain, you can block some domains in Postfix following this guide

Which PHP script is eating my memory?

For mod_php, you can just add this to your Apache LogFormat:

%{mod_php_memory_usage}n

Might be best to add it right at the end, so as not to break any log parsing.
Credit: http://tech.superhappykittymeow.com/?p=220

For PHP-FPM (i.e. anyone with NginX or anyone with one of our optimised Magento setups.), put the following into your FPM pool config file, probably here:
– /etc/php-fpm.d/website.conf (RHEL/CentOS)
– /etc/php5-fpm/pools.d/website.conf (Ubuntu)

access.log = /var/log/php-fpm/domain.com-access.log
access.format = "%p %{HTTP_X_FORWARDED_FOR}e - %u %t \"%m %{REQUEST_URI}e\" %s %f %{mili}d %{kilo}M %C%% \"%{HTTP_USER_AGENT}e\""

 
Notes:
• Permissions matter. Check that the User and/or Group (of THIS fpm pool) can write to /var/log/php-fpm (or /php5-fpm, whatever)
• %{HTTP_X_FORWARDED_FOR}e is there because I was behind a Load Balancer, Varnish and/or other reverse proxy.
• %{REQUEST_URI}e\ is there because this CMS (like most, now), rewrite everything to index.php. I want to know the original request, not just the script name.
• %{kilo}M %C – These are the kickers: Memory usage and CPU PerformanceOptimized usage per request. Ker-pow. Pick your favourite awk/sort one-liner to weed out the heavy hitters.

You can log pretty much anything: any arbitrary header, (like REQUEST_URI), and you should find all the options in the comments under the default “www.conf” packaged config file.
Given that this starts with the PID, it should also help track down any segfaults you see in /var/log/messages / kern.log.
Oh, and while you’re at it, PHP-FPM has a slow log you can enable.
Oh, and don’t forget your old friend logrotate.

Counting the average PHP-FPM process memory usage

Looking at the ‘ps’ output isn’t always accurate because of shared memory. Here’s a one-liner to count up my FPM pool, where “www” is in name of the FPM pool:

for pid in $(ps aux | grep fpm | grep "pool www" | awk '{print $2}'); do pmap -d $pid | tail -1 ; done | sed 's/K//' | awk '{sum+=$4} END {print sum/NR/1024}’

Thanks to Dan Farmer for his original Apache memory script, which made me think to do this.

Divide and conquer

Do make sure that multiple websites on the same server are using their own FPM pools, that way it should be much easier to see what’s what.

But you can also separate by URL. I’ve done this with the M-word, but the theory should stand for any CMS where the tasks performed under the admin section will be more intensive than normal front-end user traffic.
The following should work with WordPress, assuming the path is /wp-admin . Magento users can (and should) change their admin path from the default, so watch out for that (usually configured in local.xml)

Or anything really – maybe that “importvideo” script is especially memory-intensive and you don’t want to allow it to bloat up all your processes.

1. Set up a separate PHP-FPM pool.
– give it a different name, different log file names, and listen on a different socket/port.
– Perhaps with a much higher memory_limit
– Perhaps with many fewer pm.max_children (ask customer how many humans actually use the “backend”). You might only need 5 or so.
– Perhaps it needs a longer max_execution_time …. you get the idea.

2. Send your “admin” traffic there.
Here’s how you might go about it. These are excerpts and ideas, rather than solid copy-paste config.

Nginx: something like this:

upstream backend {
server unix:/var/run/php5-domain.com.sock;
}
upstream backend-admin {
server unix:/var/run/php5-domain.com-admin.sock;
}

map "URL:$request_uri." $fcgi_pass {
default backend;
~URL:.*admin.* backend-admin;
}

… then in your ‘server{}’ bit, use the variable for fastcgi_pass:

fastcgi_pass $fcgi_pass;

Apache: something like this…

# Normal backend alias, corresponds with FastCGIExgternalServer 
Alias /php.fcgi /dev/shm/domain-php.fcgi
<Location ~ admin>
# Override Action for “admin” URLs
Action application/x-httpd-php /domain-admin.fcgi
</Location>
Alias /domain-admin.fcgi /dev/shm/domain-admin-php.fcgi

 

3. You can probably now LOWER the memory_limit for your “main” pool
– if the rest of the website doesn’t use much memory. Now bask in the memory you just saved for the whole application server.

4. Bonus: NewRelic app separation
Don’t let all that heavy Admin work interfere with your nice / renice appdex statistics – we know (and expect) your backend dashboard to be slower.
Put this in the FPM pool config:

php_value[newrelic.appname] = "www.domain.com Admin"

 

Credits: https://willparsons.tech/