Tag Archives: memory

Which PHP script is eating my memory?

For mod_php, you can just add this to your Apache LogFormat:

%{mod_php_memory_usage}n

Might be best to add it right at the end, so as not to break any log parsing.
Credit: http://tech.superhappykittymeow.com/?p=220

For PHP-FPM (i.e. anyone with NginX or anyone with one of our optimised Magento setups.), put the following into your FPM pool config file, probably here:
– /etc/php-fpm.d/website.conf (RHEL/CentOS)
– /etc/php5-fpm/pools.d/website.conf (Ubuntu)

access.log = /var/log/php-fpm/domain.com-access.log
access.format = "%p %{HTTP_X_FORWARDED_FOR}e - %u %t \"%m %{REQUEST_URI}e\" %s %f %{mili}d %{kilo}M %C%% \"%{HTTP_USER_AGENT}e\""

 
Notes:
• Permissions matter. Check that the User and/or Group (of THIS fpm pool) can write to /var/log/php-fpm (or /php5-fpm, whatever)
• %{HTTP_X_FORWARDED_FOR}e is there because I was behind a Load Balancer, Varnish and/or other reverse proxy.
• %{REQUEST_URI}e\ is there because this CMS (like most, now), rewrite everything to index.php. I want to know the original request, not just the script name.
• %{kilo}M %C – These are the kickers: Memory usage and CPU PerformanceOptimized usage per request. Ker-pow. Pick your favourite awk/sort one-liner to weed out the heavy hitters.

You can log pretty much anything: any arbitrary header, (like REQUEST_URI), and you should find all the options in the comments under the default “www.conf” packaged config file.
Given that this starts with the PID, it should also help track down any segfaults you see in /var/log/messages / kern.log.
Oh, and while you’re at it, PHP-FPM has a slow log you can enable.
Oh, and don’t forget your old friend logrotate.

Counting the average PHP-FPM process memory usage

Looking at the ‘ps’ output isn’t always accurate because of shared memory. Here’s a one-liner to count up my FPM pool, where “www” is in name of the FPM pool:

for pid in $(ps aux | grep fpm | grep "pool www" | awk '{print $2}'); do pmap -d $pid | tail -1 ; done | sed 's/K//' | awk '{sum+=$4} END {print sum/NR/1024}’

Thanks to Dan Farmer for his original Apache memory script, which made me think to do this.

Divide and conquer

Do make sure that multiple websites on the same server are using their own FPM pools, that way it should be much easier to see what’s what.

But you can also separate by URL. I’ve done this with the M-word, but the theory should stand for any CMS where the tasks performed under the admin section will be more intensive than normal front-end user traffic.
The following should work with WordPress, assuming the path is /wp-admin . Magento users can (and should) change their admin path from the default, so watch out for that (usually configured in local.xml)

Or anything really – maybe that “importvideo” script is especially memory-intensive and you don’t want to allow it to bloat up all your processes.

1. Set up a separate PHP-FPM pool.
– give it a different name, different log file names, and listen on a different socket/port.
– Perhaps with a much higher memory_limit
– Perhaps with many fewer pm.max_children (ask customer how many humans actually use the “backend”). You might only need 5 or so.
– Perhaps it needs a longer max_execution_time …. you get the idea.

2. Send your “admin” traffic there.
Here’s how you might go about it. These are excerpts and ideas, rather than solid copy-paste config.

Nginx: something like this:

upstream backend {
server unix:/var/run/php5-domain.com.sock;
}
upstream backend-admin {
server unix:/var/run/php5-domain.com-admin.sock;
}

map "URL:$request_uri." $fcgi_pass {
default backend;
~URL:.*admin.* backend-admin;
}

… then in your ‘server{}’ bit, use the variable for fastcgi_pass:

fastcgi_pass $fcgi_pass;

Apache: something like this…

# Normal backend alias, corresponds with FastCGIExgternalServer 
Alias /php.fcgi /dev/shm/domain-php.fcgi
<Location ~ admin>
# Override Action for “admin” URLs
Action application/x-httpd-php /domain-admin.fcgi
</Location>
Alias /domain-admin.fcgi /dev/shm/domain-admin-php.fcgi

 

3. You can probably now LOWER the memory_limit for your “main” pool
– if the rest of the website doesn’t use much memory. Now bask in the memory you just saved for the whole application server.

4. Bonus: NewRelic app separation
Don’t let all that heavy Admin work interfere with your nice / renice appdex statistics – we know (and expect) your backend dashboard to be slower.
Put this in the FPM pool config:

php_value[newrelic.appname] = "www.domain.com Admin"

 

Credits: https://willparsons.tech/

MySQL – How much memory for InnoDB buffer?

Use this to find out how much storage each engine uses:

mysql -e "select engine, SUM(DATA_LENGTH)/1024 as data_size, SUM(INDEX_LENGTH)/1024 as index_size, ((sum(DATA_LENGTH)+sum(INDEX_LENGTH))/1024) as total_size from information_schema.tables group by engine;"
+--------------------+-----------+------------+------------+
| engine | data_size | index_size | total_size |
+--------------------+-----------+------------+------------+
| CSV | 0.0000 | 0.0000 | 0.0000 |
| InnoDB | 480.0000 | 0.0000 | 480.0000 |
| MEMORY | 0.0000 | 0.0000 | 0.0000 |
| MyISAM | 0.0000 | 1.0000 | 1220.5859 |
| PERFORMANCE_SCHEMA | 0.0000 | 0.0000 | 0.0000 |
+--------------------+-----------+------------+------------+

From the data above, it may seem like a good idea to keep innodb_buffer_pool low, because we have so much MyISAM data.
BUT if I get one query a minute against that MyISAM data, and 500 queries per second against the considerably smaller InnoDB data-set, that doesn’t make sense anymore.
Then we’d want all of the InnoDB data in memory, hence a buffer pool of 512M+

It’s down to data usage and where the hotspots are!


Percona Toolkit to ‘get the hotspots’

wget http://www.percona.com/downloads/percona-toolkit/2.2.12/tarball/percona-toolkit-2.2.12.tar.gz
tar -zxvf percona-toolkit-2.2.12.tar.gz -C /opt/ 
cd /opt/percona-toolkit-2.2.12/ 
perl Makefile.PL 
make

Get the hotspots:
you can set NUMPACKETS or keep the tcpdump running for 20 minutes. JUST keep an eye on the file size!

NUMPACKETS=1000
tcpdump -s 65535 -x -nn -q -tttt -i any -c $NUMPACKETS port 3306 > /tmp/mysql.tcp.txt 

bin/pt-query-digest --type tcpdump /tmp/mysql.tcp.txt

FYI: https://bugs.launchpad.net/percona-toolkit/+bug/1402776  official bug for that database being wrong thing