Author Archives: thtieig

What to do with a down Magento site

1. Application level logs – First place to look.

If you are seeing the very-default-looking Magento page saying “There has been an error processing your request”, then look in here:

ls -lart <DOCROOT>/var/report/ | tail

The stack trace will be in the latest file (there might be a lot), and should highlight what broke.
Maybe the error was in a database library, or a Redis library…see next step if that’s the case.

General errors, often non-fatal, are in <DOCROOT>/var/log/exception.log
Other module-specific logs will be in the same log/ directory, for example SagePay.

NB: check /tmp/magento/var/ .
If the directories in the DocumentRoot are not writable (or weren’t in the past), Magento will use /tmp/magento/var and you’ll find the logs/reports/cache in there.

2. Backend services – Magento fails hard if something is inaccessible

First, find the local.xml. It should be under <DOCROOT>/app/etc/local.xml or possibly a subdirectory like <DOCROOT>/store/app/etc/local.xml

From that, take note of the database credentials, the <session_save>, and the <cache><backend>. If there’s no <cache> section, then you are using filesystem so it won’t be memcache or redis.

– Can you connect to that database from this server? authenticate? or is it at max-connections?
– To test memcache, “telnet host 11211” and type “STATS“.
– To test Redis, “telnet host 6379” and type “INFO”.
You could also use:

redis-cli -s /tmp/redis.sock -a PasswordIfThereIsOne info

 

If you can’t connect to those from the web server, check that the relevant services are started, pay close attention to the port numbers, and make sure any firewalls allow the connection.
If the memcache/redis info shows evictions > 0, then it’s probably filled up at some point and restarting that service might get you out of the water.

ls -la /etc/init.d/mem* /etc/init.d/redis*

3. Check the normal places – sometimes it’s nothing to do with Magento!

  • – PHP-FPM logs – good place for PHP fatal errors. usually in var/log/php[5]-fpm/

– Apache or nginx logs
– Is Apache just at MaxClients?
– PHP-FPM max_children?

ps aux | grep fpm | grep -v root | awk '{print $11, $12, $13}' | sort | uniq -c

– Is your error really just a timeout, because the server’s too busy?
– Did OOM-killer break something?

grep oom /var/log/messages /var/log/kern.log

– Has a developer been caught out by apc.stat=0 (or opcache.validate_timestamp=0) ?

 

Credits: https://willparsons.tech/

Linux resource checks notes 

top

1 -> CUP utilisation
> -> order by memory

check for CPU Performance
Optimized waiting%
(press '1' to see all the CPUs)

-----------------

I/O - check for %util
iostat -kx 1 1000

-----------------

Memory
free -m
free -m | grep "buffers/cache" | awk '{print $3}'

atop utility

 

 

sudo sudores notes

Add user to a group (sudores)

Enabled wheel group in sudores

## Allows people in group wheel to run all commands
%wheel ALL=(ALL) ALL

Add pippo to group wheel

# usermod -a -G wheel pippo

 

Add user passwordless 

pippo ALL=(ALL)       NOPASSWD: ALL

…and NO TTY

pippo ALL=(ALL)       NOPASSWD: ALL
Defaults:pippo !requiretty

 

How to chown and setuid

Something to watch out for.

If you change the owner of a file that has setuid permission, the setuid bit gets removed:

# ls -l /usr/bin/gpasswd
-rwsr-xr-x. 1 root root 78144 Mar 19 2013 /usr/bin/gpasswd 
# chown nobody /usr/bin/gpasswd 
# ls -l /usr/bin/gpasswd 
-rwxr-xr-x. 1 nobody root 78144 Mar 19 2013 /usr/bin/gpasswd

Therefore if you reset the owner of such a file (for instance, after an accidental recursive chown), then you need to reset the permissions afterwards.
If you reset both the owner AND the permissions, it has to be done in the correct order – ownership first, then permissions.

So what will this do?

# rpm --setperms --setugids

 

It turns out (on RHEL5 at any rate) that it changes the permissions first. So the binaries that ought to have the setuid bit were left without it.

The moral of this story is that you should do the work in two passes:

# rpm --setugids

# rpm --setperms

 

Atop – notes 

atop -a | Display only active processes
atop -g | Display general process info
atop -m | Display memory usage info
atop -n | Display network usage info
atop -d | Display Dick usage info

Alternatively you can just use atop and then key in the letters above to switch between.

atop -r | read raw data.

Use this to basically start looking at the processes from the start of the day 00:00

atop -r -b 09:00    | read raw data from 09:00 today
atop -r y           | read raw data from yesterday
atop -r yy          | read raw data from the day before yesterday
atop -r y -b 09:00  | read raw data from 09:00 yesterday
atop -r yy -b 09:00 | read raw data from 09:00 the day before yesterday
atop -r <log>       | read data from a log stored in /var/log/atop

 

Fail2ban notes

General notes about Fail2ban

### Fail2Ban ###

Best practise:
- do NOT edit /etc/fail2ban/jail.conf BUT create a new /etc/fail2ban/jail.local file

=============================================================
# Test fail2ban regex:
example: fail2ban-regex /var/log/secure /etc/fail2ban/filter.d/sshd.conf
example2: fail2ban-regex --print-all-matched/var/log/secure /etc/fail2ban/filter.d/sshd.conf

=============================================================
# Remove email notifications:

comment out 'sendmail-whois' from action in [ssh-iptables]
FYI, comment with # at the BEGINNING of the line like this or it won't work!!!

[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
#           sendmail-whois[name=SSH, dest=root, [email protected], sendername="Fail2Ban"]
logpath  = /var/log/secure
maxretry = 5


=============================================================
# Wordpress wp-login - block POST attacks

/etc/fail2ban/jail.local

[apache-wp-login]
enabled = true
port = http,https
filter = apache-wp-login
logpath = /var/log/httpd/blog.tian.it-access.log
maxretry = 3
bantime = 604800 ; 1 week
findtime = 120

----------------------------------------------------------------------------------------------------------------------

/etc/fail2ban/filter.d/apache-wp-login.conf
[Definition]
failregex = <HOST>.*POST.*wp-login.php HTTP/1.1
ignoreregex =

=============================================================

# Manually ban an IP:
fail2ban-client -vvv set <CHAIN> banip <IP>

# Check status of sshd chain
fail2ban-client status sshd

How to “SSH” brute force

If you want to make safer your remote server, it is good practise to use a good combination of sshd setup and fail2ban.

Firstly, you should setup your server to allow only key auth, and no passwords. This will drastically reduce the risk. This means anyway that you need to keep your ssh key safe and you won’t be able to access your server unless you have this key. Most of the time is something possible 🙂

For this reason, I’m explaining here how I configured my server.

SSHD

/etc/ssh/sshd_config

Have these settings in the config file (NOTE: the verbosity is for Fail2ban)

LogLevel VERBOSE

PasswordAuthentication no

(restart sshd)

/etc/fail2ban/jail.local

[DEFAULT]
# Ban hosts for 
# one hour:
#bantime = 3600

# one day:
bantime = 86400

# A host is banned if it has generated "maxretry" during the last "findtime"
# # seconds.
findtime  = 30

# # "maxretry" is the number of failures before a host get banned.
maxretry = 5

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true
filter = sshd-aggressive
port     = ssh
logpath  = /var/log/secure
maxretry = 3
findtime = 30
bantime  = 86400

/etc/fail2ban/filter.d/sshd.conf

Add a custom section after the ddos one:

custom = ^%(__prefix_line_sl)sDisconnected from <HOST> port [0-9]+ \[preauth\]$

This line matches whoever tries to connect without a proper ssh key.

Add this line to include custom to the sshd-aggressive setup:

aggressive = %(normal)s
             %(ddos)s
             %(custom)s