Tag Archives: basic

Varnish – basic notes 

ACLs: /etc/varnish/default.vcl

Memory usage:
grep VARNISH_STORAGE_SIZE /etc/sysconfig/varnish

Check how much memory can use: (check last parameter in the output line)

# ps aux | grep varnish
root     27093  0.0  0.1 112304  1140 ?        Ss   16:28   0:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,256M
varnish  27094  0.1  0.9 21760528 9240 ?       Sl   16:28   0:02 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,256M


>> Test VCL
# varnishd -C -f /etc/varnish/default.vcl


>> Test if varnish works
# varnishstat

 

Sophos antivirus notes

Generic checks

ps aux |grep sav (check process)

/opt/sophos-av/bin/savdstatus --version (version, last update, thread data)
/opt/sophos-av/bin/savconfig -v (info about exclusions, where the Datacentre is that hosts that Sophos device, named scans etc )
/opt/sophos-av/bin/savconfig get TalpaOperations (check disabled mode)

cat /proc/sys/talpa/intercept-filters/VettingController/ops (check all modes)
/opt/sophos-av/bin/savconfig set TalpaOperations -- -open (set mode to disabled for open/read)
/opt/sophos-av/bin/savconfig get TalpaOperations

cat /proc/sys/talpa/intercept-filters/VettingController/ops
-open
+close
+exec
+mount
+umount

/opt/sophos-av/bin/savconfig query NamedScans (Check Scheduled Scans)
/opt/sophos-av/bin/savconfig query NamedScans SEC:FullSystemScan (Check Scheduled Scans with argument)
/opt/sophos-av/bin/savconfig add ExcludeFilePaths /home/user1/ (ADD Exclude files' path)
/opt/sophos-av/bin/savconfig remove ExcludeFilePaths /home/user1/ (REMOVE Exclude files' path)


# Check Global exclusions 
/opt/sophos-av/bin/savconfig query ExcludeFileOnGlob && /opt/sophos-av/bin/savconfig query ExcludeFilePaths

/opt/sophos-av/bin/savdctl disable (disable on-access scanning)
/opt/sophos-av/bin/savdstatus (check)
Sophos Anti-Virus is active but on-access scanning is not running

To get ON-Access Scanning back, restart all Sophos related services:
for i in `chkconfig --list |grep sav |awk '{print $1}'`;do echo -e "\n\e[93mShow service $i restart \e[0m\n";service $i restart;done

Scan

>> Perform the scan -> this will create a log
savscan -nc -f -s --no-follow-symlinks --backtrack-protection --quarantine <path/to/scan> (manual scan)

>> Than, check the log to see what it has been found from the manual scan
/opt/sophos-av/bin/savlog --today --utc | grep detected (check threats for today -)
grep INFECTED /opt/sophos-av/log/savd.log | grep -P -o '(?<=arg>)/[^<]*(?=</arg)' | sort -u (check  all threats)
savscan --help

Example for multiple folders with final report:

(suggested to run in a screen session)

  1. Create a temporary folder: 
    mkdir -p /tmp/scantmp/ > && cd $_
  2. list all directories that you want to scan (full path) into a file called list_folder.txt within the temp folder;
  3. Run the following: 
    for i in `cat list_folder.txt` ; do nice / renice -n 19 savscan -nc -f -s --no-follow-symlinks --backtrack-protection --quarantine $i 2>&1 >> scan.log ; done
/opt/sophos-av/bin/savlog --today --utc | grep "Threat detected" | awk -F" " '{print $2}' > report.txt
  4. Check report.txt 

 

RPM – Yum notes 

>> update all, including what is set as exclude in yum.conf
# yum check-update --disableexcludes=all

>> update only the security related patches the server
# yum update --security

>> how to rollback a recent package
# rpm -Uvh --rollback '1 hour ago'

>> check if a packaged is patched against a particular CVE with:
rpm -q --changelog {package-name} | grep CVE-NUMBER 

>> Check if a package is from repository or not
Example: httpd

1) Get the PID of httpd
# netstat -tnlp | grep httpd
tcp        0      0 :::80                       :::*                        LISTEN      7568/httpd          
tcp        0      0 :::443                      :::*                        LISTEN      7568/httpd        

# lsof -p 7568 | less
(and find what's the "bin" for httpd, in this case is /usr/sbin/httpd )

# rpm -qf /usr/sbin/httpd
httpd-2.2.15-30.el6_5.x86_64

Verified: package part of RH repositories

 

Plesk notes

 

>> Get FTP passwords
# mysql psa -e "select sys_users.login,sys_users.home,domains.name,accounts.password from sys_users,domains,accounts,hosting where sys_users.id=hosting.sys_user_id AND domains.id=hosting.dom_id AND accounts.id=sys_users.account_id"


>> Get email passwords
# /usr/local/psa/admin/sbin/mail_auth_view/usr/local/psa/bin/admin --show-password <----- Plesk 10 and up
cat /etc/psa/.psa.shadow <----- Plesk 6 and up


>> Check which MTA
# alternatives --display mta


>> check mailq (yum install pfHandle)
# pfHandle -s

!!! if you use qmail -> qmHandle


>> Check list of messages queued
# pfHandle -d

!!! If pfHandle does not work, just check inside /var/spool/postfix/



>> Connect to MySQL
mysql -uadmin -p`cat /etc/psa/.psa.shadow`


>> Check version
# cat /usr/local/psa/version 


>> Setup Holland
backupsets/default.conf

[mysql:client]
user = admin
password = file:/etc/psa/.psa.shadow 


>> Check license
/usr/bin/curl -s -k https://127.0.0.1:8443/enterprise/control/agent.php -H "HTTP_AUTH_LOGIN: admin" -H "HTTP_AUTH_PASSWD: `/usr/local/psa/bin/admin --show-password`" -H "HTTP_PRETTY_PRINT: true" -H "Content-Type: text/xml" -d "<packet> <server> <get> <key/> </get> </server> </packet>" | egrep -ohm 1 "PLSK\.[0-9]{8}"


>> Remove license (physically from the server)
[root@344668-web1 ~]# mv /etc/sw/keys/keys/keyXXNb8YmF  /home/user/
[root@344668-web1 ~]#


>> Plesk main logs
MAIL: /usr/local/psa/var/log/maillog
ACCESS LOGS: /var/www/vhosts/*/logs/access_log



>> One-liner to generate reports from the Access Logs

> General report
grep -h "04.Jun.2015" /var/www/vhosts/*/logs/access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 20

> per site report
for i in `find /var/www/vhosts/*/logs/access_log -not -empty `;do echo -n "$i - " ; awk '{print $1}' $i | sort | uniq -c | sort -n | tail -1 ; done | sort –k3 -n | column –t




>> Add custom configuration to Apache under Plesk

# cd /var/www/vhosts/system/DOMAIN.com/conf        
If there is no vhost.conf file then I can create it and add the necessary custom configuration

Need to reconfigure the Plesk Domain - this will Include the custome vhost.conf file
# /usr/local/psa/admin/sbin/httpdmng  -h
# /usr/local/psa/admin/sbin/httpdmng --reconfigure-domain DOMAIN.com



>> Disable SSLv3 on Plesk

If you need to disable SSLv3 on Plesk boxes, here is how to do it:

If nginx is running on port 443, use the following KB: http://kb.sp.parallels.com/en/120083
If Apache is configured on port 443, create /etc/httpd/conf.d/ zz050-psa-disable-weak-ssl-ciphers.conf:

SSLHonorCipherOrder on
SSLProtocol -ALL +TLSv1
SSLCipherSuite ALL:!ADH:!LOW:!SSLv2:!EXP:+HIGH:+MEDIUM


# /usr/local/psa/bin/ipmanage -l
State Type IP                               Clients Hosting PublicIP 
1     S    eth0:172.54.10.212/255.255.252.0 0       0                
0     E    eth2:10.0.1.128/255.255.254.0 0       0                
0     S    eth0:172.54.10.27/255.255.252.0  0       161              
0     E    eth0:172.54.10.28/255.255.252.0  0       1  

# /usr/local/psa/bin/ipmanage -r 172.54.10.212
Error occured while sending feedback. HTTP code returned: 502
SUCCESS: Removal of IP '172.54.10.212' completed.

# /usr/local/psa/bin/ipmanage -l
State Type IP                               Clients Hosting PublicIP 
0     E    eth2:10.0.1.128/255.255.254.0 0       0                
0     S    eth0:172.54.10.27/255.255.252.0  0       161              
0     E    eth0:172.54.10.28/255.255.252.0  0       1

 

What to do with a down Magento site

1. Application level logs – First place to look.

If you are seeing the very-default-looking Magento page saying “There has been an error processing your request”, then look in here:

ls -lart <DOCROOT>/var/report/ | tail

The stack trace will be in the latest file (there might be a lot), and should highlight what broke.
Maybe the error was in a database library, or a Redis library…see next step if that’s the case.

General errors, often non-fatal, are in <DOCROOT>/var/log/exception.log
Other module-specific logs will be in the same log/ directory, for example SagePay.

NB: check /tmp/magento/var/ .
If the directories in the DocumentRoot are not writable (or weren’t in the past), Magento will use /tmp/magento/var and you’ll find the logs/reports/cache in there.

2. Backend services – Magento fails hard if something is inaccessible

First, find the local.xml. It should be under <DOCROOT>/app/etc/local.xml or possibly a subdirectory like <DOCROOT>/store/app/etc/local.xml

From that, take note of the database credentials, the <session_save>, and the <cache><backend>. If there’s no <cache> section, then you are using filesystem so it won’t be memcache or redis.

– Can you connect to that database from this server? authenticate? or is it at max-connections?
– To test memcache, “telnet host 11211” and type “STATS“.
– To test Redis, “telnet host 6379” and type “INFO”.
You could also use:

redis-cli -s /tmp/redis.sock -a PasswordIfThereIsOne info

 

If you can’t connect to those from the web server, check that the relevant services are started, pay close attention to the port numbers, and make sure any firewalls allow the connection.
If the memcache/redis info shows evictions > 0, then it’s probably filled up at some point and restarting that service might get you out of the water.

ls -la /etc/init.d/mem* /etc/init.d/redis*

3. Check the normal places – sometimes it’s nothing to do with Magento!

  • – PHP-FPM logs – good place for PHP fatal errors. usually in var/log/php[5]-fpm/

– Apache or nginx logs
– Is Apache just at MaxClients?
– PHP-FPM max_children?

ps aux | grep fpm | grep -v root | awk '{print $11, $12, $13}' | sort | uniq -c

– Is your error really just a timeout, because the server’s too busy?
– Did OOM-killer break something?

grep oom /var/log/messages /var/log/kern.log

– Has a developer been caught out by apc.stat=0 (or opcache.validate_timestamp=0) ?

 

Credits: https://willparsons.tech/

Linux resource checks notes 

top

1 -> CUP utilisation
> -> order by memory

check for CPU Performance
Optimized waiting%
(press '1' to see all the CPUs)

-----------------

I/O - check for %util
iostat -kx 1 1000

-----------------

Memory
free -m
free -m | grep "buffers/cache" | awk '{print $3}'

atop utility

 

 

Atop – notes 

atop -a | Display only active processes
atop -g | Display general process info
atop -m | Display memory usage info
atop -n | Display network usage info
atop -d | Display Dick usage info

Alternatively you can just use atop and then key in the letters above to switch between.

atop -r | read raw data.

Use this to basically start looking at the processes from the start of the day 00:00

atop -r -b 09:00    | read raw data from 09:00 today
atop -r y           | read raw data from yesterday
atop -r yy          | read raw data from the day before yesterday
atop -r y -b 09:00  | read raw data from 09:00 yesterday
atop -r yy -b 09:00 | read raw data from 09:00 the day before yesterday
atop -r <log>       | read data from a log stored in /var/log/atop

 

Fail2ban notes

General notes about Fail2ban

### Fail2Ban ###

Best practise:
- do NOT edit /etc/fail2ban/jail.conf BUT create a new /etc/fail2ban/jail.local file

=============================================================
# Test fail2ban regex:
example: fail2ban-regex /var/log/secure /etc/fail2ban/filter.d/sshd.conf
example2: fail2ban-regex --print-all-matched/var/log/secure /etc/fail2ban/filter.d/sshd.conf

=============================================================
# Remove email notifications:

comment out 'sendmail-whois' from action in [ssh-iptables]
FYI, comment with # at the BEGINNING of the line like this or it won't work!!!

[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
#           sendmail-whois[name=SSH, dest=root, [email protected], sendername="Fail2Ban"]
logpath  = /var/log/secure
maxretry = 5


=============================================================
# Wordpress wp-login - block POST attacks

/etc/fail2ban/jail.local

[apache-wp-login]
enabled = true
port = http,https
filter = apache-wp-login
logpath = /var/log/httpd/blog.tian.it-access.log
maxretry = 3
bantime = 604800 ; 1 week
findtime = 120

----------------------------------------------------------------------------------------------------------------------

/etc/fail2ban/filter.d/apache-wp-login.conf
[Definition]
failregex = <HOST>.*POST.*wp-login.php HTTP/1.1
ignoreregex =

=============================================================

# Manually ban an IP:
fail2ban-client -vvv set <CHAIN> banip <IP>

# Check status of sshd chain
fail2ban-client status sshd

How to “SSH” brute force

If you want to make safer your remote server, it is good practise to use a good combination of sshd setup and fail2ban.

Firstly, you should setup your server to allow only key auth, and no passwords. This will drastically reduce the risk. This means anyway that you need to keep your ssh key safe and you won’t be able to access your server unless you have this key. Most of the time is something possible 🙂

For this reason, I’m explaining here how I configured my server.

SSHD

/etc/ssh/sshd_config

Have these settings in the config file (NOTE: the verbosity is for Fail2ban)

LogLevel VERBOSE

PasswordAuthentication no

(restart sshd)

/etc/fail2ban/jail.local

[DEFAULT]
# Ban hosts for 
# one hour:
#bantime = 3600

# one day:
bantime = 86400

# A host is banned if it has generated "maxretry" during the last "findtime"
# # seconds.
findtime  = 30

# # "maxretry" is the number of failures before a host get banned.
maxretry = 5

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true
filter = sshd-aggressive
port     = ssh
logpath  = /var/log/secure
maxretry = 3
findtime = 30
bantime  = 86400

/etc/fail2ban/filter.d/sshd.conf

Add a custom section after the ddos one:

custom = ^%(__prefix_line_sl)sDisconnected from <HOST> port [0-9]+ \[preauth\]$

This line matches whoever tries to connect without a proper ssh key.

Add this line to include custom to the sshd-aggressive setup:

aggressive = %(normal)s
             %(ddos)s
             %(custom)s