Tag Archives: monitoring

Email notification for successful SSH connection

If you manage a remote server, and you are a bit paranoiac about the bad guys outside, it could be nice to have some sort of notification every time a SSH connection is successful.

I found this post and it seems working pretty well for me as well.
I’ve installed this on my CentOS7 server and seems working good! Of course, this in addition to an aggressive Fail2Ban setup.

  1. Make sure you have your MTA (Postfix/Sendmail…) configured to deliver emails to the user root
  2. Make sure you get the emails for the user root (otherwise doesn’t make any sense 😛 )
  3. Create this script (this is a slightly modified version comparing with the one in the original post: 
    #!/bin/sh
if [ "$PAM_TYPE" != "open_session" ]
then
  exit 0
else
  {
    echo "User: $PAM_USER"
    echo "Remote Host: $PAM_RHOST"
    echo "Service: $PAM_SERVICE"
    echo "TTY: $PAM_TTY"
    echo "Date: `date`"
    echo "Server: `uname -a`"
  } | mail -s "$PAM_SERVICE login on `hostname -s` from user $PAM_USER@$PAM_RHOST" root
fi
exit 0
  4. Set the permission: 
    chmod +x /usr/local/bin/send-mail-on-ssh-login.sh
  5. Append this line to /etc/pam.d/sshd 
    session optional pam_exec.so /usr/local/bin/send-mail-on-ssh-login.sh
  6.  …and that’s it! 😉

 

If you’d like to have a specific domain/IP whitelisted, for example if you don’t want to get notified when you connect from your office or your home (fixed IP or dynamic IP is required), you can use this version of the script:

#!/bin/bash
if [ "$PAM_TYPE" != "open_session" ]; then
  exit 0
else
  MSG="$PAM_SERVICE login on `hostname -s` from user $PAM_USER@$PAM_RHOST"
  # check if the PAM_RHOST is shown as IP
  echo "$PAM_RHOST" | grep -q -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
  if [ $? -eq 0 ]; then
    SRCIP=$PAM_RHOST
  else
    SRCIP=$(dig +short $PAM_RHOST)
  fi
  SAFEIP=$(dig +short myofficedomain.com)
  if [ "$SRCIP" == "$SAFEIP" ]; then
    echo "Authorised $MSG" | logger
  else
  {
    echo "User: $PAM_USER"
    echo "Remote Host: $PAM_RHOST"
    echo "Service: $PAM_SERVICE"
    echo "TTY: $PAM_TTY"
    echo "Date: `date`"
    echo "Server: `uname -a`"
  } | mail -s "Unexpected $MSG" root
  fi
fi
exit 0

The script will send an email ONLY if the source IP is not the one from myofficedomain.com; however, it will log the authentication in /var/log/messages using logger command.

Lsync monitoring on Rackspace Cloud 

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins/
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.githubusercontent.com/racker/rackspace-monitoring-agent-plugins-contrib/master/lsyncd-status.sh
chmod 755 lsyncd-status.sh

You can test the above script by calling it directly to see if it is working and reporting stats:

/usr/lib/rackspace-monitoring-agent/plugins/lsyncd-status.sh

 

Now, we need to create the alert itself.
[To get the token, you can use this]

curl -i -X POST \
-H 'X-Auth-Token: [AUTH_TOKEN]' \
-H 'Content-Type: application/json; charset=UTF-8' \
-H 'Accept: application/json' \
--data-binary \
'{"label": "Lsyncd", "type": "agent.plugin", "details": {"file": "lsyncd-status.sh","args": ["arg1","arg2"]}}' \
'https://monitoring.api.rackspacecloud.com/v1.0/[ACCOUNT_ID]/entities/[ENTITY_ID]/checks'

NOTE: ENTITY_ID is the Monitoring ID, NOT the server ID!!

Once the alert has been created, you can add the alarm manually via the Control Panel:

if (metric['lsyncd_status'] != 'running') {
return new AlarmStatus(CRITICAL, 'Lsyncd Service is NOT running.');
}
if (metric['lsyncd_status'] == 'running' && metric['percent_used_watches'] >= 80) {
return new AlarmStatus(WARNING, 'Lsyncd is running but the number of directories has reached 80% of notify watches.');
}
if (metric['lsyncd_status'] == 'running' && metric['percent_used_watches'] >= 95) {
return new AlarmStatus(CRITICAL, 'Lsyncd is running but the number of directories has reached 95% of notify watches.');
}
return new AlarmStatus(OK, 'Lsyncd Service is running.');

Make sure to test and save the alert.

Check Sky Hub speed script

Bash script that extract the router speed.

#!/bin/bash

# Broadband Speed within limits Download 14000 and Upload 750

ROUTER=<router_ip>
USER=<router_username>
PASS=<router_password>
# Download Link speed
DL=14000
# Upload Link speed
UL=750

SPEED=($(wget -qO- http://$ROUTER/sky_router_status.html --user=$USER --password=$PASS | awk -F "'" '$0 ~ /Modemstat\[0\]/  {print $2, $4}'))

if [[ ${SPEED[0]} -lt $DL || ${SPEED[1]} -lt $UL ]] ; then
  exit 2
else
  echo "Broadband Speed within limits Download ${SPEED[0]} and Upload ${SPEED[1]}"
  exit 0
fi

 

This can be integrated in Nagios to send an alert if the speed drops.

Rackspace – Cloud Monitoring – Ansible plugins

Install the required packages (Ubunto/Centos):

apt-get update && apt-get install python-apt python-pip build-essential python-dev git python-virtualenv -y

yum install python-pip git python-devel python-virtualenv gcc -y

Prepare the virtual environment

virtualenv /root/monitorenv
. /root/monitorenv/bin/activate
pip install paramiko PyYAML jinja2 httplib2 ansible

Download the playbook

git clone https://github.com/stevekaten/cloud-monitoring-plugin-deploy
cd cloud-monitoring-plugin-deploy

Install the required plugin:

ansible-playbook -i hosts holland_mysqldump.yml

	This will configure the holland_mysqldump plugin on the localhost.

ansible-playbook -i hosts mysql_slave.yml

	This will configure the mysql_slave plugin on the localhost.

ansible-playbook -i hosts port_check.yml

	This will fail with an error message informing you that you need to set a port.

ansible-playbook -i hosts port_check.yml -e port=8080

	This will configure the port_check plugin on the localhost checking if port 8080 is open.

ansible-playbook -i hosts port_check.yml -e '{"host":"rackspace.com","port":"80"}'

	This will configure the port_check plugin to check rackspace.com:80.

ansible-playbook -i hosts port_check.yml -e '{"host":"10.X.X.X","port":"3306"}'

	This will configure the port_check plugin to check mysql's port 3306 on the ServiceNet address.

ansible-playbook -i hosts lsyncd_status.yml

	This will configure the lsyncd_check plugin.

To UNINSTALL the monitoring, you need to delete the check, removing the related file from /etc/rackspace-monitoring-agent.conf.d/ and restart the Cloud Monitoring agent.

Nagios3 and Lighttpd

This guide will explain how to install Nagios3 on a machine with Debian and Lighttpd webserver.

If you haven’t installed Lighttpd yet, please follow this tutorial.

Install Nagios server

Now, let’s install Nagios.

apt-get install nagios3 nagios-plugins nagios-nrpe-plugin

This will automatically install all the required dependencies.

Enable check_external_commands in /etc/nagios3/nagios.cfg

check_external_commands=1

Add www-data in nagios’ group:

usermod -a -G nagios www-data

And fix some permission issues to avoid some errors like “error: Could not stat() command file”

chmod g+x /var/lib/nagios3/rw

Let’s configure a bit Lighttpd.
Make sure cgi and php modules are enabled.

Then, create a new conf file and enable it:

vim /etc/lighttpd/conf-available/10-nagios3.conf
# Nagios3
 
alias.url =     (
                "/cgi-bin/nagios3" => "/usr/lib/cgi-bin/nagios3",
                "/nagios3/cgi-bin" => "/usr/lib/cgi-bin/nagios3",
                "/nagios3/stylesheets" => "/etc/nagios3/stylesheets",
                "/nagios3" => "/usr/share/nagios3/htdocs"
                )
 
$HTTP["url"] =~ "^/nagios3/cgi-bin" {
        cgi.assign = ( "" => "" )
}
 
$HTTP["url"] =~ "nagios" {
        auth.backend = "htpasswd"
        auth.backend.htpasswd.userfile = "/etc/nagios3/htpasswd.users"
        auth.require = ( "" => (
                "method" => "basic",
                "realm" => "nagios",
                "require" => "user=nagiosadmin"
                )
        )
        setenv.add-environment = ( "REMOTE_USER" => "user" )
}
lighttpd-enable-mod nagios3

Let’s apply the changes:

/etc/init.d/lighttpd force-reload

We need to setup the “nagiosadmin” password:

htpasswd -c /etc/nagios3/htpasswd.users nagiosadmin

Now, open your browser and digit http://yourserver/nagios3
Insert username: nagiosadmin and the password you’ve just chosen… and voila`… 🙂

And now we have installed our nagios server. As you can see, it’s currently monitoring itself.

But what about the other hosts in the network?

Adding hosts

Host configuration

To let our Nagios server to monitor other hosts, we need to follow these steps on any client we want to add:

apt-get install -y nagios-plugins nagios-nrpe-server

Once completed, we need to add the IP of our monitoring host in /etc/nagios/nrpe.cfg under allowed_hosts=xxx.xxx.xxx.xxx.

Also, add this line in /etc/nagios/nrpe_local.cfg:

command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e -A

This will be used from our monitor server to query nrpe and provide info about ALL the disks.
You can use also -I flag to exclude a specific path. For example on my Time Capsule Pi, I’ve used the following line, to exclude the mount point “TimeMachine” from the checks:

command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e -A -I '/TimeMachine/*

Monitoring configuration for new host

Now back to our Nagios monitoring machine
In /etc/nagios3/conf.d create a file called for example host1_nagios2.cfg and add the following basic services (add/remove/modify based on your local configuration):

define host{
        use             generic-host
        host_name       host1
        alias           host1
        address         xxx.xxx.xxx.xxx
}

define service{
        use                     generic-service
        host_name               host1
        service_description     Current Load
        check_command           check_nrpe_1arg!check_load
}

define service{
        use                     generic-service
        host_name               host1
        service_description     Current Users
        check_command           check_nrpe_1arg!check_users
}
define service{
        use                     generic-service
        host_name               host1
        service_description     Disk Space
        check_command           check_nrpe_1arg!check_all_disks
}
define service{
        use                     generic-service
        host_name               host1
        service_description     Total Processes
        check_command           check_nrpe_1arg!check_total_procs
}

Also, you can add the new host host1 to be part of any related groups, modifying /etc/nagios3/conf.d/hostgroups_nagios2.cfg

For example, we can add it to debian-servers and ssh-servers groups. This will automatically get some checks like SSH.

# Some generic hostgroup definitions

# A simple wildcard hostgroup
define hostgroup
        hostgroup_name  all
		alias           All Servers
		members         *
        }

# A list of your Debian GNU/Linux servers
define hostgroup {
        hostgroup_name  debian-servers
		alias           Debian GNU/Linux Servers
		members         localhost,host1
        }

# A list of your web servers
define hostgroup {
        hostgroup_name  http-servers
		alias           HTTP servers
		members         localhost
        }

# A list of your ssh-accessible servers
define hostgroup {
        hostgroup_name  ssh-servers
		alias           SSH servers
		members         localhost,host1
        }

Sources:
http://zeldor.biz/2010/11/nagios3-with-lighttpd/comment-page-1/
https://www.digitalocean.com/community/articles/how-to-install-nagios-on-ubuntu-12-10
http://cloud101.eu/blog/2012/03/01/setting-up-nagios-on-debian-or-ubuntu/
http://technosophos.com/2010/01/13/nagios-fixing-error-could-not-stat-command-file-debian.html