Tag Archives: monitoring

Email notification for successful SSH connection

If you manage a remote server, and you are a bit paranoiac about the bad guys outside, it could be nice to have some sort of notification every time a SSH connection is successful.

I found this post and it seems working pretty well for me as well.
I’ve installed this on my CentOS7 server and seems working good! Of course, this in addition to an aggressive Fail2Ban setup.

Make sure you have your MTA (Postfix/Sendmail…) configured to deliver emails to the user root
Make sure you get the emails for the user root (otherwise doesn’t make any sense 😛 )

Create this script (this is a slightly modified version comparing with the one in the original post:

#!/bin/sh
if [ "$PAM_TYPE" != "open_session" ]
then
  exit 0
else
  {
    echo "User: $PAM_USER"
    echo "Remote Host: $PAM_RHOST"
    echo "Service: $PAM_SERVICE"
    echo "TTY: $PAM_TTY"
    echo "Date: `date`"
    echo "Server: `uname -a`"
  } | mail -s "$PAM_SERVICE login on `hostname -s` from user $PAM_USER@$PAM_RHOST" root
fi
exit 0

#!/bin/sh

if [ "$PAM_TYPE" != "open_session" ]

then

exit 0

else

{

echo "User: $PAM_USER"

echo "Remote Host: $PAM_RHOST"

echo "Service: $PAM_SERVICE"

echo "TTY: $PAM_TTY"

echo "Date: `date`"

echo "Server: `uname -a`"

} | mail -s "$PAM_SERVICE login on `hostname -s` from user $PAM_USER@$PAM_RHOST" root

exit 0

Set the permission:

Shell

chmod +x /usr/local/bin/send-mail-on-ssh-login.sh

1

chmod +x /usr/local/bin/send-mail-on-ssh-login.sh
Append this line to /etc/pam.d/sshd

Shell

session optional pam_exec.so /usr/local/bin/send-mail-on-ssh-login.sh

1

session optional pam_exec.so /usr/local/bin/send-mail-on-ssh-login.sh
…and that’s it! 😉

If you’d like to have a specific domain/IP whitelisted, for example if you don’t want to get notified when you connect from your office or your home (fixed IP or dynamic IP is required), you can use this version of the script:

#!/bin/bash
if [ "$PAM_TYPE" != "open_session" ]; then
  exit 0
else
  MSG="$PAM_SERVICE login on `hostname -s` from user $PAM_USER@$PAM_RHOST"
  # check if the PAM_RHOST is shown as IP
  echo "$PAM_RHOST" | grep -q -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
  if [ $? -eq 0 ]; then
    SRCIP=$PAM_RHOST
  else
    SRCIP=$(dig +short $PAM_RHOST)
  fi
  SAFEIP=$(dig +short myofficedomain.com)
  if [ "$SRCIP" == "$SAFEIP" ]; then
    echo "Authorised $MSG" | logger
  else
  {
    echo "User: $PAM_USER"
    echo "Remote Host: $PAM_RHOST"
    echo "Service: $PAM_SERVICE"
    echo "TTY: $PAM_TTY"
    echo "Date: `date`"
    echo "Server: `uname -a`"
  } | mail -s "Unexpected $MSG" root
  fi
fi
exit 0

#!/bin/bash

if [ "$PAM_TYPE" != "open_session" ]; then

exit 0

else

MSG="$PAM_SERVICE login on `hostname -s` from user $PAM_USER@$PAM_RHOST"

# check if the PAM_RHOST is shown as IP

echo "$PAM_RHOST" | grep -q -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

if [ $? -eq 0 ]; then

SRCIP=$PAM_RHOST

else

SRCIP=$(dig +short $PAM_RHOST)

SAFEIP=$(dig +short myofficedomain.com)

if [ "$SRCIP" == "$SAFEIP" ]; then

echo "Authorised $MSG" | logger

else

{

echo "User: $PAM_USER"

echo "Remote Host: $PAM_RHOST"

echo "Service: $PAM_SERVICE"

echo "TTY: $PAM_TTY"

echo "Date: `date`"

echo "Server: `uname -a`"

} | mail -s "Unexpected $MSG" root

exit 0

The script will send an email ONLY if the source IP is not the one from myofficedomain.com; however, it will log the authentication in /var/log/messages using logger command.

Lsync monitoring on Rackspace Cloud

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins/
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.githubusercontent.com/racker/rackspace-monitoring-agent-plugins-contrib/master/lsyncd-status.sh
chmod 755 lsyncd-status.sh

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins/

cd /usr/lib/rackspace-monitoring-agent/plugins/

wget https://raw.githubusercontent.com/racker/rackspace-monitoring-agent-plugins-contrib/master/lsyncd-status.sh

chmod 755 lsyncd-status.sh

You can test the above script by calling it directly to see if it is working and reporting stats:

/usr/lib/rackspace-monitoring-agent/plugins/lsyncd-status.sh

1	/usr/lib/rackspace-monitoring-agent/plugins/lsyncd-status.sh

Now, we need to create the alert itself.
[To get the token, you can use this]

curl -i -X POST \
-H 'X-Auth-Token: [AUTH_TOKEN]' \
-H 'Content-Type: application/json; charset=UTF-8' \
-H 'Accept: application/json' \
--data-binary \
'{"label": "Lsyncd", "type": "agent.plugin", "details": {"file": "lsyncd-status.sh","args": ["arg1","arg2"]}}' \
'https://monitoring.api.rackspacecloud.com/v1.0/[ACCOUNT_ID]/entities/[ENTITY_ID]/checks'

curl -i -X POST \

-H 'X-Auth-Token: [AUTH_TOKEN]' \

-H 'Content-Type: application/json; charset=UTF-8' \

-H 'Accept: application/json' \

--data-binary \

'{"label": "Lsyncd", "type": "agent.plugin", "details": {"file": "lsyncd-status.sh","args": ["arg1","arg2"]}}' \

'https://monitoring.api.rackspacecloud.com/v1.0/[ACCOUNT_ID]/entities/[ENTITY_ID]/checks'

NOTE: ENTITY_ID is the Monitoring ID, NOT the server ID!!

Once the alert has been created, you can add the alarm manually via the Control Panel:

if (metric['lsyncd_status'] != 'running') {
return new AlarmStatus(CRITICAL, 'Lsyncd Service is NOT running.');
}
if (metric['lsyncd_status'] == 'running' && metric['percent_used_watches'] >= 80) {
return new AlarmStatus(WARNING, 'Lsyncd is running but the number of directories has reached 80% of notify watches.');
}
if (metric['lsyncd_status'] == 'running' && metric['percent_used_watches'] >= 95) {
return new AlarmStatus(CRITICAL, 'Lsyncd is running but the number of directories has reached 95% of notify watches.');
}
return new AlarmStatus(OK, 'Lsyncd Service is running.');

if (metric['lsyncd_status'] != 'running') {

return new AlarmStatus(CRITICAL, 'Lsyncd Service is NOT running.');

}

if (metric['lsyncd_status'] == 'running' && metric['percent_used_watches'] >= 80) {

return new AlarmStatus(WARNING, 'Lsyncd is running but the number of directories has reached 80% of notify watches.');

}

if (metric['lsyncd_status'] == 'running' && metric['percent_used_watches'] >= 95) {

return new AlarmStatus(CRITICAL, 'Lsyncd is running but the number of directories has reached 95% of notify watches.');

}

return new AlarmStatus(OK, 'Lsyncd Service is running.');

Make sure to test and save the alert.

Check Sky Hub speed script

Bash script that extract the router speed.

#!/bin/bash

# Broadband Speed within limits Download 14000 and Upload 750

ROUTER=<router_ip>
USER=<router_username>
PASS=<router_password>
# Download Link speed
DL=14000
# Upload Link speed
UL=750

SPEED=($(wget -qO- http://$ROUTER/sky_router_status.html --user=$USER --password=$PASS | awk -F "'" '$0 ~ /Modemstat\[0\]/  {print $2, $4}'))

if [[ ${SPEED[0]} -lt $DL || ${SPEED[1]} -lt $UL ]] ; then
  exit 2
else
  echo "Broadband Speed within limits Download ${SPEED[0]} and Upload ${SPEED[1]}"
  exit 0
fi

#!/bin/bash

# Broadband Speed within limits Download 14000 and Upload 750

ROUTER=<router_ip>

USER=<router_username>

PASS=<router_password>

# Download Link speed

DL=14000

# Upload Link speed

UL=750

SPEED=($(wget -qO- http://$ROUTER/sky_router_status.html --user=$USER --password=$PASS | awk -F "'" '$0 ~ /Modemstat\[0\]/ {print $2, $4}'))

if [[ ${SPEED[0]} -lt $DL || ${SPEED[1]} -lt $UL ]] ; then

exit 2

else

echo "Broadband Speed within limits Download ${SPEED[0]} and Upload ${SPEED[1]}"

exit 0

This can be integrated in Nagios to send an alert if the speed drops.

Rackspace – Cloud Monitoring – Ansible plugins

Install the required packages (Ubunto/Centos):

apt-get update && apt-get install python-apt python-pip build-essential python-dev git python-virtualenv -y

yum install python-pip git python-devel python-virtualenv gcc -y

apt-get update && apt-get install python-apt python-pip build-essential python-dev git python-virtualenv -y

yum install python-pip git python-devel python-virtualenv gcc -y

Prepare the virtual environment

virtualenv /root/monitorenv
. /root/monitorenv/bin/activate
pip install paramiko PyYAML jinja2 httplib2 ansible

virtualenv /root/monitorenv

. /root/monitorenv/bin/activate

pip install paramiko PyYAML jinja2 httplib2 ansible

Download the playbook

git clone https://github.com/stevekaten/cloud-monitoring-plugin-deploy
cd cloud-monitoring-plugin-deploy

1 2	git clone https://github.com/stevekaten/cloud-monitoring-plugin-deploy cd cloud-monitoring-plugin-deploy

Install the required plugin:

ansible-playbook -i hosts holland_mysqldump.yml

	This will configure the holland_mysqldump plugin on the localhost.

ansible-playbook -i hosts mysql_slave.yml

	This will configure the mysql_slave plugin on the localhost.

ansible-playbook -i hosts port_check.yml

	This will fail with an error message informing you that you need to set a port.

ansible-playbook -i hosts port_check.yml -e port=8080

	This will configure the port_check plugin on the localhost checking if port 8080 is open.

ansible-playbook -i hosts port_check.yml -e '{"host":"rackspace.com","port":"80"}'

	This will configure the port_check plugin to check rackspace.com:80.

ansible-playbook -i hosts port_check.yml -e '{"host":"10.X.X.X","port":"3306"}'

	This will configure the port_check plugin to check mysql's port 3306 on the ServiceNet address.

ansible-playbook -i hosts lsyncd_status.yml

	This will configure the lsyncd_check plugin.

ansible-playbook -i hosts holland_mysqldump.yml

This will configure the holland_mysqldump plugin on the localhost.

ansible-playbook -i hosts mysql_slave.yml

This will configure the mysql_slave plugin on the localhost.

ansible-playbook -i hosts port_check.yml

This will fail with an error message informing you that you need to set a port.

ansible-playbook -i hosts port_check.yml -e port=8080

This will configure the port_check plugin on the localhost checking if port 8080 is open.

ansible-playbook -i hosts port_check.yml -e '{"host":"rackspace.com","port":"80"}'

This will configure the port_check plugin to check rackspace.com:80.

ansible-playbook -i hosts port_check.yml -e '{"host":"10.X.X.X","port":"3306"}'

This will configure the port_check plugin to check mysql's port 3306 on the ServiceNet address.

ansible-playbook -i hosts lsyncd_status.yml

This will configure the lsyncd_check plugin.

To UNINSTALL the monitoring, you need to delete the check, removing the related file from /etc/rackspace-monitoring-agent.conf.d/ and restart the Cloud Monitoring agent.

Nagios3 and Lighttpd

This guide will explain how to install Nagios3 on a machine with Debian and Lighttpd webserver.

If you haven’t installed Lighttpd yet, please follow this tutorial.

Install Nagios server

Now, let’s install Nagios.

apt-get install nagios3 nagios-plugins nagios-nrpe-plugin

1	apt-get install nagios3 nagios-plugins nagios-nrpe-plugin

This will automatically install all the required dependencies.

Enable check_external_commands in /etc/nagios3/nagios.cfg

check_external_commands=1

1	check_external_commands=1

Add www-data in nagios’ group:

usermod -a -G nagios www-data

1	usermod -a -G nagios www-data

And fix some permission issues to avoid some errors like “error: Could not stat() command file”

chmod g+x /var/lib/nagios3/rw

1	chmod g+x /var/lib/nagios3/rw

Let’s configure a bit Lighttpd.
Make sure cgi and php modules are enabled.

Then, create a new conf file and enable it:

vim /etc/lighttpd/conf-available/10-nagios3.conf

1	vim /etc/lighttpd/conf-available/10-nagios3.conf

# Nagios3
 
alias.url =     (
                "/cgi-bin/nagios3" =&gt; "/usr/lib/cgi-bin/nagios3",
                "/nagios3/cgi-bin" =&gt; "/usr/lib/cgi-bin/nagios3",
                "/nagios3/stylesheets" =&gt; "/etc/nagios3/stylesheets",
                "/nagios3" =&gt; "/usr/share/nagios3/htdocs"
                )
 
$HTTP["url"] =~ "^/nagios3/cgi-bin" {
        cgi.assign = ( "" =&gt; "" )
}
 
$HTTP["url"] =~ "nagios" {
        auth.backend = "htpasswd"
        auth.backend.htpasswd.userfile = "/etc/nagios3/htpasswd.users"
        auth.require = ( "" =&gt; (
                "method" =&gt; "basic",
                "realm" =&gt; "nagios",
                "require" =&gt; "user=nagiosadmin"
                )
        )
        setenv.add-environment = ( "REMOTE_USER" =&gt; "user" )
}

# Nagios3

alias.url = (

"/cgi-bin/nagios3" => "/usr/lib/cgi-bin/nagios3",

"/nagios3/cgi-bin" => "/usr/lib/cgi-bin/nagios3",

"/nagios3/stylesheets" => "/etc/nagios3/stylesheets",

"/nagios3" => "/usr/share/nagios3/htdocs"

)

$HTTP["url"] =~ "^/nagios3/cgi-bin" {

cgi.assign = ( "" => "" )

}

$HTTP["url"] =~ "nagios" {

auth.backend = "htpasswd"

auth.backend.htpasswd.userfile = "/etc/nagios3/htpasswd.users"

auth.require = ( "" => (

"method" => "basic",

"realm" => "nagios",

"require" => "user=nagiosadmin"

)

setenv.add-environment = ( "REMOTE_USER" => "user" )

}

lighttpd-enable-mod nagios3

1	lighttpd-enable-mod nagios3

Let’s apply the changes:

/etc/init.d/lighttpd force-reload

1	/etc/init.d/lighttpd force-reload

We need to setup the “nagiosadmin” password:

htpasswd -c /etc/nagios3/htpasswd.users nagiosadmin

1	htpasswd -c /etc/nagios3/htpasswd.users nagiosadmin

Now, open your browser and digit http://yourserver/nagios3
Insert username: nagiosadmin and the password you’ve just chosen… and voila`… 🙂

And now we have installed our nagios server. As you can see, it’s currently monitoring itself.

But what about the other hosts in the network?

Adding hosts

Host configuration

To let our Nagios server to monitor other hosts, we need to follow these steps on any client we want to add:

apt-get install -y nagios-plugins nagios-nrpe-server

1	apt-get install -y nagios-plugins nagios-nrpe-server

Once completed, we need to add the IP of our monitoring host in /etc/nagios/nrpe.cfg under allowed_hosts=xxx.xxx.xxx.xxx.

Also, add this line in /etc/nagios/nrpe_local.cfg:

command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e -A

1	command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e -A

This will be used from our monitor server to query nrpe and provide info about ALL the disks.
You can use also -I flag to exclude a specific path. For example on my Time Capsule Pi, I’ve used the following line, to exclude the mount point “TimeMachine” from the checks:

command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e -A -I '/TimeMachine/*

1	command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e -A -I '/TimeMachine/*

Monitoring configuration for new host

Now back to our Nagios monitoring machine
In /etc/nagios3/conf.d create a file called for example host1_nagios2.cfg and add the following basic services (add/remove/modify based on your local configuration):

define host{
        use             generic-host
        host_name       host1
        alias           host1
        address         xxx.xxx.xxx.xxx
}

define service{
        use                     generic-service
        host_name               host1
        service_description     Current Load
        check_command           check_nrpe_1arg!check_load
}

define service{
        use                     generic-service
        host_name               host1
        service_description     Current Users
        check_command           check_nrpe_1arg!check_users
}
define service{
        use                     generic-service
        host_name               host1
        service_description     Disk Space
        check_command           check_nrpe_1arg!check_all_disks
}
define service{
        use                     generic-service
        host_name               host1
        service_description     Total Processes
        check_command           check_nrpe_1arg!check_total_procs
}

define host{

use generic-host

host_name host1

alias host1

address xxx.xxx.xxx.xxx

}

define service{

use generic-service

host_name host1

service_description Current Load

check_command check_nrpe_1arg!check_load

}

define service{

use generic-service

host_name host1

service_description Current Users

check_command check_nrpe_1arg!check_users

}

define service{

use generic-service

host_name host1

service_description Disk Space

check_command check_nrpe_1arg!check_all_disks

}

define service{

use generic-service

host_name host1

service_description Total Processes

check_command check_nrpe_1arg!check_total_procs

}

Also, you can add the new host host1 to be part of any related groups, modifying /etc/nagios3/conf.d/hostgroups_nagios2.cfg

For example, we can add it to debian-servers and ssh-servers groups. This will automatically get some checks like SSH.

# Some generic hostgroup definitions

# A simple wildcard hostgroup
define hostgroup
        hostgroup_name  all
		alias           All Servers
		members         *
        }

# A list of your Debian GNU/Linux servers
define hostgroup {
        hostgroup_name  debian-servers
		alias           Debian GNU/Linux Servers
		members         localhost,host1
        }

# A list of your web servers
define hostgroup {
        hostgroup_name  http-servers
		alias           HTTP servers
		members         localhost
        }

# A list of your ssh-accessible servers
define hostgroup {
        hostgroup_name  ssh-servers
		alias           SSH servers
		members         localhost,host1
        }

# Some generic hostgroup definitions

# A simple wildcard hostgroup

define hostgroup

hostgroup_name all

alias All Servers

members *

}

# A list of your Debian GNU/Linux servers

define hostgroup {

hostgroup_name debian-servers

alias Debian GNU/Linux Servers

members localhost,host1

}

# A list of your web servers

define hostgroup {

hostgroup_name http-servers

alias HTTP servers

members localhost

}

# A list of your ssh-accessible servers

define hostgroup {

hostgroup_name ssh-servers

alias SSH servers

members localhost,host1

}

Sources:
http://zeldor.biz/2010/11/nagios3-with-lighttpd/comment-page-1/
https://www.digitalocean.com/community/articles/how-to-install-nagios-on-ubuntu-12-10
http://cloud101.eu/blog/2012/03/01/setting-up-nagios-on-debian-or-ubuntu/
http://technosophos.com/2010/01/13/nagios-fixing-error-could-not-stat-command-file-debian.html

My notepad

Chris' IT notes…

Tag Archives: monitoring

Email notification for successful SSH connection

Lsync monitoring on Rackspace Cloud

Check Sky Hub speed script

Rackspace – Cloud Monitoring – Ansible plugins

Nagios3 and Lighttpd

Install Nagios server

Adding hosts

Host configuration

Monitoring configuration for new host