Author Archives: thtieig

Dynamic MOTD on Centos7

Just few steps!

Install figlet package:

yum install figlet

 

Create /etc/motd.sh script with this content:

#!/bin/sh
#
clear
figlet -f slant $(hostnamectl --pretty)
printf "\n"
printf "\t- %s\n\t- Kernel %s\n" "$(cat /etc/redhat-release)" "$(uname -r)"
printf "\n"



date=`date`
load=`cat /proc/loadavg | awk '{print $1}'`
root_usage=`df -h / | awk '/\// {print $(NF-1)}'`
memory_usage=`free -m | awk '/Mem:/ { total=$2 } /buffers\/cache/ { used=$3 } END { printf("%3.1f%%", used/total*100)}'`
swap_usage=`free -m | awk '/Swap/ { printf("%3.1f%%", "exit !$2;$3/$2*100") }'`
users=`users | wc -w`
time=`uptime | grep -ohe 'up .*' | sed 's/,/\ hours/g' | awk '{ printf $2" "$3 }'`
processes=`ps aux | wc -l`
ethup=$(ip -4 ad | grep 'state UP' | awk -F ":" '!/^[0-9]*: ?lo/ {print $2}')
ip=$(ip ad show dev $ethup |grep -v inet6 | grep inet|awk '{print $2}')

echo "System information as of: $date"
echo
printf "System load:\t%s\tIP Address:\t%s\n" $load $ip
printf "Memory usage:\t%s\tSystem uptime:\t%s\n" $memory_usage "$time"
printf "Usage on /:\t%s\tSwap usage:\t%s\n" $root_usage $swap_usage
printf "Local Users:\t%s\tProcesses:\t%s\n" $users $processes
echo

[ -f /etc/motd.tail ] && cat /etc/motd.tail || true

Make the script executable:

chmod +x /etc/motd.sh

Append this script to /etc/profile in order to be executed as last command once a user logs in:

echo "/etc/motd.sh" >> /etc/profile

Try and have fun! 🙂

 

If you are using Debian, here the other guide.

LVM – How to

Intro

LVM is a very powerful technology, and can really help the Sysadmin’s life.
However, this is something that we generally setup at the beginning (most of the time now it’s automatically setup during the installation process), and it’s well know… when we stop using something, we tend to forget how to use it.

This is why I’m writing this how to, mostly to keep track of the major features and commands, in case I will need them again in the future 😉

Before proceeding, please digest the following journey of this poor physical device that gets abstracted up to usable pieces.

                                                          VG                        VG
                                                   +---------------+         +---------------+
                                                   |      PV       |  +--->  |               |
                                   PV              | +-----------+ |         |  LV           |
                              +-----------+        | |  8E LVM   | |         |               |
              8E LVM          |  8E LVM   |        | | +-------+ | |         +---------------+
             +-------+        | +-------+ |        | | | +---+ | | |  +--->  +---------------+
+---+        | +---+ |        | | +---+ | |        | | | |DEV| | | |         |               |
|DEV| +----> | |DEV| | +----> | | |DEV| | | +----> | | | +---+ | | |         |  LV           |
+---+        | +---+ |        | | +---+ | |        | | +-------+ | |         |               |
  1.         +-------+        | +-------+ |        | +-----------+ |  +--->  |               |
                2.            +-----------+        |               |         +---------------+
                                   3.              |      PV       |         +---------------+
                                                   | +-----------+ |         |               |
 1. Original Device:                               | |  8E LVM   | |  +--->  |  LV           |
    (physical/virtual disk/partition/raid)         | | +-------+ | |         |               |
 2. fdisk'd to label 8E for LVM                    | | | +---+ | | |         |               |
 3. initialised as LVM Physical Volume             | | | |DEV| | | |         |               |
 4. Added in a LVM Volume Group                    | | | +---+ | | |  +--->  |               |
 5. "partitioned" in single/multiple               | | +-------+ | |         |               |
    LVM Logical Groups                          4. | +-----------+ |      5. |               |
                                                   +---------------+         +---------------+

 

 

Prepare partions

First of all, we need to find which device(s) we want to setup for LVM

fdisk -l

[root@n1 ~]# fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b7f16

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *        2048    41943039    20970496   83  Linux

Disk /dev/md1: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md2: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md3: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

We can see 3 md devices, probably RAID devices. These are the ones that we are going to use for our LVM exercise.

Now, let’s create an LVM partition.

fdisk <device> => n , p , 1 , (enter) , (enter) , t , 8e , w

[root@n1 ~]# fdisk /dev/md1
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x50b03cd2.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-9759231, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-9759231, default 9759231): 
Using default value 9759231
Partition 1 of type Linux and of size 4.7 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Do the same for all the devices that you want to use for LVM. In my example, I’ve done this for /dev/md1, /dev/md2 and /dev/md3.

Shortcut (risky but quicker) 🙂

echo -e "o\nn\np\n1\n\n\nt\n8e\nw" | fdisk /dev/mdx

All seems now good to go: we have Linux LVM partitions!

Disk /dev/md1: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x50b03cd2

    Device Boot      Start         End      Blocks   Id  System
/dev/md1p1            2048     9759231     4878592   8e  Linux LVM

Disk /dev/md2: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4b68e9a3

    Device Boot      Start         End      Blocks   Id  System
/dev/md2p1            2048     9759231     4878592   8e  Linux LVM

Disk /dev/md3: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8dcf9ba0

    Device Boot      Start         End      Blocks   Id  System
/dev/md3p1            2048     9759231     4878592   8e  Linux LVM

Time to start to configure LVM

Configure LVM

First of all, we need to make these Linux LVM partition able to be part of a group (vg). I always find tricky to remember the logic behind. Let’s imagine that the device itself now is just labelled “Linux LVM” but we need to initiate it in somehow.

pvcreate <dev>

[root@n1 ~]# pvcreate /dev/md1p1
  Physical volume "/dev/md1p1" successfully created.
[root@n1 ~]# pvcreate /dev/md2p1
  Physical volume "/dev/md2p1" successfully created.
[root@n1 ~]# pvcreate /dev/md3p1
  Physical volume "/dev/md3p1" successfully created.

Now these guys are ready to be part of a group. In this case a Virtual Group (vg).
Let’s check that it’s actually true:
pvs

[root@n1 ~]# pvs
  PV         VG Fmt  Attr PSize PFree
  /dev/md1p1    lvm2 ---  4.65g 4.65g
  /dev/md2p1    lvm2 ---  4.65g 4.65g
  /dev/md3p1    lvm2 ---  4.65g 4.65g

Time to create a group with these devices (this could be done also with just a single device):

vgcreate <lvmgroupname> <dev> <dev> …

[root@n1 ~]# vgcreate mylvmvg /dev/md1p1 /dev/md2p1 /dev/md3p1
  Volume group "mylvmvg" successfully created

Now, let’s check again with pvs and vgs

[root@n1 ~]# pvs
  PV         VG      Fmt  Attr PSize PFree
  /dev/md1p1 mylvmvg lvm2 a--  4.65g 4.65g
  /dev/md2p1 mylvmvg lvm2 a--  4.65g 4.65g
  /dev/md3p1 mylvmvg lvm2 a--  4.65g 4.65g
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   0   0 wz--n- 13.95g 13.95g

Now pvs shows the VG group no longer empty but with mylvmvg. And vgs tells us that the VG is about 14GB in size, fully free with no LV in it.

Good! Now, let’s make some LVs (logical volumes). These will be the new “partitions/disks” that we will be actually able to format, mount and use! 🙂

lvcreate -n <name> -L xGB <vg_group_name>

[root@n1 ~]# lvcreate -n part1 -L 2GB mylvmvg
  Logical volume "part1" created.

Some checks to verify:

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   1   0 wz--n- 13.95g 11.95g
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-a----- 2.00g

A new LV appears in vgs and lvs shows the 2GB volume that we have created.

Let’s create another one, but this time, using the full remaining space (using -l 100%VG option instead of -L xGB)

[root@n1 ~]# lvcreate -n part2 -l 100%VG mylvmvg
  Logical volume "part2" created.
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g    0 
[root@n1 ~]# lvs
  LV    VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-a-----  2.00g                                                    
  part2 mylvmvg -wi-a----- 11.95g                                                    
[root@n1 ~]#

Magic!

Now, we have two devices, both ‘a’ -> active and ready to be formatted:
mkfs.ext4 <device>

[root@n1 ~]# mkfs.ext4 /dev/mylvmvg/part1 
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

I’ve done this for /dev/mylvmvg/part1 and /dev/mylvmvg/part2.

Let’s create the mount points and mount them:

[root@n1 ~]# mkdir -p /mountpoint1 /mountpoint2

[root@n1 ~]# mount -t ext4 /dev/mylvmvg/part1 /mountpoint1

[root@n1 ~]# mount -t ext4 /dev/mylvmvg/part2 /mountpoint2

[root@n1 ~]# mount | grep mylvmvg
/dev/mapper/mylvmvg-part1 on /mountpoint1 type ext4 (rw,relatime,data=ordered)
/dev/mapper/mylvmvg-part2 on /mountpoint2 type ext4 (rw,relatime,data=ordered)

[root@n1 ~]# df -Th | grep mapper
/dev/mapper/mylvmvg-part1 ext4      2.0G  6.0M  1.8G   1% /mountpoint1
/dev/mapper/mylvmvg-part2 ext4       12G   41M   11G   1% /mountpoint2

As you can see, the devices are appearing now as /dev/mapper/mylvmvg-partX. You can use either /dev/mylvmvg/partX or /dev/mapper/mylvmvg-partX. Theoretically, the mapper one is recommended (my bad!).

Now the 2 devices are ready to be used as a typical disk/partition formatted with ext4 filesystem.

Resize Logical Volume

Now, imagine that part1 is too small, and you need more space. And luckily, your part2 volume has plenty. Is there any way to “steal” some space from part2 and give it to part1?
Ooohh yesss! 🙂

How?

  1. shrink part2 logical volume AND its filesystem
  2. expand part1 logical volume AND its filesystem

Here the comments inline:

# Important the -r (this RESIZE the filesystem during the process)
[root@n1 ~]# lvreduce -L -5GB -r /dev/mylvmvg/part2 
Do you want to unmount "/mountpoint2"? [Y|n] y
fsck from util-linux 2.23.2
/dev/mapper/mylvmvg-part2: 12/783360 files (0.0% non-contiguous), 92221/3131392 blocks
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/mapper/mylvmvg-part2 to 1820672 (4k) blocks.
The filesystem on /dev/mapper/mylvmvg-part2 is now 1820672 blocks long.

  Size of logical volume mylvmvg/part2 changed from 11.95 GiB (3058 extents) to 6.95 GiB (1778 extents).
  Logical volume mylvmvg/part2 successfully resized.

# Here we can see that part2 is now smaller than before
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-ao---- 2.00g                                                    
  part2 mylvmvg -wi-ao---- 6.95g                                                    

# And here we can see 5GB available in the vg
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g 5.00g

# We assign the 5GB available to part1
[root@n1 ~]# lvextend -L +5GB -r /dev/mylvmvg/part1
  Size of logical volume mylvmvg/part1 changed from 2.00 GiB (512 extents) to 7.00 GiB (1792 extents).
  Logical volume mylvmvg/part1 successfully resized.
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/mylvmvg-part1 is mounted on /mountpoint1; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/mylvmvg-part1 is now 1835008 blocks long.

# No more Free space
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g    0 

# part1 is now 7GB (prev 2GB)
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-ao---- 7.00g                                                    
  part2 mylvmvg -wi-ao---- 6.95g                                                    

# df shows as well the new size
[root@n1 ~]# df -Th | grep mapper
/dev/mapper/mylvmvg-part1 ext4      6.9G  9.1M  6.6G   1% /mountpoint1
/dev/mapper/mylvmvg-part2 ext4      6.8G   37M  6.4G   1% /mountpoint2

 

Move logical volume onto a new RAID array

Now, let’s imagine that one of the 3 initial md devices are having problems, or simply we want to move on a faster/bigger raid array.
The magic of LVM is that we can actually do this with NO DOWNTIME!

How?

In this example we assume that a new /dev/md10 device is attached to our server and we want to remove /dev/md2 device.

  1. We need to take the new device and go through all the previous steps:
    1. fdisk
    2. pvcreate
  2. After that, we need to add this initialised device in the existing volume group (vg)
  3. Move whatever is stored on the physical device
  4. Shrink the volume group
  5. Remove the device
[root@n1 ~]# echo -e "o\nn\np\n1\n\n\nt\n8e\nw" | fdisk /dev/md10
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x465fad01.

Command (m for help): Building a new DOS disklabel with disk identifier 0x5aa41f03.

Command (m for help): Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): Partition number (1-4, default 1): First sector (2048-104791935, default 2048): Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-104791935, default 104791935): Using default value 104791935
Partition 1 of type Linux and of size 50 GiB is set

Command (m for help): Selected partition 1
Hex code (type L to list all codes): Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[root@n1 ~]# fdisk -l | grep md10
Disk /dev/md10: 53.7 GB, 53653471232 bytes, 104791936 sectors
/dev/md10p1            2048   104791935    52394944   8e  Linux LVM

[root@n1 ~]# pvcreate /dev/md10p1
  Physical volume "/dev/md10p1" successfully created.

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1         lvm2 ---  49.97g 49.97g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

[root@n1 ~]# vgextend mylvmvg /dev/md10p1
  Volume group "mylvmvg" successfully extended

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 49.96g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   4   2   0 wz--n- 63.91g 49.96g

Now where the new bits are starting:
pvmove, vgreduce, pvremove

[root@n1 ~]# pvmove /dev/md2p1
  /dev/md2p1: Moved: 0.00%
  /dev/md2p1: Moved: 5.63%
  /dev/md2p1: Moved: 11.51%
  ...
  /dev/md2p1: Moved: 92.61%
  /dev/md2p1: Moved: 98.07%
  /dev/md2p1: Moved: 100.00%

# Here we can see 4 phisical volumes, and a size of ~64GB
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   4   2   0 wz--n- 63.91g 49.96g

# We can see also that /dev/md2p1 is now fully FREE
[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g  4.65g
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

# we can safely remove this device from the vg
[root@n1 ~]# vgreduce mylvmvg /dev/md2p1
  Removed "/dev/md2p1" from volume group "mylvmvg"

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   2   0 wz--n- 59.26g 45.32g

#/dev/md2p1 doesn't belong to any VG anymore
[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1          lvm2 ---   4.65g  4.65g
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

# Removing and confirm: no more /dev/md2p1
[root@n1 ~]# pvremove /dev/md2p1
  Labels on physical volume "/dev/md2p1" successfully wiped.

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0

 

In this example we have left LVM to decide where to put the data that was stored on /dev/md2 device.
Just for reference, we could have specified the destination physical device (e.g. if we were thinking to remove more devices and make sure that the data was ending up on the new RAID and not sprat across the other disks):

pvmove /dev/md2p1 /dev/md10p1

Or, in case we just wanted to move a specific logical volume, let’s say part1

pvmove -n part1 /dev/md2p1 /dev/md10p1

 

…happy LVM’ing! 😉

Vim – remove the yellow highlight

Be honest. Sometimes happened also to you to see a file that you edited a while ago with Vim, still showing that terribly annoying yellow highlight. And I’m sure you probably gave up, thinking that the time was going to remove it.

Wrong! 😛

Here how to get rid of it:

  1. open the file
  2. press ESC
  3. type :nohl
  4. press Enter

Alternatively, the way I kept to use (which seems easier to remember), is actually search for some crazy string.
For example:

  1. open the file
  2. press ESC
  3. type /fkjsaddflkjasd;flka
    (randomly type stuffs)
  4. press Enter

Done 😉

Vim – Comment multi lines

First way:

v -> select whatever needs to be commented out
:
(appears this) :'<,’>
add s/^/#
(it will be now :'<,’>s/^/#)
Press Enter

Second way:

For commenting out a block of text, do the following:

  1. hit CTRL + v (visual block mode)
  2. use the down arrow keys to select the lines you want (it won’t highlight everything)
  3. Shift + i (capital I)
  4. insert the caracter/text you want to add at the beginning of the line (e.g. #)
  5. Press ESC.

 

To uncomment a block:

  1. Go the to first line of code where you want to start uncommenting from.
  2. Press 0 (To bring the cursor to the beginning of the line.)
  3. CTRL + v and select lines to be uncommented.
  4. x that will delete all the # characters vertically.

 

Source: https://www.quora.com/How-can-I-un-comment-a-block-of-text-in-Vim

phpMyAdmin under Varnish

Add the below in the Varnish VCL file, under sub vcl_recv

if (req.url ~ "(phpMyAdmin|phpmyadmin|phpMyAdmin/setup)") {
return (pass);
}

Than test VCL

# varnishd -C -f /etc/varnish/default.vcl

…and if all is good

# service varnish reload

NOTE: sometimes, altering phpMyAdmin auth method in /etc/phpMyAdmin/config.inc.php from cookie to http

Varnish – basic notes 

ACLs: /etc/varnish/default.vcl

Memory usage:
grep VARNISH_STORAGE_SIZE /etc/sysconfig/varnish

Check how much memory can use: (check last parameter in the output line)

# ps aux | grep varnish
root     27093  0.0  0.1 112304  1140 ?        Ss   16:28   0:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,256M
varnish  27094  0.1  0.9 21760528 9240 ?       Sl   16:28   0:02 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,256M


>> Test VCL
# varnishd -C -f /etc/varnish/default.vcl


>> Test if varnish works
# varnishstat

 

Sophos antivirus notes

Generic checks

ps aux |grep sav (check process)

/opt/sophos-av/bin/savdstatus --version (version, last update, thread data)
/opt/sophos-av/bin/savconfig -v (info about exclusions, where the Datacentre is that hosts that Sophos device, named scans etc )
/opt/sophos-av/bin/savconfig get TalpaOperations (check disabled mode)

cat /proc/sys/talpa/intercept-filters/VettingController/ops (check all modes)
/opt/sophos-av/bin/savconfig set TalpaOperations -- -open (set mode to disabled for open/read)
/opt/sophos-av/bin/savconfig get TalpaOperations

cat /proc/sys/talpa/intercept-filters/VettingController/ops
-open
+close
+exec
+mount
+umount

/opt/sophos-av/bin/savconfig query NamedScans (Check Scheduled Scans)
/opt/sophos-av/bin/savconfig query NamedScans SEC:FullSystemScan (Check Scheduled Scans with argument)
/opt/sophos-av/bin/savconfig add ExcludeFilePaths /home/user1/ (ADD Exclude files' path)
/opt/sophos-av/bin/savconfig remove ExcludeFilePaths /home/user1/ (REMOVE Exclude files' path)


# Check Global exclusions 
/opt/sophos-av/bin/savconfig query ExcludeFileOnGlob && /opt/sophos-av/bin/savconfig query ExcludeFilePaths

/opt/sophos-av/bin/savdctl disable (disable on-access scanning)
/opt/sophos-av/bin/savdstatus (check)
Sophos Anti-Virus is active but on-access scanning is not running

To get ON-Access Scanning back, restart all Sophos related services:
for i in `chkconfig --list |grep sav |awk '{print $1}'`;do echo -e "\n\e[93mShow service $i restart \e[0m\n";service $i restart;done

Scan

>> Perform the scan -> this will create a log
savscan -nc -f -s --no-follow-symlinks --backtrack-protection --quarantine <path/to/scan> (manual scan)

>> Than, check the log to see what it has been found from the manual scan
/opt/sophos-av/bin/savlog --today --utc | grep detected (check threats for today -)
grep INFECTED /opt/sophos-av/log/savd.log | grep -P -o '(?<=arg>)/[^<]*(?=</arg)' | sort -u (check  all threats)
savscan --help

Example for multiple folders with final report:

(suggested to run in a screen session)

  1. Create a temporary folder: 
    mkdir -p /tmp/scantmp/ > && cd $_
  2. list all directories that you want to scan (full path) into a file called list_folder.txt within the temp folder;
  3. Run the following: 
    for i in `cat list_folder.txt` ; do nice / renice -n 19 savscan -nc -f -s --no-follow-symlinks --backtrack-protection --quarantine $i 2>&1 >> scan.log ; done
/opt/sophos-av/bin/savlog --today --utc | grep "Threat detected" | awk -F" " '{print $2}' > report.txt
  4. Check report.txt 

 

RPM – Yum notes 

>> update all, including what is set as exclude in yum.conf
# yum check-update --disableexcludes=all

>> update only the security related patches the server
# yum update --security

>> how to rollback a recent package
# rpm -Uvh --rollback '1 hour ago'

>> check if a packaged is patched against a particular CVE with:
rpm -q --changelog {package-name} | grep CVE-NUMBER 

>> Check if a package is from repository or not
Example: httpd

1) Get the PID of httpd
# netstat -tnlp | grep httpd
tcp        0      0 :::80                       :::*                        LISTEN      7568/httpd          
tcp        0      0 :::443                      :::*                        LISTEN      7568/httpd        

# lsof -p 7568 | less
(and find what's the "bin" for httpd, in this case is /usr/sbin/httpd )

# rpm -qf /usr/sbin/httpd
httpd-2.2.15-30.el6_5.x86_64

Verified: package part of RH repositories

 

Develop your site using WordPress and publish a static version on Cloud Files

WordPress is a CMS widely used nowadays to develop sites. Even if it wasn’t created with this intent (blogging was actually the original reason), due to its simplicity and popularity, loads of people are using this software to build their websites.

When the end goal is no longer a blog but a pure website, and you’re not using any ‘search’ or ‘comment’ features which still require php functions, it could be worth considering a completely “static’ed” site which is then published in a Cloud Files Public Container. You could take advantage of the built-in CDN capability and forget about scaling on demand and all the limitations that WordPress has in the Cloud.
Having a static site as opposed to a PHP site means you can bypass security patch installations and avoid the possible risk of compromised servers.
And last but not least: cost reduction!
When you serve a static site from Cloud Files you pay only for the space utilized and the bandwidth – no more charges for Cloud Server(s) and better performance! So… why not give it a try?! 🙂

Side note: this concept can be applied to any CMS and any Cloud Platform. In this post I will use WordPress and Rackspace Public Cloud as examples.

Here is a list of user cases:

  • Existing website built on WordPress; without search fields, comments, forms (dynamic content) which is still being updated on regular basis with extra pages.
  • Brand new project based on WordPress where the end goal is to use WordPress as a Content Management System (CMS) to build a site, without any comment or search fields.
  • Old website built on an old version of WordPress that cannot be updated due to some plugins that won’t work on newest versions of this CMS, with no dynamic functionalities required.
  • Legacy site built on WordPress that is no longer being developed/updated but is still required to stay online, with no dynamic functionalities required.

To be able to make a site build using a CMS completely static, any dynamic functionality needs to be disabled (e.g. comments, forms, search fields…).
It’s possible to manually re-integrate some of them relying on trusted external sources (e.g. Facebook comments) with the introduction of some javascript. But this will require some editing of the static pages generated, something that is outside the scope of this article. However I wanted to mention this, as it could be a limitation in the decision to make the site static.

What do you need?

  • a Cloud Load Balancer: this is just to keep a static IP
  • a small Cloud Server LAMP stack (1/2GB max) where you will host your site
  • a Cloud Files Public container to host the static version of the site
  • access to your DNS
  • some basic Linux knowledge to install some packages and run a few commands via command line

How to set this up?

Imagine you have www.mywpsite.com built on WordPress.
It’s a site that you keep updating generally once or twice a week.
Comments are all disabled and you don’t have any search fields.
You found this post and you’re interested in trying to save some money, improve performance and forget about patching and updating your CMS.

So, how can you achieve this?

You will have to build a development infrastructure made up of a Cloud Load Balancer and a single Linux LAMP Cloud Server plus a Cloud Files Public Container for your production static site.

Your main domain www.mywpsite.com will need to point to the CNAME of the Cloud Files Container and no longer to your original site/server. You also need to create a new subdomain (e.g. admin.mywpsite.com) and point it to the development infrastructure: in specific, to the Cloud Load Balancer’s IP. You will then use admin.mywpsite.com/wp-admin/ instead of www.mywpsite.com/wp-admin/ to manage your site.

The only reason to use a Load Balancer in a single server’s solution is to keep the public IP, meaning you will no longer need to touch the DNS.
The goal is to image the server after you’ve completed making the changes and then delete it. In this way you won’t get charged for a resource that doesn’t serve any traffic and is there just for development. In this example, I’ve mentioned that changes are made a couple of times a week, so it’s cheaper and easier to always keep a Load Balancer with admin.mywpsite.com pointing to it, instead of a Cloud Server where you would be changing the DNS every time.
If you’re planning to keep the dev server up continuously, you could avoid using a Load Balancer entirely. However Cloud Servers might fail (we all know that Cloud is not designed to be resilient) so you may need to spin up a new server and have the DNS re-pointed again. In this case I do recommend keeping the DNS TTL as low as possible (e.g. 300 seconds).

Please note that the Site URL in your WordPress setup needs to be changed to admin.mywpsite.com. If you’re starting a new project you need to remember that the Site URL has to match the development subdomain that you have chosen. In our example, to admin.mywpsite.com.

Once you are happy with the site and everything works under admin.mywpsite.com, you will just need to run a command to make a static version of it, and another command to push the content onto the Cloud Files Container.

Once complete and verified, you can change the DNS for www.mywpsite.com to point to the Container, then image your dev server and delete it once the image has completed. You will create a new server from this image the next time you want to make a new change.

Here is a diagram that should help you to visualise the setup:

diagram

diagram

All clear?

So let’s do it!

How do you put this in practice?

Step by step guide (no downtime migration)
  1. Create a Cloud Server LAMP stack.
  2. Create a Cloud Load Balancer. Take note of its new IP.
  3. Add the server under the Load Balancer.
  4. Create a Cloud Files Public Container. Take note of its HTTP Public Link.
  5. Create the subdomain admin.mywpsite.com in your DNS and point it to the IP of your Cloud Load Balancer. Keep your www.mywpsite.com domain pointing to your live site to avoid downtime.
  6. Migrate WordPress files and database onto the new server and make sure NO changes on the live site will be done during the migration to avoid inconsistency of data.
  7. Change the Site URL to admin.mywpsite.com. Please note that this can be a bit tricky. Make sure all the references are properly updated. If it’s a new project, just install WordPress on it and set the Site URL directly to admin.mywpsite.com.
  8. Verify that you can access your site using admin.mywpsite.com and that also the wp-admin panel works correctly. Troubleshoot until this step is fully verified.

    NOTE: At this stage we have a new development site setup and the original live site still serving traffic. Next steps will complete the migration, having live traffic directly served from the Cloud Files Container.

  9. Install httrack and turbolift on your server.
    On a CentOS server you should be able to run the following:

    yum -y install pip httrack
pip install turbolift
  10. [OPTIONAL, but recommended to avoid useless bandwidth charges]
    Manually set /etc/hosts to resolve to admin.mywpsite.com locally, to force httrack to pull the content without going via public net:

    echo "127.0.0.1 admin.mywpsite.com' >> /etc/hosts
  11. Combine these two commands to locally generate a static version of your admin.mywpsite.com and push it to the Cloud Files Container.
    To achieve that, you can use a simple BASH script like the one below:
    Script project: https://bitbucket.org/thtieig/wp2static/

    #!/bin/bash

#########################################################
# Replace the quoted values with the right information: #
# ------------------------------------------------------#

USERNAME="yourusername"
APIKEY="your_api_key"
REGION="datacenter_region"
CONTAINERNAME="mycontainer"
DEVSITE="admin.mywpsite.com"

#########################################################

DEST=/tmp/$DEVSITE/ 
USER=$(echo "$USERNAME" | tr '[:upper:]' '[:lower:]') 
DC=$(echo "$REGION | tr '[:upper:]' '[:lower:]') 

# Clean up / create destination directory
ls $DEST > /dev/null 2&>1 && rm -rf $DEST || mkdir -p $DEST

# Create static copy of the site
httrack $DEVSITE -w -O "$DEST" -q -%i0 -I0 -K0 -X -*/wp-admin/* -*/wp-login*

# Upload the static copy into the container via local network SNET (-I)
turbolift -u $USER -a $APIKEY --os-rax-auth $DC -I upload -s $DEST/$DEVSITE -c $CONTAINERNAME
  12. Now you can test if the upload went well.
    Just open the HTTP link of your Cloud Files Container in your browser and make sure the site is displayed correctly.
  13. If all works as expected, you can now go and make your www.mywpsite.com domain to be a CNAME of that HTTP link instead. Once done, your live traffic will be served by your Cloud Files Container and no longer from your old live site!
  14. And yes, you can now image your Cloud Server and delete it once completed. Next time you need to edit/add some content, just spin up another server from its latest image and add it under the Cloud Load Balancer.

Happy “static’ing“! 🙂

Plesk notes

 

>> Get FTP passwords
# mysql psa -e "select sys_users.login,sys_users.home,domains.name,accounts.password from sys_users,domains,accounts,hosting where sys_users.id=hosting.sys_user_id AND domains.id=hosting.dom_id AND accounts.id=sys_users.account_id"


>> Get email passwords
# /usr/local/psa/admin/sbin/mail_auth_view/usr/local/psa/bin/admin --show-password <----- Plesk 10 and up
cat /etc/psa/.psa.shadow <----- Plesk 6 and up


>> Check which MTA
# alternatives --display mta


>> check mailq (yum install pfHandle)
# pfHandle -s

!!! if you use qmail -> qmHandle


>> Check list of messages queued
# pfHandle -d

!!! If pfHandle does not work, just check inside /var/spool/postfix/



>> Connect to MySQL
mysql -uadmin -p`cat /etc/psa/.psa.shadow`


>> Check version
# cat /usr/local/psa/version 


>> Setup Holland
backupsets/default.conf

[mysql:client]
user = admin
password = file:/etc/psa/.psa.shadow 


>> Check license
/usr/bin/curl -s -k https://127.0.0.1:8443/enterprise/control/agent.php -H "HTTP_AUTH_LOGIN: admin" -H "HTTP_AUTH_PASSWD: `/usr/local/psa/bin/admin --show-password`" -H "HTTP_PRETTY_PRINT: true" -H "Content-Type: text/xml" -d "<packet> <server> <get> <key/> </get> </server> </packet>" | egrep -ohm 1 "PLSK\.[0-9]{8}"


>> Remove license (physically from the server)
[root@344668-web1 ~]# mv /etc/sw/keys/keys/keyXXNb8YmF  /home/user/
[root@344668-web1 ~]#


>> Plesk main logs
MAIL: /usr/local/psa/var/log/maillog
ACCESS LOGS: /var/www/vhosts/*/logs/access_log



>> One-liner to generate reports from the Access Logs

> General report
grep -h "04.Jun.2015" /var/www/vhosts/*/logs/access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 20

> per site report
for i in `find /var/www/vhosts/*/logs/access_log -not -empty `;do echo -n "$i - " ; awk '{print $1}' $i | sort | uniq -c | sort -n | tail -1 ; done | sort –k3 -n | column –t




>> Add custom configuration to Apache under Plesk

# cd /var/www/vhosts/system/DOMAIN.com/conf        
If there is no vhost.conf file then I can create it and add the necessary custom configuration

Need to reconfigure the Plesk Domain - this will Include the custome vhost.conf file
# /usr/local/psa/admin/sbin/httpdmng  -h
# /usr/local/psa/admin/sbin/httpdmng --reconfigure-domain DOMAIN.com



>> Disable SSLv3 on Plesk

If you need to disable SSLv3 on Plesk boxes, here is how to do it:

If nginx is running on port 443, use the following KB: http://kb.sp.parallels.com/en/120083
If Apache is configured on port 443, create /etc/httpd/conf.d/ zz050-psa-disable-weak-ssl-ciphers.conf:

SSLHonorCipherOrder on
SSLProtocol -ALL +TLSv1
SSLCipherSuite ALL:!ADH:!LOW:!SSLv2:!EXP:+HIGH:+MEDIUM


# /usr/local/psa/bin/ipmanage -l
State Type IP                               Clients Hosting PublicIP 
1     S    eth0:172.54.10.212/255.255.252.0 0       0                
0     E    eth2:10.0.1.128/255.255.254.0 0       0                
0     S    eth0:172.54.10.27/255.255.252.0  0       161              
0     E    eth0:172.54.10.28/255.255.252.0  0       1  

# /usr/local/psa/bin/ipmanage -r 172.54.10.212
Error occured while sending feedback. HTTP code returned: 502
SUCCESS: Removal of IP '172.54.10.212' completed.

# /usr/local/psa/bin/ipmanage -l
State Type IP                               Clients Hosting PublicIP 
0     E    eth2:10.0.1.128/255.255.254.0 0       0                
0     S    eth0:172.54.10.27/255.255.252.0  0       161              
0     E    eth0:172.54.10.28/255.255.252.0  0       1