Tag Archives: command line

Reverse SSH Tunnel

To allow LOCAL_SERVER behind a firewall/NAT/Home Router to be accessible via SSH from a REMOTE_SERVER you can use a ssh tunnel (reverse).

Basically, from your LOCAL_SERVER you forward port 22 (ssh) to another port on REMOTE_SERVER, for example 8000 and you can ssh into your LOCAL_SERVER from the public IP of the REMOTE_SERVER via port 8000.

To do so, you need to run the following from LOCAL_SERVER:

 local-server: ~ ssh -fNR 8000:localhost:22 <user>@<REMOTE_SERVER>

On REMOTE_SERVER you can use netstat -nlpt to check if there is a service listening on port 8000.

Example:

remote-server ~# netstat -nplt | grep 8000
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      1396/sshd: root
tcp6       0      0 :::8000                 :::*                    LISTEN      1396/sshd: root

In this case, the REMOTE_SERVER allows connection from ALL the interfaces (0.0.0.0) to port 8000.
This means that, if the REMOTE_SERVER has IP 217.160.150.123, if you can connect to LOCAL_SERVER from a THIRD_SERVER using the following:

third-server: ~ ssh -p 8000 <user_local_server>@217.160.150.123

NOTE. If you see that the LISTEN connection on REMOTE_SERVER is bound to 127.0.0.1 and not to 0.0.0.0, it is probably related to the setting GatewayPorts set to no in /etc/ssh/sshd_config on REMOTE_SERVER.
Best setting is clientspecified (rather than yes) as per this post.

Set this value to yes and restart sshd service.

With that setting, you can potentially allow only connection from the REMOTE_SERVER to the LOCAL_SERVER, to increase security.
To do so, you need to use the following ssh command from LOCAL_SERVER:

 local-server: ~ ssh -fNR 127.0.0.1:8000:localhost:22 <user>@<REMOTE_SERVER>

With netstat, you’ll see now this:

remote-server:~# netstat -nplt | grep 8000
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      1461/sshd: root

With this forward, you will be able to access LOCAL_SERVER ONLY from the REMOTE_SERVER itself:

remote-server: ~ ssh -p 8000 <user_local_server>@localhost

I hope this helps 🙂

Happy tunnelling!

Virtualhost and Letsencrypt

Quick guideline about how to install multiple sites on a single server using Virtualhosting, and have the SSL certificate installed and automatically renewed using Letsencrypt.

There are plenty of how to online, but I wanted to have a quick reference page for myself 🙂

Firstly, this has been tested on Debian 12, but it should work on previous Debian versions and Ubuntu too.

Apache setup and virtualhosts

Firstly, install Apache and other packages that you will mostly likely need, especially if you run WordPress or any php based framework:

apt-get install apache2 php php-mysql libapache2-mod-php php-gd php-curl net-tools telnet dos2unix

Now, you should create the folder structure to host your sites. I used /var/www/virtualhosts/<site>/public_html

I made sure permissions were set correctly too:

chown -R www-data:www-data /var/www/
find /var/www -type -d -exec chmod 775 {} \;

Now, create a virtualhost file for each site. In the following example we are going to create the conf file for site1.

Create /etc/apache2/sites-available/site1.conf

<VirtualHost *:80>
    ServerName site1.com
    ServerAlias www.site1.com
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/virtualhosts/site1/public_html

    <Directory /var/www/virtualhosts/site1/public_html>
        Options -Indexes +FollowSymLinks
        AllowOverride All
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/site1-error.log
    CustomLog ${APACHE_LOG_DIR}/site1-access.log combined
</VirtualHost>

Do the same for all the sites you have.

Once done, upload the content of your sites in public_html folder.

Disable all the default Apache sites and enable the ones you have created. You can use the commands a2dissite and a2ensite or manually create symbolic links into /etc/apache2/sites-enabled/

Check that all the virtualhosts are properly loaded:

source /etc/apache2/envvars
apache2 -S

You should see all your sites under *80 section.
Right now we have enabled only Apache on port 80 to return the sites we have hosted. No 443 yet.

Now, you can use curl to do some tests to see if the virtual hosts are responding correctly.

~ curl -IH'Host: site1.com' http://<server_IP>  # to get the header of site1.com
~ curl -H'Host: site1.com' http://<server_IP>  # to get the full page of site1.com

Hopefully all works (if not, troubleshoot it heheh), let’s point our DNS to our server, and test directly using the domain names.

All good? Cool!

Make sure now that your firewall allows port 80 and port 443. Even if you’re considering to serve your site ONLY over SSL (port 443), the certbot tool that does the auto-renewal of the certificate needs port 80 open.

Installation and configuration of certbot – Letsencrypt

As root, issue the below commands:

apt-get install snapd
snap install core
snap refresh core
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot

You have now the certbot tool installed.

Following the above example of site1.com, we are going now to get the SSL certificate for that site (even the www.site1.com one), and let the tool install and configure everything automatically.

certbot --apache -d site1.com -d www.site1.com

Hopefully all goes well 🙂 Repeat for each of your sites accordingly.

Once done with all the sites, just to make sure the auto-renewal works, you can also issue a dry-run check:

certbot renew --dry-run

Letsencrypt certificates last 90 days (afaik), but the certbot tool installed in this way does the auto-renewal in an automatic fashion.
If you’re curios where this is written (you might think about cron but unable to find anything – like it happend to me).
If this is the case, you can try to run this command, and you may find the certbot listed:

systemctl list-timers

More information are available on the official website at this address.

You can now test using curl again, but hitting https instead of http:

~ curl -IH'Host: site1.com' https://<server_IP>  # to get the header of site1.com
~ curl -H'Host: site1.com' https://<server_IP>  # to get the full page of site1.com

Oh, one note.
By default, at least at the time when I’m writing this article, once you install the certificate, the *80 virtualhost of your site will be modified, adding the following lines, which force a 302 redirect from http to https. 

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.site1.com [OR]
RewriteCond %{SERVER_NAME} =site1.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

If it’s what you want – cool.
If you still want to serve your site on http AND https, comment out (or delete) those new lines.

Happy virtualhosting and ssl’ing! 🙂

Manage PDF files

Merge multiple files into single PDF

I’m sure that we all had the need to send a single PDF file, maybe a signed contract. Yes, those 20 or more pages that you need to return, probably with just two of them filled up and signed.

Some PDF give you the ability to digitally sign them. But in my experience, most of them aren’t so modern.

So, what do I do?

I print ONLY the pages that I need to sign, scan them and here I am, with the need to “rebuild” the PDF, replacing the pages signed.

Example.
You have the file contract.pdf, with 20 pages and you need to sign page 10 and page 20.
The scan has a different resolution (or, even worse, it’s a different format, like jpg).

Here the command to make the magic happen:

convert contract.pdf[0-8] mypage10.jpg contract.pdf[10-18] mypage20.jpg -resize 1240x1753 -extent 1240x1753 -gravity center -units PixelsPerInch -density 150x150 contract_signed.pdf

The bit before -resize is pretty self explanatory. The bit after is a way to have the size of all pages fitting an A4 format, with a good printable resolution.

Of course, to make this happen, you need Linux (or WSL on Windows 10) and imagemagick installed.

Another way is using ghostscript.

A simple Ghostscript command to merge two PDFs in a single file is shown below:

gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=combine.pdf -dBATCH 1.pdf 2.pdf

What about a quick onliner to reduce and convert to grayscale your pdf?

ghostscript -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -sProcessColorModel=DeviceGray -sColorConversionStrategy=Gray -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

PDF size reduce

Sometimes instead, you need to reduce the size of an existing PDF. Here a handy oneliner, using ghostscript:

ghostscript -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/printer -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Other options for PDFSETTINGS:

  • /screen selects low-resolution output similar to the Acrobat Distiller “Screen Optimized” setting.
  • /ebook selects medium-resolution output similar to the Acrobat Distiller “eBook” setting.
  • /printer selects output similar to the Acrobat Distiller “Print Optimized” setting.
  • /prepress selects output similar to Acrobat Distiller “Prepress Optimized” setting.
  • /default selects output intended to be useful across a wide variety of uses, possibly at the expense of a larger output file.

Happy PDF’ing 🙂

Sources:
https://stackoverflow.com/questions/23214617/imagemagick-convert-image-to-pdf-with-a4-page-size-and-image-fit-to-page
https://www.shellhacks.com/merge-pdf-files-linux-command-line/
https://gist.github.com/firstdoit/6390547

TOP – memory explanation

(just few notes – to avoid to forget)

  • VIRT: not really relevant nowadays. It’s the memory that the process could use. But the OS loads only what needed, so rarely really used. On 32bit OS, it could be the only time when you need to keep an eye as the OS can allocate up to 2-3GB only.
  • RES: Resident Set Size memory – this is the actual memory in RAM. On low used machines, it might still show high usage even if not utilised as the process to free-up the memory costs more than leaving it. In fact, Linux OS tends to use as much memory available (“unused memory is wasted memory“).
  • SH: this is the shared memory which generally contains libraries etc

LVM – How to

Intro

LVM is a very powerful technology, and can really help the Sysadmin’s life.
However, this is something that we generally setup at the beginning (most of the time now it’s automatically setup during the installation process), and it’s well know… when we stop using something, we tend to forget how to use it.

This is why I’m writing this how to, mostly to keep track of the major features and commands, in case I will need them again in the future 😉

Before proceeding, please digest the following journey of this poor physical device that gets abstracted up to usable pieces.

                                                          VG                        VG
                                                   +---------------+         +---------------+
                                                   |      PV       |  +--->  |               |
                                   PV              | +-----------+ |         |  LV           |
                              +-----------+        | |  8E LVM   | |         |               |
              8E LVM          |  8E LVM   |        | | +-------+ | |         +---------------+
             +-------+        | +-------+ |        | | | +---+ | | |  +--->  +---------------+
+---+        | +---+ |        | | +---+ | |        | | | |DEV| | | |         |               |
|DEV| +----> | |DEV| | +----> | | |DEV| | | +----> | | | +---+ | | |         |  LV           |
+---+        | +---+ |        | | +---+ | |        | | +-------+ | |         |               |
  1.         +-------+        | +-------+ |        | +-----------+ |  +--->  |               |
                2.            +-----------+        |               |         +---------------+
                                   3.              |      PV       |         +---------------+
                                                   | +-----------+ |         |               |
 1. Original Device:                               | |  8E LVM   | |  +--->  |  LV           |
    (physical/virtual disk/partition/raid)         | | +-------+ | |         |               |
 2. fdisk'd to label 8E for LVM                    | | | +---+ | | |         |               |
 3. initialised as LVM Physical Volume             | | | |DEV| | | |         |               |
 4. Added in a LVM Volume Group                    | | | +---+ | | |  +--->  |               |
 5. "partitioned" in single/multiple               | | +-------+ | |         |               |
    LVM Logical Groups                          4. | +-----------+ |      5. |               |
                                                   +---------------+         +---------------+

 

 

Prepare partions

First of all, we need to find which device(s) we want to setup for LVM

fdisk -l

[root@n1 ~]# fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b7f16

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *        2048    41943039    20970496   83  Linux

Disk /dev/md1: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md2: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md3: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

We can see 3 md devices, probably RAID devices. These are the ones that we are going to use for our LVM exercise.

Now, let’s create an LVM partition.

fdisk <device> => n , p , 1 , (enter) , (enter) , t , 8e , w

[root@n1 ~]# fdisk /dev/md1
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x50b03cd2.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-9759231, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-9759231, default 9759231): 
Using default value 9759231
Partition 1 of type Linux and of size 4.7 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Do the same for all the devices that you want to use for LVM. In my example, I’ve done this for /dev/md1, /dev/md2 and /dev/md3.

Shortcut (risky but quicker) 🙂

echo -e "o\nn\np\n1\n\n\nt\n8e\nw" | fdisk /dev/mdx

All seems now good to go: we have Linux LVM partitions!

Disk /dev/md1: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x50b03cd2

    Device Boot      Start         End      Blocks   Id  System
/dev/md1p1            2048     9759231     4878592   8e  Linux LVM

Disk /dev/md2: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4b68e9a3

    Device Boot      Start         End      Blocks   Id  System
/dev/md2p1            2048     9759231     4878592   8e  Linux LVM

Disk /dev/md3: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8dcf9ba0

    Device Boot      Start         End      Blocks   Id  System
/dev/md3p1            2048     9759231     4878592   8e  Linux LVM

Time to start to configure LVM

Configure LVM

First of all, we need to make these Linux LVM partition able to be part of a group (vg). I always find tricky to remember the logic behind. Let’s imagine that the device itself now is just labelled “Linux LVM” but we need to initiate it in somehow.

pvcreate <dev>

[root@n1 ~]# pvcreate /dev/md1p1
  Physical volume "/dev/md1p1" successfully created.
[root@n1 ~]# pvcreate /dev/md2p1
  Physical volume "/dev/md2p1" successfully created.
[root@n1 ~]# pvcreate /dev/md3p1
  Physical volume "/dev/md3p1" successfully created.

Now these guys are ready to be part of a group. In this case a Virtual Group (vg).
Let’s check that it’s actually true:
pvs

[root@n1 ~]# pvs
  PV         VG Fmt  Attr PSize PFree
  /dev/md1p1    lvm2 ---  4.65g 4.65g
  /dev/md2p1    lvm2 ---  4.65g 4.65g
  /dev/md3p1    lvm2 ---  4.65g 4.65g

Time to create a group with these devices (this could be done also with just a single device):

vgcreate <lvmgroupname> <dev> <dev> …

[root@n1 ~]# vgcreate mylvmvg /dev/md1p1 /dev/md2p1 /dev/md3p1
  Volume group "mylvmvg" successfully created

Now, let’s check again with pvs and vgs

[root@n1 ~]# pvs
  PV         VG      Fmt  Attr PSize PFree
  /dev/md1p1 mylvmvg lvm2 a--  4.65g 4.65g
  /dev/md2p1 mylvmvg lvm2 a--  4.65g 4.65g
  /dev/md3p1 mylvmvg lvm2 a--  4.65g 4.65g
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   0   0 wz--n- 13.95g 13.95g

Now pvs shows the VG group no longer empty but with mylvmvg. And vgs tells us that the VG is about 14GB in size, fully free with no LV in it.

Good! Now, let’s make some LVs (logical volumes). These will be the new “partitions/disks” that we will be actually able to format, mount and use! 🙂

lvcreate -n <name> -L xGB <vg_group_name>

[root@n1 ~]# lvcreate -n part1 -L 2GB mylvmvg
  Logical volume "part1" created.

Some checks to verify:

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   1   0 wz--n- 13.95g 11.95g
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-a----- 2.00g

A new LV appears in vgs and lvs shows the 2GB volume that we have created.

Let’s create another one, but this time, using the full remaining space (using -l 100%VG option instead of -L xGB)

[root@n1 ~]# lvcreate -n part2 -l 100%VG mylvmvg
  Logical volume "part2" created.
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g    0 
[root@n1 ~]# lvs
  LV    VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-a-----  2.00g                                                    
  part2 mylvmvg -wi-a----- 11.95g                                                    
[root@n1 ~]#

Magic!

Now, we have two devices, both ‘a’ -> active and ready to be formatted:
mkfs.ext4 <device>

[root@n1 ~]# mkfs.ext4 /dev/mylvmvg/part1 
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

I’ve done this for /dev/mylvmvg/part1 and /dev/mylvmvg/part2.

Let’s create the mount points and mount them:

[root@n1 ~]# mkdir -p /mountpoint1 /mountpoint2

[root@n1 ~]# mount -t ext4 /dev/mylvmvg/part1 /mountpoint1

[root@n1 ~]# mount -t ext4 /dev/mylvmvg/part2 /mountpoint2

[root@n1 ~]# mount | grep mylvmvg
/dev/mapper/mylvmvg-part1 on /mountpoint1 type ext4 (rw,relatime,data=ordered)
/dev/mapper/mylvmvg-part2 on /mountpoint2 type ext4 (rw,relatime,data=ordered)

[root@n1 ~]# df -Th | grep mapper
/dev/mapper/mylvmvg-part1 ext4      2.0G  6.0M  1.8G   1% /mountpoint1
/dev/mapper/mylvmvg-part2 ext4       12G   41M   11G   1% /mountpoint2

As you can see, the devices are appearing now as /dev/mapper/mylvmvg-partX. You can use either /dev/mylvmvg/partX or /dev/mapper/mylvmvg-partX. Theoretically, the mapper one is recommended (my bad!).

Now the 2 devices are ready to be used as a typical disk/partition formatted with ext4 filesystem.

Resize Logical Volume

Now, imagine that part1 is too small, and you need more space. And luckily, your part2 volume has plenty. Is there any way to “steal” some space from part2 and give it to part1?
Ooohh yesss! 🙂

How?

  1. shrink part2 logical volume AND its filesystem
  2. expand part1 logical volume AND its filesystem

Here the comments inline:

# Important the -r (this RESIZE the filesystem during the process)
[root@n1 ~]# lvreduce -L -5GB -r /dev/mylvmvg/part2 
Do you want to unmount "/mountpoint2"? [Y|n] y
fsck from util-linux 2.23.2
/dev/mapper/mylvmvg-part2: 12/783360 files (0.0% non-contiguous), 92221/3131392 blocks
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/mapper/mylvmvg-part2 to 1820672 (4k) blocks.
The filesystem on /dev/mapper/mylvmvg-part2 is now 1820672 blocks long.

  Size of logical volume mylvmvg/part2 changed from 11.95 GiB (3058 extents) to 6.95 GiB (1778 extents).
  Logical volume mylvmvg/part2 successfully resized.

# Here we can see that part2 is now smaller than before
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-ao---- 2.00g                                                    
  part2 mylvmvg -wi-ao---- 6.95g                                                    

# And here we can see 5GB available in the vg
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g 5.00g

# We assign the 5GB available to part1
[root@n1 ~]# lvextend -L +5GB -r /dev/mylvmvg/part1
  Size of logical volume mylvmvg/part1 changed from 2.00 GiB (512 extents) to 7.00 GiB (1792 extents).
  Logical volume mylvmvg/part1 successfully resized.
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/mylvmvg-part1 is mounted on /mountpoint1; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/mylvmvg-part1 is now 1835008 blocks long.

# No more Free space
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g    0 

# part1 is now 7GB (prev 2GB)
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-ao---- 7.00g                                                    
  part2 mylvmvg -wi-ao---- 6.95g                                                    

# df shows as well the new size
[root@n1 ~]# df -Th | grep mapper
/dev/mapper/mylvmvg-part1 ext4      6.9G  9.1M  6.6G   1% /mountpoint1
/dev/mapper/mylvmvg-part2 ext4      6.8G   37M  6.4G   1% /mountpoint2

 

Move logical volume onto a new RAID array

Now, let’s imagine that one of the 3 initial md devices are having problems, or simply we want to move on a faster/bigger raid array.
The magic of LVM is that we can actually do this with NO DOWNTIME!

How?

In this example we assume that a new /dev/md10 device is attached to our server and we want to remove /dev/md2 device.

  1. We need to take the new device and go through all the previous steps:
    1. fdisk
    2. pvcreate
  2. After that, we need to add this initialised device in the existing volume group (vg)
  3. Move whatever is stored on the physical device
  4. Shrink the volume group
  5. Remove the device
[root@n1 ~]# echo -e "o\nn\np\n1\n\n\nt\n8e\nw" | fdisk /dev/md10
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x465fad01.

Command (m for help): Building a new DOS disklabel with disk identifier 0x5aa41f03.

Command (m for help): Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): Partition number (1-4, default 1): First sector (2048-104791935, default 2048): Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-104791935, default 104791935): Using default value 104791935
Partition 1 of type Linux and of size 50 GiB is set

Command (m for help): Selected partition 1
Hex code (type L to list all codes): Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[root@n1 ~]# fdisk -l | grep md10
Disk /dev/md10: 53.7 GB, 53653471232 bytes, 104791936 sectors
/dev/md10p1            2048   104791935    52394944   8e  Linux LVM

[root@n1 ~]# pvcreate /dev/md10p1
  Physical volume "/dev/md10p1" successfully created.

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1         lvm2 ---  49.97g 49.97g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

[root@n1 ~]# vgextend mylvmvg /dev/md10p1
  Volume group "mylvmvg" successfully extended

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 49.96g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   4   2   0 wz--n- 63.91g 49.96g

Now where the new bits are starting:
pvmove, vgreduce, pvremove

[root@n1 ~]# pvmove /dev/md2p1
  /dev/md2p1: Moved: 0.00%
  /dev/md2p1: Moved: 5.63%
  /dev/md2p1: Moved: 11.51%
  ...
  /dev/md2p1: Moved: 92.61%
  /dev/md2p1: Moved: 98.07%
  /dev/md2p1: Moved: 100.00%

# Here we can see 4 phisical volumes, and a size of ~64GB
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   4   2   0 wz--n- 63.91g 49.96g

# We can see also that /dev/md2p1 is now fully FREE
[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g  4.65g
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

# we can safely remove this device from the vg
[root@n1 ~]# vgreduce mylvmvg /dev/md2p1
  Removed "/dev/md2p1" from volume group "mylvmvg"

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   2   0 wz--n- 59.26g 45.32g

#/dev/md2p1 doesn't belong to any VG anymore
[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1          lvm2 ---   4.65g  4.65g
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

# Removing and confirm: no more /dev/md2p1
[root@n1 ~]# pvremove /dev/md2p1
  Labels on physical volume "/dev/md2p1" successfully wiped.

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0

 

In this example we have left LVM to decide where to put the data that was stored on /dev/md2 device.
Just for reference, we could have specified the destination physical device (e.g. if we were thinking to remove more devices and make sure that the data was ending up on the new RAID and not sprat across the other disks):

pvmove /dev/md2p1 /dev/md10p1

Or, in case we just wanted to move a specific logical volume, let’s say part1

pvmove -n part1 /dev/md2p1 /dev/md10p1

 

…happy LVM’ing! 😉

Varnish – basic notes 

ACLs: /etc/varnish/default.vcl

Memory usage:
grep VARNISH_STORAGE_SIZE /etc/sysconfig/varnish

Check how much memory can use: (check last parameter in the output line)

# ps aux | grep varnish
root     27093  0.0  0.1 112304  1140 ?        Ss   16:28   0:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,256M
varnish  27094  0.1  0.9 21760528 9240 ?       Sl   16:28   0:02 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,256M


>> Test VCL
# varnishd -C -f /etc/varnish/default.vcl


>> Test if varnish works
# varnishstat

 

Sophos antivirus notes

Generic checks

ps aux |grep sav (check process)

/opt/sophos-av/bin/savdstatus --version (version, last update, thread data)
/opt/sophos-av/bin/savconfig -v (info about exclusions, where the Datacentre is that hosts that Sophos device, named scans etc )
/opt/sophos-av/bin/savconfig get TalpaOperations (check disabled mode)

cat /proc/sys/talpa/intercept-filters/VettingController/ops (check all modes)
/opt/sophos-av/bin/savconfig set TalpaOperations -- -open (set mode to disabled for open/read)
/opt/sophos-av/bin/savconfig get TalpaOperations

cat /proc/sys/talpa/intercept-filters/VettingController/ops
-open
+close
+exec
+mount
+umount

/opt/sophos-av/bin/savconfig query NamedScans (Check Scheduled Scans)
/opt/sophos-av/bin/savconfig query NamedScans SEC:FullSystemScan (Check Scheduled Scans with argument)
/opt/sophos-av/bin/savconfig add ExcludeFilePaths /home/user1/ (ADD Exclude files' path)
/opt/sophos-av/bin/savconfig remove ExcludeFilePaths /home/user1/ (REMOVE Exclude files' path)


# Check Global exclusions 
/opt/sophos-av/bin/savconfig query ExcludeFileOnGlob && /opt/sophos-av/bin/savconfig query ExcludeFilePaths

/opt/sophos-av/bin/savdctl disable (disable on-access scanning)
/opt/sophos-av/bin/savdstatus (check)
Sophos Anti-Virus is active but on-access scanning is not running

To get ON-Access Scanning back, restart all Sophos related services:
for i in `chkconfig --list |grep sav |awk '{print $1}'`;do echo -e "\n\e[93mShow service $i restart \e[0m\n";service $i restart;done

Scan

>> Perform the scan -> this will create a log
savscan -nc -f -s --no-follow-symlinks --backtrack-protection --quarantine <path/to/scan> (manual scan)

>> Than, check the log to see what it has been found from the manual scan
/opt/sophos-av/bin/savlog --today --utc | grep detected (check threats for today -)
grep INFECTED /opt/sophos-av/log/savd.log | grep -P -o '(?<=arg>)/[^<]*(?=</arg)' | sort -u (check  all threats)
savscan --help

Example for multiple folders with final report:

(suggested to run in a screen session)

  1. Create a temporary folder: 
    mkdir -p /tmp/scantmp/ > && cd $_
  2. list all directories that you want to scan (full path) into a file called list_folder.txt within the temp folder;
  3. Run the following: 
    for i in `cat list_folder.txt` ; do nice / renice -n 19 savscan -nc -f -s --no-follow-symlinks --backtrack-protection --quarantine $i 2>&1 >> scan.log ; done
/opt/sophos-av/bin/savlog --today --utc | grep "Threat detected" | awk -F" " '{print $2}' > report.txt
  4. Check report.txt 

 

Plesk notes

 

>> Get FTP passwords
# mysql psa -e "select sys_users.login,sys_users.home,domains.name,accounts.password from sys_users,domains,accounts,hosting where sys_users.id=hosting.sys_user_id AND domains.id=hosting.dom_id AND accounts.id=sys_users.account_id"


>> Get email passwords
# /usr/local/psa/admin/sbin/mail_auth_view/usr/local/psa/bin/admin --show-password <----- Plesk 10 and up
cat /etc/psa/.psa.shadow <----- Plesk 6 and up


>> Check which MTA
# alternatives --display mta


>> check mailq (yum install pfHandle)
# pfHandle -s

!!! if you use qmail -> qmHandle


>> Check list of messages queued
# pfHandle -d

!!! If pfHandle does not work, just check inside /var/spool/postfix/



>> Connect to MySQL
mysql -uadmin -p`cat /etc/psa/.psa.shadow`


>> Check version
# cat /usr/local/psa/version 


>> Setup Holland
backupsets/default.conf

[mysql:client]
user = admin
password = file:/etc/psa/.psa.shadow 


>> Check license
/usr/bin/curl -s -k https://127.0.0.1:8443/enterprise/control/agent.php -H "HTTP_AUTH_LOGIN: admin" -H "HTTP_AUTH_PASSWD: `/usr/local/psa/bin/admin --show-password`" -H "HTTP_PRETTY_PRINT: true" -H "Content-Type: text/xml" -d "<packet> <server> <get> <key/> </get> </server> </packet>" | egrep -ohm 1 "PLSK\.[0-9]{8}"


>> Remove license (physically from the server)
[root@344668-web1 ~]# mv /etc/sw/keys/keys/keyXXNb8YmF  /home/user/
[root@344668-web1 ~]#


>> Plesk main logs
MAIL: /usr/local/psa/var/log/maillog
ACCESS LOGS: /var/www/vhosts/*/logs/access_log



>> One-liner to generate reports from the Access Logs

> General report
grep -h "04.Jun.2015" /var/www/vhosts/*/logs/access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 20

> per site report
for i in `find /var/www/vhosts/*/logs/access_log -not -empty `;do echo -n "$i - " ; awk '{print $1}' $i | sort | uniq -c | sort -n | tail -1 ; done | sort –k3 -n | column –t




>> Add custom configuration to Apache under Plesk

# cd /var/www/vhosts/system/DOMAIN.com/conf        
If there is no vhost.conf file then I can create it and add the necessary custom configuration

Need to reconfigure the Plesk Domain - this will Include the custome vhost.conf file
# /usr/local/psa/admin/sbin/httpdmng  -h
# /usr/local/psa/admin/sbin/httpdmng --reconfigure-domain DOMAIN.com



>> Disable SSLv3 on Plesk

If you need to disable SSLv3 on Plesk boxes, here is how to do it:

If nginx is running on port 443, use the following KB: http://kb.sp.parallels.com/en/120083
If Apache is configured on port 443, create /etc/httpd/conf.d/ zz050-psa-disable-weak-ssl-ciphers.conf:

SSLHonorCipherOrder on
SSLProtocol -ALL +TLSv1
SSLCipherSuite ALL:!ADH:!LOW:!SSLv2:!EXP:+HIGH:+MEDIUM


# /usr/local/psa/bin/ipmanage -l
State Type IP                               Clients Hosting PublicIP 
1     S    eth0:172.54.10.212/255.255.252.0 0       0                
0     E    eth2:10.0.1.128/255.255.254.0 0       0                
0     S    eth0:172.54.10.27/255.255.252.0  0       161              
0     E    eth0:172.54.10.28/255.255.252.0  0       1  

# /usr/local/psa/bin/ipmanage -r 172.54.10.212
Error occured while sending feedback. HTTP code returned: 502
SUCCESS: Removal of IP '172.54.10.212' completed.

# /usr/local/psa/bin/ipmanage -l
State Type IP                               Clients Hosting PublicIP 
0     E    eth2:10.0.1.128/255.255.254.0 0       0                
0     S    eth0:172.54.10.27/255.255.252.0  0       161              
0     E    eth0:172.54.10.28/255.255.252.0  0       1

 

Atop – notes 

atop -a | Display only active processes
atop -g | Display general process info
atop -m | Display memory usage info
atop -n | Display network usage info
atop -d | Display Dick usage info

Alternatively you can just use atop and then key in the letters above to switch between.

atop -r | read raw data.

Use this to basically start looking at the processes from the start of the day 00:00

atop -r -b 09:00    | read raw data from 09:00 today
atop -r y           | read raw data from yesterday
atop -r yy          | read raw data from the day before yesterday
atop -r y -b 09:00  | read raw data from 09:00 yesterday
atop -r yy -b 09:00 | read raw data from 09:00 the day before yesterday
atop -r <log>       | read data from a log stored in /var/log/atop