Tag Archives: bash

Restore root access on Linux server

I have been working in IT since a while already, and I have faced multiple times customers that have accidentally lost the root password or the ssh key. In general, their servers is nicely up and running but they can’t connect anymore.

In the Cloud era, 99% of the time is going to be a virtual server, which makes things much easier. Of course, the same approach can be used with physical servers, but the “move disk” that I’m going to explain in a bit requires… literally… plug/unplug the disk 🙂

Firstly, you can connect from your pc to a remote Linux server using:

  • username and password
  • ssh key

In both cases, the Linux server has “something stored” in it, a file (all Linux is based on files)… or more, that we can potentially edit/replace if we have access to the disk.

Getting there, don’t we? 🙂

Cool. So, if we loose the password or the ssh key, and we still desperately need to access that server because we didn’t think about backing up (veeery bad – slap on your hands now!) or our laptop with the ssh key broke (didn’t you backed it up either?? Really? another slap!), one option is actually the following:

  1. Spin up a new server (we’re going to call it Saviour), and verify we can connect to it
  2. Remove the OS disk from the inaccessible server (called Desperate) and connect to Saviour
  3. Modify/replace files from Saviour onto Desperate’s disk
  4. Move back the Desperate’s disk into Desperate server
  5. Test if you can now connect to Desperate
  6. Delete Saviour

As I said before, this could work also with physical servers. The only bit that changes is that you literally need to remove the disk, install to the new server, and put it back. If you don’t have another server, you can use your laptop and a USB adapter, but you still need to have Linux. Anyway, I’m sure you can figure out how to adapt these instructions.

Now, let’s start, but DO IT AT YOUR OWN RISK.

I assume we have Saviour up and running. And we can connect via ssh. If not, stop reading and do something, come on! 🙂

Works? Niiice!
Next, we move Desperate’s disk to Saviour and we make sure it’s visible.

You can use fdisk -l to see if there is a new disk (it’s generally the latest entry).
Here an example:

root@saviour:~# fdisk -l
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6CBB44F1-D559-9B42-A076-7C0EA2B76310

Device      Start      End  Sectors  Size Type
/dev/vda1  262144 20971486 20709343  9.9G Linux root (x86-64)
/dev/vda14   2048     8191     6144    3M BIOS boot
/dev/vda15   8192   262143   253952  124M EFI System

Partition table entries are not in disk order.


Disk /dev/vdb: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6CBB44F1-D559-9B42-A076-7C0EA2B76310

Device      Start      End  Sectors  Size Type
/dev/vdb1  262144 20971486 20709343  9.9G Linux root (x86-64)
/dev/vdb14   2048     8191     6144    3M BIOS boot
/dev/vdb15   8192   262143   253952  124M EFI System

Partition table entries are not in disk order.

We need to get the device ID. Using the example above I can see that Desperate’s disk is /dev/vdb1. How do I know it? Well, bit of experience I guess. But mainly, in this case, disks are listed as vdx. The “x” starts from “a” and continues till “z“. Saviour has one single disk of 10GB, which is the first (a) – of course. The second (b), has to be our Desperate’s disk.

Let’s create a temporary mount point /desperate to mount that disk and let’s mount it!
I assume that our disk was formatted as ext4. If not, you can try to skip the -t option and let the mount command to guess the filesystem or pass the right parameter.

root@saviour:~# mkdir /desperate
root@saviour:~# mount -t ext4 /dev/vdb1 /desperate
root@saviour:~# ls /desperate/
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

If the ls command worked, it means we’re good to continue!

Restore SSH KEY connectivity

If we were able to connect to Savoiur as root using ssh key, it means that the root user on Saviour is properly setup. So, we can copy the same configuration onto the Desperate’s disk to restore ssh key connectivity!

SSH key works storing the public key into a file called authorized_key in .ssh of the user you’re connecting to the server, in this case root.

Simply, let’s copy that file onto Desperate’s disk, in the same path!

root@saviour:~# cp /root/.ssh/authorized_keys /desperate/root/.ssh/authorized_keys
root@saviour:~#

To be extremely sure, we can verify the copy using md5sum (OPTIONAL), and see that the number generated from both files is identical:

root@saviour:~# md5sum /root/.ssh/authorized_keys
17c1bba0ef42de1899b650b60dede12b  /root/.ssh/authorized_keys
root@saviour:~# md5sum /desperate/root/.ssh/authorized_keys
17c1bba0ef42de1899b650b60dede12b  /desperate/root/.ssh/authorized_keys

If we just want to restore ssh key connectivity, we should be good to go, simply turning off Saviour, move back Desperate’s disk in Desperate server, and once up and running, trying to ssh to it using Desperate’s IP/fqdn.

If you have also username and password connectivity to restore, here an example just for root user – but it can be used for all the users, but I won’t explain here how to do so, as I would recommend to restore root and make all the changes from the restored server to avoid misconfigurations.

Restore SSH root password access

The password of a user is stored into /etc/shadow, of course, not in clear.

If you have forgotten the root password, let’s set a root password on Saviour, and use what we can find in that file to update the one on Desperate.

Using the command passwd, as root, we can immediately set/update the password – this time, take a note of it! 😉

And after have set it, we can get the line where it is stored, using grep, for example (or simply opening the file).

root@saviour:~# passwd
New password:
Retype new password:
passwd: password updated successfully
root@saviour:~# grep root /etc/shadow
root:$y$j9T$usUs5.xlf7HQj90AaeYYN.$SR2YO6yamYA1L59bUa193ndPgiyt1nEgCfkgjXEAxJ9:19986:0:99999:7:::

In this example we need to replace the line that starts with root in /desperate/etc/shadow with this one: root:$y$j9T$usUs5.xlf7HQj90AaeYYN.$SR2YO6yamYA1L59bUa193ndPgiyt1nEgCfkgjXEAxJ9:19986:0:99999:7:::
You can use your favourite editor to do so. Make sure you change ONLY that line and save.
You can use the same grep command to verify that it matches.

root@saviour:~# grep root /etc/shadow
root:$y$j9T$usUs5.xlf7HQj90AaeYYN.$SR2YO6yamYA1L59bUa193ndPgiyt1nEgCfkgjXEAxJ9:19986:0:99999:7:::
root@saviour:~# grep root /desperate/etc/shadow
root:$y$j9T$usUs5.xlf7HQj90AaeYYN.$SR2YO6yamYA1L59bUa193ndPgiyt1nEgCfkgjXEAxJ9:19986:0:99999:7:::

At this stage, the root user on Desperate should have the same password that you have set on Saviour.

Turn off Saviour, move the disk back to Desperate, turn it on and test! If all works, let’s thank Saviour, and delete it.

I hope this helps and yes… time to think about backups strategies 😉

Happy restoring! 😉

Convert FLV to MP4

Quick and dirty lines to convert FLV files into MP4.

To do so, you need the package ffmpeg.

# Temporary move all the .flv files into a TMP directory
$ find . -type f -name "*.flv" -exec mv {} ../TMP/ \;

# Remove all spaces from the filename
$ cd ../TMP/
$ for file in *' '* ; do mv -- "$file" "${file// /_}" ; done

# Now, convert all the files from FLV to MP4 using ffmpeg
$ for file in `ls` ; do ffmpeg -i "$file" -c copy "${file//flv/mp4}" ; done

During the process, you might get some errors. You will see some .mp4 files with size 0 bites.

You can then try to remove the broken files and try converting re-encoding

# Find and delete files size 0
$ find . -type f -size 0 -print0 | xargs -0 rm

# Try re-encoding
$ for file in `ls *.flv` ; do ffmpeg -i "$file" -vcodec libx264 -acodec copy "${file//flv/mp4}" ; done

Reverse SSH Tunnel

To allow LOCAL_SERVER behind a firewall/NAT/Home Router to be accessible via SSH from a REMOTE_SERVER you can use a ssh tunnel (reverse).

Basically, from your LOCAL_SERVER you forward port 22 (ssh) to another port on REMOTE_SERVER, for example 8000 and you can ssh into your LOCAL_SERVER from the public IP of the REMOTE_SERVER via port 8000.

To do so, you need to run the following from LOCAL_SERVER:

 local-server: ~ ssh -fNR 8000:localhost:22 <user>@<REMOTE_SERVER>

On REMOTE_SERVER you can use netstat -nlpt to check if there is a service listening on port 8000.

Example:

remote-server ~# netstat -nplt | grep 8000
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      1396/sshd: root
tcp6       0      0 :::8000                 :::*                    LISTEN      1396/sshd: root

In this case, the REMOTE_SERVER allows connection from ALL the interfaces (0.0.0.0) to port 8000.
This means that, if the REMOTE_SERVER has IP 217.160.150.123, if you can connect to LOCAL_SERVER from a THIRD_SERVER using the following:

third-server: ~ ssh -p 8000 <user_local_server>@217.160.150.123

NOTE. If you see that the LISTEN connection on REMOTE_SERVER is bound to 127.0.0.1 and not to 0.0.0.0, it is probably related to the setting GatewayPorts set to no in /etc/ssh/sshd_config on REMOTE_SERVER.
Best setting is clientspecified (rather than yes) as per this post.

Set this value to yes and restart sshd service.

With that setting, you can potentially allow only connection from the REMOTE_SERVER to the LOCAL_SERVER, to increase security.
To do so, you need to use the following ssh command from LOCAL_SERVER:

 local-server: ~ ssh -fNR 127.0.0.1:8000:localhost:22 <user>@<REMOTE_SERVER>

With netstat, you’ll see now this:

remote-server:~# netstat -nplt | grep 8000
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      1461/sshd: root

With this forward, you will be able to access LOCAL_SERVER ONLY from the REMOTE_SERVER itself:

remote-server: ~ ssh -p 8000 <user_local_server>@localhost

I hope this helps 🙂

Happy tunnelling!

LVM – How to

Intro

LVM is a very powerful technology, and can really help the Sysadmin’s life.
However, this is something that we generally setup at the beginning (most of the time now it’s automatically setup during the installation process), and it’s well know… when we stop using something, we tend to forget how to use it.

This is why I’m writing this how to, mostly to keep track of the major features and commands, in case I will need them again in the future 😉

Before proceeding, please digest the following journey of this poor physical device that gets abstracted up to usable pieces.

                                                          VG                        VG
                                                   +---------------+         +---------------+
                                                   |      PV       |  +--->  |               |
                                   PV              | +-----------+ |         |  LV           |
                              +-----------+        | |  8E LVM   | |         |               |
              8E LVM          |  8E LVM   |        | | +-------+ | |         +---------------+
             +-------+        | +-------+ |        | | | +---+ | | |  +--->  +---------------+
+---+        | +---+ |        | | +---+ | |        | | | |DEV| | | |         |               |
|DEV| +----> | |DEV| | +----> | | |DEV| | | +----> | | | +---+ | | |         |  LV           |
+---+        | +---+ |        | | +---+ | |        | | +-------+ | |         |               |
  1.         +-------+        | +-------+ |        | +-----------+ |  +--->  |               |
                2.            +-----------+        |               |         +---------------+
                                   3.              |      PV       |         +---------------+
                                                   | +-----------+ |         |               |
 1. Original Device:                               | |  8E LVM   | |  +--->  |  LV           |
    (physical/virtual disk/partition/raid)         | | +-------+ | |         |               |
 2. fdisk'd to label 8E for LVM                    | | | +---+ | | |         |               |
 3. initialised as LVM Physical Volume             | | | |DEV| | | |         |               |
 4. Added in a LVM Volume Group                    | | | +---+ | | |  +--->  |               |
 5. "partitioned" in single/multiple               | | +-------+ | |         |               |
    LVM Logical Groups                          4. | +-----------+ |      5. |               |
                                                   +---------------+         +---------------+

 

 

Prepare partions

First of all, we need to find which device(s) we want to setup for LVM

fdisk -l

[root@n1 ~]# fdisk -l

Disk /dev/xvda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b7f16

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *        2048    41943039    20970496   83  Linux

Disk /dev/md1: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md2: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md3: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

We can see 3 md devices, probably RAID devices. These are the ones that we are going to use for our LVM exercise.

Now, let’s create an LVM partition.

fdisk <device> => n , p , 1 , (enter) , (enter) , t , 8e , w

[root@n1 ~]# fdisk /dev/md1
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x50b03cd2.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-9759231, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-9759231, default 9759231): 
Using default value 9759231
Partition 1 of type Linux and of size 4.7 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Do the same for all the devices that you want to use for LVM. In my example, I’ve done this for /dev/md1, /dev/md2 and /dev/md3.

Shortcut (risky but quicker) 🙂

echo -e "o\nn\np\n1\n\n\nt\n8e\nw" | fdisk /dev/mdx

All seems now good to go: we have Linux LVM partitions!

Disk /dev/md1: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x50b03cd2

    Device Boot      Start         End      Blocks   Id  System
/dev/md1p1            2048     9759231     4878592   8e  Linux LVM

Disk /dev/md2: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4b68e9a3

    Device Boot      Start         End      Blocks   Id  System
/dev/md2p1            2048     9759231     4878592   8e  Linux LVM

Disk /dev/md3: 4996 MB, 4996726784 bytes, 9759232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x8dcf9ba0

    Device Boot      Start         End      Blocks   Id  System
/dev/md3p1            2048     9759231     4878592   8e  Linux LVM

Time to start to configure LVM

Configure LVM

First of all, we need to make these Linux LVM partition able to be part of a group (vg). I always find tricky to remember the logic behind. Let’s imagine that the device itself now is just labelled “Linux LVM” but we need to initiate it in somehow.

pvcreate <dev>

[root@n1 ~]# pvcreate /dev/md1p1
  Physical volume "/dev/md1p1" successfully created.
[root@n1 ~]# pvcreate /dev/md2p1
  Physical volume "/dev/md2p1" successfully created.
[root@n1 ~]# pvcreate /dev/md3p1
  Physical volume "/dev/md3p1" successfully created.

Now these guys are ready to be part of a group. In this case a Virtual Group (vg).
Let’s check that it’s actually true:
pvs

[root@n1 ~]# pvs
  PV         VG Fmt  Attr PSize PFree
  /dev/md1p1    lvm2 ---  4.65g 4.65g
  /dev/md2p1    lvm2 ---  4.65g 4.65g
  /dev/md3p1    lvm2 ---  4.65g 4.65g

Time to create a group with these devices (this could be done also with just a single device):

vgcreate <lvmgroupname> <dev> <dev> …

[root@n1 ~]# vgcreate mylvmvg /dev/md1p1 /dev/md2p1 /dev/md3p1
  Volume group "mylvmvg" successfully created

Now, let’s check again with pvs and vgs

[root@n1 ~]# pvs
  PV         VG      Fmt  Attr PSize PFree
  /dev/md1p1 mylvmvg lvm2 a--  4.65g 4.65g
  /dev/md2p1 mylvmvg lvm2 a--  4.65g 4.65g
  /dev/md3p1 mylvmvg lvm2 a--  4.65g 4.65g
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   0   0 wz--n- 13.95g 13.95g

Now pvs shows the VG group no longer empty but with mylvmvg. And vgs tells us that the VG is about 14GB in size, fully free with no LV in it.

Good! Now, let’s make some LVs (logical volumes). These will be the new “partitions/disks” that we will be actually able to format, mount and use! 🙂

lvcreate -n <name> -L xGB <vg_group_name>

[root@n1 ~]# lvcreate -n part1 -L 2GB mylvmvg
  Logical volume "part1" created.

Some checks to verify:

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   1   0 wz--n- 13.95g 11.95g
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-a----- 2.00g

A new LV appears in vgs and lvs shows the 2GB volume that we have created.

Let’s create another one, but this time, using the full remaining space (using -l 100%VG option instead of -L xGB)

[root@n1 ~]# lvcreate -n part2 -l 100%VG mylvmvg
  Logical volume "part2" created.
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g    0 
[root@n1 ~]# lvs
  LV    VG      Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-a-----  2.00g                                                    
  part2 mylvmvg -wi-a----- 11.95g                                                    
[root@n1 ~]#

Magic!

Now, we have two devices, both ‘a’ -> active and ready to be formatted:
mkfs.ext4 <device>

[root@n1 ~]# mkfs.ext4 /dev/mylvmvg/part1 
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

I’ve done this for /dev/mylvmvg/part1 and /dev/mylvmvg/part2.

Let’s create the mount points and mount them:

[root@n1 ~]# mkdir -p /mountpoint1 /mountpoint2

[root@n1 ~]# mount -t ext4 /dev/mylvmvg/part1 /mountpoint1

[root@n1 ~]# mount -t ext4 /dev/mylvmvg/part2 /mountpoint2

[root@n1 ~]# mount | grep mylvmvg
/dev/mapper/mylvmvg-part1 on /mountpoint1 type ext4 (rw,relatime,data=ordered)
/dev/mapper/mylvmvg-part2 on /mountpoint2 type ext4 (rw,relatime,data=ordered)

[root@n1 ~]# df -Th | grep mapper
/dev/mapper/mylvmvg-part1 ext4      2.0G  6.0M  1.8G   1% /mountpoint1
/dev/mapper/mylvmvg-part2 ext4       12G   41M   11G   1% /mountpoint2

As you can see, the devices are appearing now as /dev/mapper/mylvmvg-partX. You can use either /dev/mylvmvg/partX or /dev/mapper/mylvmvg-partX. Theoretically, the mapper one is recommended (my bad!).

Now the 2 devices are ready to be used as a typical disk/partition formatted with ext4 filesystem.

Resize Logical Volume

Now, imagine that part1 is too small, and you need more space. And luckily, your part2 volume has plenty. Is there any way to “steal” some space from part2 and give it to part1?
Ooohh yesss! 🙂

How?

  1. shrink part2 logical volume AND its filesystem
  2. expand part1 logical volume AND its filesystem

Here the comments inline:

# Important the -r (this RESIZE the filesystem during the process)
[root@n1 ~]# lvreduce -L -5GB -r /dev/mylvmvg/part2 
Do you want to unmount "/mountpoint2"? [Y|n] y
fsck from util-linux 2.23.2
/dev/mapper/mylvmvg-part2: 12/783360 files (0.0% non-contiguous), 92221/3131392 blocks
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/mapper/mylvmvg-part2 to 1820672 (4k) blocks.
The filesystem on /dev/mapper/mylvmvg-part2 is now 1820672 blocks long.

  Size of logical volume mylvmvg/part2 changed from 11.95 GiB (3058 extents) to 6.95 GiB (1778 extents).
  Logical volume mylvmvg/part2 successfully resized.

# Here we can see that part2 is now smaller than before
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-ao---- 2.00g                                                    
  part2 mylvmvg -wi-ao---- 6.95g                                                    

# And here we can see 5GB available in the vg
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g 5.00g

# We assign the 5GB available to part1
[root@n1 ~]# lvextend -L +5GB -r /dev/mylvmvg/part1
  Size of logical volume mylvmvg/part1 changed from 2.00 GiB (512 extents) to 7.00 GiB (1792 extents).
  Logical volume mylvmvg/part1 successfully resized.
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/mylvmvg-part1 is mounted on /mountpoint1; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/mylvmvg-part1 is now 1835008 blocks long.

# No more Free space
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree
  mylvmvg   3   2   0 wz--n- 13.95g    0 

# part1 is now 7GB (prev 2GB)
[root@n1 ~]# lvs
  LV    VG      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  part1 mylvmvg -wi-ao---- 7.00g                                                    
  part2 mylvmvg -wi-ao---- 6.95g                                                    

# df shows as well the new size
[root@n1 ~]# df -Th | grep mapper
/dev/mapper/mylvmvg-part1 ext4      6.9G  9.1M  6.6G   1% /mountpoint1
/dev/mapper/mylvmvg-part2 ext4      6.8G   37M  6.4G   1% /mountpoint2

 

Move logical volume onto a new RAID array

Now, let’s imagine that one of the 3 initial md devices are having problems, or simply we want to move on a faster/bigger raid array.
The magic of LVM is that we can actually do this with NO DOWNTIME!

How?

In this example we assume that a new /dev/md10 device is attached to our server and we want to remove /dev/md2 device.

  1. We need to take the new device and go through all the previous steps:
    1. fdisk
    2. pvcreate
  2. After that, we need to add this initialised device in the existing volume group (vg)
  3. Move whatever is stored on the physical device
  4. Shrink the volume group
  5. Remove the device
[root@n1 ~]# echo -e "o\nn\np\n1\n\n\nt\n8e\nw" | fdisk /dev/md10
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x465fad01.

Command (m for help): Building a new DOS disklabel with disk identifier 0x5aa41f03.

Command (m for help): Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): Partition number (1-4, default 1): First sector (2048-104791935, default 2048): Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-104791935, default 104791935): Using default value 104791935
Partition 1 of type Linux and of size 50 GiB is set

Command (m for help): Selected partition 1
Hex code (type L to list all codes): Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[root@n1 ~]# fdisk -l | grep md10
Disk /dev/md10: 53.7 GB, 53653471232 bytes, 104791936 sectors
/dev/md10p1            2048   104791935    52394944   8e  Linux LVM

[root@n1 ~]# pvcreate /dev/md10p1
  Physical volume "/dev/md10p1" successfully created.

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1         lvm2 ---  49.97g 49.97g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

[root@n1 ~]# vgextend mylvmvg /dev/md10p1
  Volume group "mylvmvg" successfully extended

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 49.96g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   4   2   0 wz--n- 63.91g 49.96g

Now where the new bits are starting:
pvmove, vgreduce, pvremove

[root@n1 ~]# pvmove /dev/md2p1
  /dev/md2p1: Moved: 0.00%
  /dev/md2p1: Moved: 5.63%
  /dev/md2p1: Moved: 11.51%
  ...
  /dev/md2p1: Moved: 92.61%
  /dev/md2p1: Moved: 98.07%
  /dev/md2p1: Moved: 100.00%

# Here we can see 4 phisical volumes, and a size of ~64GB
[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   4   2   0 wz--n- 63.91g 49.96g

# We can see also that /dev/md2p1 is now fully FREE
[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1  mylvmvg lvm2 a--   4.65g  4.65g
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

# we can safely remove this device from the vg
[root@n1 ~]# vgreduce mylvmvg /dev/md2p1
  Removed "/dev/md2p1" from volume group "mylvmvg"

[root@n1 ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  mylvmvg   3   2   0 wz--n- 59.26g 45.32g

#/dev/md2p1 doesn't belong to any VG anymore
[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md2p1          lvm2 ---   4.65g  4.65g
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0 

# Removing and confirm: no more /dev/md2p1
[root@n1 ~]# pvremove /dev/md2p1
  Labels on physical volume "/dev/md2p1" successfully wiped.

[root@n1 ~]# pvs
  PV          VG      Fmt  Attr PSize  PFree 
  /dev/md10p1 mylvmvg lvm2 a--  49.96g 45.32g
  /dev/md1p1  mylvmvg lvm2 a--   4.65g     0 
  /dev/md3p1  mylvmvg lvm2 a--   4.65g     0

 

In this example we have left LVM to decide where to put the data that was stored on /dev/md2 device.
Just for reference, we could have specified the destination physical device (e.g. if we were thinking to remove more devices and make sure that the data was ending up on the new RAID and not sprat across the other disks):

pvmove /dev/md2p1 /dev/md10p1

Or, in case we just wanted to move a specific logical volume, let’s say part1

pvmove -n part1 /dev/md2p1 /dev/md10p1

 

…happy LVM’ing! 😉

Screen – basic commands 

>> Create a screen session (labelled)
screen -R 'myscreen'

>> Detach screen
ctrl+A (hold them) + D

>> Check current screen sessions
screen -ls

>>Re-attach screen session
screen -r <screen name, from screen -ls including the PID>
e.g. screen -r 4238.myscreen

>> Quit session
screen -X -S [session # you want to kill] quit


=======================================

>> When it gets badly stuck

screen -ls | grep pts | cut -d. -f1 | awk '{print $1}' | xargs kill 
screen -ls | grep Attached | cut -d. -f1 | awk '{print $1}' | xargs kill 

ref: http://askubuntu.com/questions/356006/kill-a-screen-session

 

Bash log redirect / stdout and stderr 

# For cron/crontab
*/5 * * * * /path/myscript.sh > /dev/null 2>&1

==================================================
# Redirect ALL output/error automatically to a file 
# AND print to console too
LOG_OUTPUT=output_error.log

exec 1> >(tee -i ${LOG_OUTPUT}) 2>&1

==================================================
# *ALL* redirected to the log files: NO console output at all
LOG_OUTPUT=output.log

exec 1>>${LOG_OUTPUT} 2>&1

==================================================
# All redirected to 2 different log files
LOG_OUTPUT=etup_output.log
LOG_ERROR=setup_error.log

exec 3>&1 1>>${LOG_OUTPUT}
exec 2>>${LOG_ERROR}

# use 'P "my message"' instead of echo
P () {
# Print on console AND file
echo -e "\n$1" | tee /dev/fd/3

# Print ONLY on console
#echo -e "\n$1" 1>&3
}

==================================================
# ALL stdout and stderr to $LOG_OUTPUT
# Also stderr to $LOG_ERROR (for extra checks)
# P function to print to the console AND logged into $LOG_OUTPUT
LOG_OUTPUT=output.log
LOG_ERROR=error.log

exec 3>&1 1>>${LOG_OUTPUT}
exec 2> >(tee -i ${LOG_ERROR}) >> ${LOG_OUTPUT}

# use 'P "my message"' instead of echo
P () {
# Print on console AND file
echo -e "$1" | tee /dev/fd/3

# Print ONLY on console
#echo -e "\n$1" 1>&3
}

# use 'P "my message"' instead of echo to print in BLUE
P () {
BLUE='\033[1;34m'
NC='\033[0m' # No Color
echo -e "\n${BLUE}${1}${NC}" | tee /dev/fd/3
}

 

Find files based on date/time 

# ONE LINERS


> Modified in the last 12 hours (720 min)
find . -cmin -720 

> Modified in the last day
find . -mtime -1



# => ctime - for hacked/modified files 
# look for ctime instead, hacked scripts can't set that to what they want as opposed to mtime:

find -cmin -$n_minutes_ago
find -ctime -$n_days_ago
ls -lc   ## sorted by name
ls -ltc   ## sorted by time


>> File OLDER THAN xx days:
find . -type f -ctime +$n_days_ago

>> Find files RESTORED older that xx days and MOVE them
find . -type f -mtime +$n_days_ago | xargs -I '{}' mv {} /destination/path/