Tag Archives: backup

Migrate Linux Subsystem from one PC to another

Are you enjoying your favorite Linux distro running within the Windows 10 Linux Subsystem?

Have you configured all nicely?

What happened if you get a new pc and you’d like to migrate your VM across?

This is what happened to me. And looking around, I found this post that gave me this kinda-dirty way, but did work!

After that, I decided to review the steps, and I’ve added these directories in the exclude’s list, to make clearer the process of export/import:

/dev
/proc
/sys
/run
/tmp
/media
/mnt
/var/cache
/var/run

/dev

/proc

/sys

/run

/tmp

/media

/mnt

/var/cache

/var/run

Of course, if you have important data in these folders and you want to move across too, just update the one-liner below accordingly. 😉

On your OLD PC

Open your Linux VM
Get inside your Downloads directory (replace <user> with your username): cd /mnt/c/Users/<user>/Dowloads
Make sure to be root (sudo su -)
Run:
tar -cvpzf backup.tar.gz --exclude=/backup.tar.gz --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/run --exclude=/tmp --exclude=/media --exclude=/mnt --exclude=/var/cache --exclude=/var/run --one-file-system /
NOTE: you could achieve the same using the option --exclude-from=file.txt, and having the list of exclusions in this file. I used a one-liner as it’s quicker to copy and paste.
Once done, close your Linux VM
Verify that you have a new file called backup.tar.bz in Dowloads

On your NEW PC

Install from Microsoft Store the same Linux VM (or reinstall in the same way you have done originally on your old pc)
Copy across your backup.tar.bz within your new Downloads folder
Open the VM that you’ve just installed (minimal setup – this will be completely overwritten, so don’t be bothered too much)
Once you’re inside and your backup.tar.bz is in Download, run the following (replace <user> with your username):
sudo tar -xpzf /mnt/c/Users/<user>/Dowloads/backup.tar.gz -C / --numeric-owner
Ignore the errors
Close and re-open the VM: DONE! 🙂

Happy migration! 😉

Backup Linux file/dir permissions

find /source/path/ -type d -exec ls -la '{}' \; > file_list
find /source/path/ -type d -exec ls -ld '{}' \; > dironly_list
find /source/path/ -type f -exec ls -l '{}' \; > filesonly_list

find /source/path/ -type d -exec ls -la '{}' \; > file_list

find /source/path/ -type d -exec ls -ld '{}' \; > dironly_list

find /source/path/ -type f -exec ls -l '{}' \; > filesonly_list

Holland backup setup

>> package:

yum install holland-mysqldump

1	yum install holland-mysqldump

>> Auto install script:

#!/bin/bash
which yum
rc=$?
if [ $rc != 0 ] ; then
    apt-get install -y python-setuptools python-mysqldb
else
    yum install -y MySQL-python
fi

if [ ! -f holland-1.0.10.tar.gz ] ; then
    wget http://hollandbackup.org/releases/stable/1.0/holland-1.0.10.tar.gz
fi
if [ ! -d holland-1.0.10 ] ; then
    tar zxf holland-1.0.10.tar.gz 
fi

cd holland-1.0.10
python setup.py install 

cd plugins/holland.lib.common/
python setup.py install

cd ../holland.lib.mysql/
python setup.py install

cd ../holland.backup.mysqldump/
python setup.py install

cd ../../config
mkdir -p /etc/holland/providers
mkdir -p /etc/holland/backupsets
cp holland.conf  /etc/holland/
cp providers/mysqldump.conf /etc/holland/providers/
cp backupsets/examples/mysqldump.conf /etc/holland/backupsets/default.conf

cd /etc/holland
sed -i 's=/var/spool/holland=/var/lib/mysqlbackup=g' holland.conf
sed -i 's/backups-to-keep = 1/backups-to-keep = 7/g' backupsets/default.conf
sed -i 's/file-per-database.*=.*no/file-per-database = yes/g' backupsets/default.conf
sed -i 's/file-per-database.*=.*no/file-per-database = yes/g' providers/mysqldump.conf

hour=$(python -c "import random; print random.randint(0, 4)")
minute=$(python -c "import random; print random.randint(0, 59)")
echo '# Dump mysql DB to files. Time was chosen randomly' > /etc/cron.d/holland
echo "$minute $hour * * * root /usr/local/bin/holland -q bk" >> /etc/cron.d/holland

mkdir -p /var/lib/mysqlbackup
mkdir -p /var/log/holland
chmod o-rwx /var/log/holland

echo 
echo
echo "Holland should be installed now."
echo "Test backup: holland bk"
echo "See cron job: cat /etc/cron.d/holland"
echo "See backups: find /var/lib/mysqlbackup/"

#!/bin/bash

which yum

rc=$?

if [ $rc != 0 ] ; then

apt-get install -y python-setuptools python-mysqldb

else

yum install -y MySQL-python

if [ ! -f holland-1.0.10.tar.gz ] ; then

wget http://hollandbackup.org/releases/stable/1.0/holland-1.0.10.tar.gz

if [ ! -d holland-1.0.10 ] ; then

tar zxf holland-1.0.10.tar.gz

cd holland-1.0.10

python setup.py install

cd plugins/holland.lib.common/

python setup.py install

cd ../holland.lib.mysql/

python setup.py install

cd ../holland.backup.mysqldump/

python setup.py install

cd ../../config

mkdir -p /etc/holland/providers

mkdir -p /etc/holland/backupsets

cp holland.conf /etc/holland/

cp providers/mysqldump.conf /etc/holland/providers/

cp backupsets/examples/mysqldump.conf /etc/holland/backupsets/default.conf

cd /etc/holland

sed -i 's=/var/spool/holland=/var/lib/mysqlbackup=g' holland.conf

sed -i 's/backups-to-keep = 1/backups-to-keep = 7/g' backupsets/default.conf

sed -i 's/file-per-database.*=.*no/file-per-database = yes/g' backupsets/default.conf

sed -i 's/file-per-database.*=.*no/file-per-database = yes/g' providers/mysqldump.conf

hour=$(python -c "import random; print random.randint(0, 4)")

minute=$(python -c "import random; print random.randint(0, 59)")

echo '# Dump mysql DB to files. Time was chosen randomly' > /etc/cron.d/holland

echo "$minute $hour * * * root /usr/local/bin/holland -q bk" >> /etc/cron.d/holland

mkdir -p /var/lib/mysqlbackup

mkdir -p /var/log/holland

chmod o-rwx /var/log/holland

echo

echo "Holland should be installed now."

echo "Test backup: holland bk"

echo "See cron job: cat /etc/cron.d/holland"

echo "See backups: find /var/lib/mysqlbackup/"

>> Where is the DB conf file:

/etc/holland/backupsets

1	/etc/holland/backupsets

>> When it runs

cat /etc/cron.d/holland

1	cat /etc/cron.d/holland

>> Test command to see if all works without actually run it

holland bk --dry-run

1	holland bk --dry-run

>> /etc/holland/backupsets/default.conf

[mysql:client]
defaults-extra-file = /root/.my.cnf

1 2	[mysql:client] defaults-extra-file = /root/.my.cnf

Rackspace Cloud – Remove old System IDs via command line

Rough script/instructions 🙂

>> set your variables:
TOKEN=""
REGION="lon"
DDI=""  < this is the account number

>> Generate a list of backup agents
curl -sH  "X-Auth-Token: $TOKEN" -H "Content-type: application/json" -X GET https://$REGION.backup.api.rackspacecloud.com/v1.0/$DDI/user/agents | python -m json.tool | egrep "MachineName|MachineAgentId" | awk -F":" '{print $2}' | sed 's/ //g' | sed '{N;s/\n//}' > list.txt

>> Manually remove WANTED backup agents (leave only the ones you want to remove):
vim list.txt 

>> Generate remove list
awk -F, '{print $1}' list.txt > remove.txt


>> generate the exec file to review
for AGENTID in `cat remove.txt`; do echo curl -sH \"X-Auth-Token: $TOKEN\" -H \"Content-type: application/json\" -X POST https://$REGION.backup.api.rackspacecloud.com/v1.0/$DDI/agent/delete -d \'{\"MachineAgentId\": $AGENTID}\' ; done >> exec_me

>> exec the API calls
/bin/bash exec_me

>> set your variables:

TOKEN=""

REGION="lon"

DDI="" < this is the account number

>> Generate a list of backup agents

curl -sH "X-Auth-Token: $TOKEN" -H "Content-type: application/json" -X GET https://$REGION.backup.api.rackspacecloud.com/v1.0/$DDI/user/agents | python -m json.tool | egrep "MachineName|MachineAgentId" | awk -F":" '{print $2}' | sed 's/ //g' | sed '{N;s/\n//}' > list.txt

>> Manually remove WANTED backup agents (leave only the ones you want to remove):

vim list.txt

>> Generate remove list

awk -F, '{print $1}' list.txt > remove.txt

>> generate the exec file to review

for AGENTID in `cat remove.txt`; do echo curl -sH \"X-Auth-Token: $TOKEN\" -H \"Content-type: application/json\" -X POST https://$REGION.backup.api.rackspacecloud.com/v1.0/$DDI/agent/delete -d \'{\"MachineAgentId\": $AGENTID}\' ; done >> exec_me

>> exec the API calls

/bin/bash exec_me

Rackspace Cloud Driveclient not working

First of all, checks the logs: /var/log/driveclient.log

You might find 403 errors and lines that are showing that the agent can’t connect properly.

In this case, the first step is trying to re-register the backup agent:
3) Maybe the customer has changed the API key so try re-register the backup agent:

# /usr/local/bin/driveclient --configure
WARNING: Agent already configured. Overwrite? [Y/n]: Y
Username: My_Username
Password: My_APIKey

# /usr/local/bin/driveclient --configure

WARNING: Agent already configured. Overwrite? [Y/n]: Y

Username: My_Username

Password: My_APIKey

Desired Output:

Registration successful!
Bootstrap created at: /etc/driveclient/bootstrap.json

1 2	Registration successful! Bootstrap created at: /etc/driveclient/bootstrap.json

In case you get something like “ERROR: Registration failed: Could not authenticate user. Identity returned 401“, this means that you probably need to force a bit the registration, using the following command:

# driveclient -u USER_NAME -k API_KEY -t LON -l raxcloudserver -a lon.backup.api.rackspacecloud.com -c

1	# driveclient -u USER_NAME -k API_KEY -t LON -l raxcloudserver -a lon.backup.api.rackspacecloud.com -c

Backup – rsnapshot and rdiff (multiple backups)

This is a very basic/simple guide about how to setup incremental and versioned backups of your Linux computers and Mac. 🙂

Initial problem:

- Time Machine is unreliable after a while, and when you put on sleep your Mac, most of the time it complains because the USB drive wasn’t disconnected properly :@
- I’d like to be able to have an incremental/versioning backup system local BUT also have some of critical files uploaded in the cloud [using some cron and some cloud provider’s utility]
- Time Machine on external drives uses ‘sparsebundle’ storage system, which is complicated to open and extract files from Linux command line [I’ve previously created a Time Machine on the pi, and I was thinking to create a sort of system to open the sparsebundle file, and upload the files during the night – but this doesn’t seem easy or neither really reliable]
- Backing up VMs with Time Machine takes ages, as if a little bit changes, the whole content gets copied over (space and time consuming)

So… I needed something that could:

Do incremental backups storing only the differences (for VMs) to avoid to transfer every time GBs of data for little changes
Do versioning of small files (documents, videos, music, etc…) based on a custom schedule
Be accessible on the filesystem without tricky stuffs (like opening a ‘sparsebundle’ file
Be able to run on a raspberry pi and mostly likely, able to access Linux and Mac systems, and have a centralised backup system.

Answer: combination of rsnapshot and rdiff-backups… plus some sort of Cloud Provider’s utility to sync part of this content on the Cloud (still work in progress).
I found this nice article where it explains the differences between the two tools, and it should clarify why I’ve chosen to use a combination of both of them and not just one.
The main bit is this one:

rdiff-backup stores previous versions as compressed deltas to the current version similar to a version control system. rsnapshot uses actual files and hardlinks to save space. For small files, storage size is similar. For large files that change often, such as logfiles, databases, etc., rdiff-backup requires significantly less space for a given number of versions.

So, I’ve installed rsnapshot and rdiff-backups on my pi. Packages are available using apt-get command.
After that, I have created one rsnapshot configuration file for each of my linux machines (actually pi’s) and one for my Mac. rdiff-backup will be called within rsnapshot, in a post-exec script (option available, and very handy).

It’s clearly necessary to have SSH enable on your Linux and Mac machines. Also, in this particular case, I have added the following in visudo on the Mac, to allow the user to run pmset passwordless:

user ALL=(ALL) NOPASSWD: /usr/bin/pmset

1	user ALL=(ALL) NOPASSWD: /usr/bin/pmset

Configuration files

I’m posting 2 configuration examples: one for my pi (local backup_, and the other onefor my Mac (remote backup – via ssh/rsync).
I’ve literally kept the original /etc/rsnapshot.conf just as reference – not actively using at all.

Here my custom configuration files:

/etc/default/rsnapshot

This is a file that I’ve created and I use it as “default/general” parameters that I include in any of the other custom files. Why? Just to avoid to copy and paste the same on any custom file 🙂

#####################################
# Default configuration paramenters #
#####################################
# just use include_conf &lt;tab&gt; file:
#include_conf /etc/default/rsnapshot
config_version 1.2
no_create_root 1
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_logger /usr/bin/logger
cmd_du /usr/bin/du
du_args -csh
link_dest 1
use_lazy_deletes 1
rsync_numtries 3
#stop_on_stale_lockfile 0

#####################################

# Default configuration paramenters #

#####################################

# just use include_conf <tab> file:

#include_conf /etc/default/rsnapshot

config_version 1.2

no_create_root 1

cmd_cp /bin/cp

cmd_rm /bin/rm

cmd_rsync /usr/bin/rsync

cmd_ssh /usr/bin/ssh

cmd_logger /usr/bin/logger

cmd_du /usr/bin/du

du_args -csh

link_dest 1

use_lazy_deletes 1

rsync_numtries 3

#stop_on_stale_lockfile 0

PI configuration file (local backup)

pi1_rsnap.conf

# pi1 conf file
include_conf /etc/default/rsnapshot
snapshot_root /USB/backups/pi1/
#retain hourly 6
retain daily 7
retain weekly 4
retain monthly 12
logfile /var/log/rsnapshot/p1.log
lockfile /USB/backups/rsnapshot_run/pi1.pid
#sync_first 1
verbose 2
loglevel 5
use_lazy_deletes 1
backup /home/ files/
backup /etc/ files/
backup /var/spool/cron/ files/
backup_script /usr/bin/dpkg --get-selections &gt; packages.txt installed-packages/

# pi1 conf file

include_conf /etc/default/rsnapshot

snapshot_root /USB/backups/pi1/

#retain hourly 6

retain daily 7

retain weekly 4

retain monthly 12

logfile /var/log/rsnapshot/p1.log

lockfile /USB/backups/rsnapshot_run/pi1.pid

#sync_first 1

verbose 2

loglevel 5

use_lazy_deletes 1

backup /home/ files/

backup /etc/ files/

backup /var/spool/cron/ files/

backup_script /usr/bin/dpkg --get-selections > packages.txt installed-packages/

This script copies home, etc, cron into /USB/backups/pi1/daily.0/files/.
The last line also execute the command and pull the output file and store within /USB/backups/pi1/daily.0/installed-packages/

The MAC configuration (remote backup).

This requires some extras.
What I’ve done is combining a pre and post script around the rsnapshot backup, in order to obtain the following:

waking up the MAC via wake-on-lan package (this is possible because my MAC is connected also via ethernet)
connect via ssh
send a command to keep the disk on and avoid them to go in idle
visually notify that the backup is about to run (in case someone is currently using the Mac)
run the rsnapshot backup
once finished, run rdiff-backup for the big files (VMs)
once done, kill the process that was keeping the disks on
visual notification sent to inform that backup has completed
disconnect. If no one is connected, the Mac will go back in standby (if enabled).
clean up old rdiff-backups

mac_rsnap.conf

# mac conf file
include_conf /etc/default/rsnapshot
snapshot_root /USB/backups/mac/
#retain hourly 6
#retain daily 7
retain weekly 4
retain monthly 12
logfile /var/log/rsnapshot/mac.log
lockfile /USB/backups/rsnapshot_run/mac.pid

#rsync_short_args -a
rsync_long_args --delete --numeric-ids --relative --delete-excluded --filter=". /etc/rsnapshot_configs/mac/<span style="color: #0000ff;">rsync_selections</span>"

#sync_first 1
verbose 1
loglevel 5
use_lazy_deletes 1

# Specify the path to a script (and any optional arguments) to run right
# before rsnapshot syncs files
<span style="color: #339966;">cmd_preexec</span> /etc/rsnapshot_configs/mac/<span style="color: #0000ff;">pre-exec.sh</span>

# Specify the path to a script (and any optional arguments) to run right
# after rsnapshot syncs files
<span style="color: #339966;">cmd_postexec</span> /etc/rsnapshot_configs/mac/<span style="color: #0000ff;">rdiff_vms.sh</span>

#Remote backup
</code><code>backup user@mac:/ files/

# mac conf file

include_conf /etc/default/rsnapshot

snapshot_root /USB/backups/mac/

#retain hourly 6

#retain daily 7

retain weekly 4

retain monthly 12

logfile /var/log/rsnapshot/mac.log

lockfile /USB/backups/rsnapshot_run/mac.pid

#rsync_short_args -a

rsync_long_args --delete --numeric-ids --relative --delete-excluded --filter=". /etc/rsnapshot_configs/mac/rsync_selections"

#sync_first 1

verbose 1

loglevel 5

use_lazy_deletes 1

# Specify the path to a script (and any optional arguments) to run right

# before rsnapshot syncs files

cmd_preexec /etc/rsnapshot_configs/mac/pre-exec.sh

# Specify the path to a script (and any optional arguments) to run right

# after rsnapshot syncs files

cmd_postexec /etc/rsnapshot_configs/mac/rdiff_vms.sh

#Remote backup

</code><code>backup user@mac:/ files/

The following bash scripts have some parameters that need to be set manually (highlighted in orange)

pre-exec.sh

#!/bin/bash

# --------------------------------------------- #
# This script wake up the mac box via ethernet
# using wake-on-lan, wait for ssh connection,
# connects and issue a command to keep the
# disks on for the following backup tasks.
#
# There is a timeout for number of tries. If
# reached, an email notification will be sent.
# --------------------------------------------- #

# Email parameters
EMAIL="<span style="color: #ff9900;">youremail@whatever.bla</span>"
SENDMAIL=<span style="color: #ff9900;">/usr/sbin/sendmail</span>

# MAC details
MACADDR="<span style="color: #ff9900;">xx:xx:xx:xx:xx:xx</span>"
USER=<span style="color: #ff9900;">user</span>
HOST=<span style="color: #ff9900;">mac</span>

# Estimated amount of time to get ssh available
waitBeforeTry=<span style="color: #ff9900;">40</span>

# Retries parameters
sleepSecInterval=5
maxConnectionAttempts=10

# --------------------------------------------- #
emailnotification () {
echo -e "Subject:$1\n" | $SENDMAIL $EMAIL
logger "${BASH_SOURCE[0]} PID $ - $1"
}

# Turn on your mac via Ethernet LAN
sudo /usr/sbin/etherwake $MACADDR

sleep $waitBeforeTry

index=1
while (( $index &lt;= $maxConnectionAttempts ))
do
echo quit | telnet $HOST 22 2&gt;/dev/null | grep -q Connected
if [ $? -ne 0 ] ; then
sleep $sleepSecInterval
((index+=1)) #; echo "DEBUG: $index"
else
break
fi
done

# Notify if reach max attempts
MSG="Unable to connect to $USER@$HOST after $maxConnectionAttempts attempts."
[ $index -eq $maxConnectionAttempts ] &amp;&amp; emailnotification $MSG

# Connect via ssh and disable sleep and disksleep
ssh $USER@$HOST 'sudo pmset sleep 0'
ssh $USER@$HOST 'sudo pmset disksleep 0'
#ssh $USER@$HOST 'nohup pmset noidle &gt; /dev/null 2&gt;&amp;1 &amp;'
ssh $USER@$HOST ' osascript -e '"'"'display notification "Starting Backup in few seconds" with title "Backup starts" sound name "default" '"'"' '

sleep 5

#!/bin/bash

# --------------------------------------------- #

# This script wake up the mac box via ethernet

# using wake-on-lan, wait for ssh connection,

# connects and issue a command to keep the

# disks on for the following backup tasks.

# There is a timeout for number of tries. If

# reached, an email notification will be sent.

# --------------------------------------------- #

# Email parameters

EMAIL="[email protected]"

SENDMAIL=/usr/sbin/sendmail

# MAC details

MACADDR="xx:xx:xx:xx:xx:xx"

USER=user

HOST=mac

# Estimated amount of time to get ssh available

waitBeforeTry=40

# Retries parameters

sleepSecInterval=5

maxConnectionAttempts=10

# --------------------------------------------- #

emailnotification () {

echo -e "Subject:$1\n" | $SENDMAIL $EMAIL

logger "${BASH_SOURCE[0]} PID $ - $1"

}

# Turn on your mac via Ethernet LAN

sudo /usr/sbin/etherwake $MACADDR

sleep $waitBeforeTry

index=1

while (( $index <= $maxConnectionAttempts ))

echo quit | telnet $HOST 22 2>/dev/null | grep -q Connected

if [ $? -ne 0 ] ; then

sleep $sleepSecInterval

((index+=1)) #; echo "DEBUG: $index"

else

break

done

# Notify if reach max attempts

MSG="Unable to connect to [email protected]$HOST after $maxConnectionAttempts attempts."

[ $index -eq $maxConnectionAttempts ] && emailnotification $MSG

# Connect via ssh and disable sleep and disksleep

ssh $USER@$HOST 'sudo pmset sleep 0'

ssh $USER@$HOST 'sudo pmset disksleep 0'

#ssh [email protected]$HOST 'nohup pmset noidle > /dev/null 2>&1 &'

ssh $USER@$HOST ' osascript -e '"'"'display notification "Starting Backup in few seconds" with title "Backup starts" sound name "default" '"'"' '

sleep 5

rdiff_vms.sh

#!/bin/bash

# Script executed after rsnapshot
USER=<span style="color: #ff9900;">user</span>
HOST=<span style="color: #ff9900;">mac</span>

# ===================================================
rdiff-backup --exclude-symbolic-links $USER@$HOST::Users/user/Documents/VMs/ /USB/backups/mac/VMs/

# All files should be now backed up

# Re-setting previous values for sleep and disksleep... and notify
ssh $USER@$HOST 'sudo pmset sleep 10'
ssh $USER@$HOST 'sudo pmset disksleep 10'
#ssh $USER@$HOST 'pkill pmset noidle'
ssh $USER@$HOST ' osascript -e '"'"'display notification "Backup has now completed." with title "Backup Finished" sound name "default" '"'"' '

# Putting on sleep the box - NOT REQUIRED
# sleep will happen automatically and no risk to force sleep if I'm using it
#ssh $USER@$HOST 'sudo pmset sleepnow'

# Cleaning up old backups: remove backups older than 6 months
rdiff-backup --remove-older-than 6M --force /USB/backups/mac/VMs/

#!/bin/bash

# Script executed after rsnapshot

USER=user

HOST=mac

# ===================================================

rdiff-backup --exclude-symbolic-links $USER@$HOST::Users/user/Documents/VMs/ /USB/backups/mac/VMs/

# All files should be now backed up

# Re-setting previous values for sleep and disksleep... and notify

ssh $USER@$HOST 'sudo pmset sleep 10'

ssh $USER@$HOST 'sudo pmset disksleep 10'

#ssh [email protected]$HOST 'pkill pmset noidle'

ssh $USER@$HOST ' osascript -e '"'"'display notification "Backup has now completed." with title "Backup Finished" sound name "default" '"'"' '

# Putting on sleep the box - NOT REQUIRED

# sleep will happen automatically and no risk to force sleep if I'm using it

#ssh [email protected]$HOST 'sudo pmset sleepnow'

# Cleaning up old backups: remove backups older than 6 months

rdiff-backup --remove-older-than 6M --force /USB/backups/mac/VMs/

The following file is the one used as ‘filter‘ for rsync. It uses that syntax.
To clarify, this does the backup of Documents, Pictures, Movies, Music folders ONLY from the user called ‘user‘, excluding the subfolders ‘VMs‘ in Documents, all the folders that starts with ‘Season‘ in Movies, any other possible folders in ‘user’ home dir, and any file/folder starting with .Spotlight, .Trash and .DS_Store files in ANY subfolders.

rsync_selections

+ Users/
+ Users/user/
+ Users/user/Documents/
+ Users/user/Pictures/
+ Users/user/Movies/
+ Users/user/Music/
- .Spotlight*
- .Trash*
- .DS_Store
- Users/user/Documents/VMs/
- Users/user/Movies/Season*/
- Users/user/*
- Users/*
- /*

+ Users/

+ Users/user/

+ Users/user/Documents/

+ Users/user/Pictures/

+ Users/user/Movies/

+ Users/user/Music/

- .Spotlight*

- .Trash*

- .DS_Store

- Users/user/Documents/VMs/

- Users/user/Movies/Season*/

- Users/user/*

- Users/*

- /*

/etc/cron.d/rsnapshot
This is the CRON that executes the backup jobs.
The ‘less frequent’ job needs to run before the ‘most frequent’. I’ve explained this later in this post, however the reason is that the actual active sync happens JUST in the most frequent job, and the others are just rotations made with a ‘mv’ command. So, it’s important to make the rotation BEFORE the sync.

###############
# &gt;&gt;&gt; MAC &lt;&lt;&lt; #
###############
# set to run only weekly at 10:30 am on Monday
30 10 * * 1 user /usr/bin/rsnapshot -c /etc/rsnapshot_configs/mac/mac_rsnap.conf weekly
# Monthly rotation at 10:00 am (1st every month)
0 10 1 * * user /usr/bin/rsnapshot -c /etc/rsnapshot_configs/mac/mac_rsnap.conf monthly
###############
# &gt;&gt;&gt; PI &lt;&lt;&lt; #
###############
# Daily 9:30am
30 9 * * * root /usr/bin/rsnapshot -c /etc/rsnapshot_configs/pi_rsnap.conf daily
# Weekly 9:05am (Sunday)
5 9 * * 7 root /usr/bin/rsnapshot -c /etc/rsnapshot_configs/pi_rsnap.conf weekly
# Monthly 9:00am (1st every month)
0 9 1 * * root /usr/bin/rsnapshot -c /etc/rsnapshot_configs/pi_rsnap.conf monthly

###############

# >>> MAC <<< #

###############

# set to run only weekly at 10:30 am on Monday

30 10 * * 1 user /usr/bin/rsnapshot -c /etc/rsnapshot_configs/mac/mac_rsnap.conf weekly

# Monthly rotation at 10:00 am (1st every month)

0 10 1 * * user /usr/bin/rsnapshot -c /etc/rsnapshot_configs/mac/mac_rsnap.conf monthly

###############

# >>> PI <<< #

###############

# Daily 9:30am

30 9 * * * root /usr/bin/rsnapshot -c /etc/rsnapshot_configs/pi_rsnap.conf daily

# Weekly 9:05am (Sunday)

5 9 * * 7 root /usr/bin/rsnapshot -c /etc/rsnapshot_configs/pi_rsnap.conf weekly

# Monthly 9:00am (1st every month)

0 9 1 * * root /usr/bin/rsnapshot -c /etc/rsnapshot_configs/pi_rsnap.conf monthly

Folders created:

/USB/                               [mount point of my external USB drive]
/USB/backups/                       [subfolder to keep all the backups]
/USB/backups/pi/                    [folder for 'pi' box]
/USB/backups/mac/                   [folder for 'mac']
/etc/rsnapshot_configs/             [where I keep all the conf files]
/var/log/rsnapshot/                 [log files - chmod 1777*]
/USB/backups/rsnapshot_run/         [dir for jobs' pids - chmod 1777*]

/USB/ [mount point of my external USB drive]

/USB/backups/ [subfolder to keep all the backups]

/USB/backups/pi/ [folder for 'pi' box]

/USB/backups/mac/ [folder for 'mac']

/etc/rsnapshot_configs/ [where I keep all the conf files]

/var/log/rsnapshot/ [log files - chmod 1777*]

/USB/backups/rsnapshot_run/ [dir for jobs' pids - chmod 1777*]

*Use chmod 1777 on logs and run folders if you want other users than root to run the backups and write log files.

Let’s clarify some bits and pieces

sync_first 1

To be sure to properly complete the first full backup, enable sync_first setting this to 1. Once completed, remove/comment it out.
To execute the first sync, run the following:

rsnapshot -c my_rsnapshot.conf sync

1	rsnapshot -c my_rsnapshot.conf sync

Basically, run the sync as many times you want… and once you have finished, you will start invoking (with CRON) the daily, weekly, monthly… etc backups. REMEMBER to disable it once finished, otherwise you won’t actually run any sync!

TABs no spaces!

IMPORTANT: do NOT use spaces in the rsnapshot configuration files but only TABS!!!
Copy and paste might change tabs to spaces so be sure to review all your configs. Use the -t flag to test every time if syntax is correct.

Test your configuration (-t)

rsnapshot -t -c my_rsnapshot.conf <sync|daily|weekly... >

1	rsnapshot -t -c my_rsnapshot.conf <sync\|daily\|weekly... >

The -t will also display exactly the command that it’s going to be executed – very handy! 🙂

Remote backups

Another thing to keep in mind is that ‘REMOTE’ backups (whatever uses [email protected] …) are actually launching the command on the remote host so it’s required to have rsync installed on the remote machine too (and rdiff-backup if used too). Versions should also match. If not, at least rsync should be version >= 3.
To allow this to work on my Mac, for instance, I had to install “rdiff-backup” and install a newer version of “rsync”, as the default version is 2.6.x. I’ve used the Rudix packages. Easy easy 🙂

Retain daily/weekly/monthly… sync… wtf?!

Very important to understand about rsnapshot that made me kinda mad for few hours: the job that DOES the backup is the one on the top of the list (most frequent).
So, if you have daily, weekly, monthly… set as ‘retain’ parameters in the rsnapshot conf file, the one that does actually the copy of the files is ‘daily‘ (top of the list – most frequent). The other ones are JUST some sort of rotation of the folder tree. Literally a ‘mv’ command… that’s it. You can verify this using -t flag to see the commands.
So, don’t get confused 🙂

So, to summarise:

sync: first initial backup – handy especially to create the initial backup. This creates a .sync folder in snapshot_root.
daily: this is the one that does the copy (or the ‘most frequent’ backup set – in mac for example, I set that to be ‘weekly’ and ‘monthly’ only, so in that case, weekly is the most frequent backup set and it’s the one that does the sync
weekly/monthly… (less frequent backups): these are simply ‘mv’ commands.

To explain more in details… the flow of my Mac…
You run the first sync (as many times as you want), with ‘sync_first‘ enabled.

rsnapshot -c my_rsnapshot.conf sync

1	rsnapshot -c my_rsnapshot.conf sync

This creates the backup in /USB/backups/mac/.sync/
Than you run the crons. Weekly will be the first to run:

rsnapshot -c my_rsnapshot.conf weekly

1	rsnapshot -c my_rsnapshot.conf weekly

This will actually run this move, creating the first weekly folder:

mv /USB/backups/mac/.sync/ /USB/backups/mac/weekly.0/

1	mv /USB/backups/mac/.sync/ /USB/backups/mac/weekly.0/

Than, DISABLE ‘sync_first’ and the next time the weekly cron will be executed, something like that will run, moving the weekly.0 to weekly.1, hard linking the identical files and sync’ing the ones that have been changed since:

mv /USB/backups/mac/weekly.0/ /USB/backups/mac/weekly.1/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
    --link-dest=/USB/backups/mac/weekly.1/files/ /home/ \
    /USB/backups/mac/weekly.0/files/
[...]

mv /USB/backups/mac/weekly.0/ /USB/backups/mac/weekly.1/

/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \

--link-dest=/USB/backups/mac/weekly.1/files/ /home/ \

/USB/backups/mac/weekly.0/files/

[...]

Then, next time, weekly.2 and weekly.3 will be created: same method.
Until the LAST backup is created (#3, in this case -> 4 retention – from 0 to 3), the monthly job won’t take any affects.
Once we have /USB/backups/mac/weekly.3/, and this will be executed…

rsnapshot -c my_rsnapshot.conf monthly

1	rsnapshot -c my_rsnapshot.conf monthly

… this will be executed:
mv /USB/backups/mac/weekly.3/ /USB/backups/mac/monthly.0/

And so and so…

Little note, keeping the above example. You might start this backup in the middle of month, so at the end of the month you won’t have reached the 4th weekly backup sets, but just the 2nd (#0 and #1). So.. what happens with the ‘monthly’ one that will run on the 1st of the month?
Answer: nothing.
Basically, this time the monthly backup will skip as the previous max retention limit is not reached yet. Weekly backups will continue rotating within themselves.
The first week of the second month, weekly backup will reach #2 (third backup). #1 => #2, #0 => #1 and the new backup stored in #0.
Second week #3 (4th and last). #2 => #3, #1 => #2, #0 => #1 and the new backup stored in #0. The #3 (oldest) should be the one that rotates… but the monthly cron won’t be executed until the next month. But there’s nothing to be worried about. Next weekly run, on the third week, the #3 will be marked for deletion, and a new #0 will be created. Same for the forth week. Oldest backup deleted, max limit reached.
And here, we will get into the new month, where the monthly backup will be called BEFORE the weekly one, and it will rotate weekly.3 in monthly.0, and the weekly (#3 => monthly#0, #2 => #3, #1 => #2) freeing up ‘one space’ (#0). This will be filled up from the next ‘weekly’ run, and all will be ‘in sync’ for the next months. 🙂

I hope this example clarifies. 🙂

NOTE:
If you are decide, one day, to move your backup from one disk to another one, MAKE SURE to rsync preserving the hard links, otherwise your backup will raise like a cake in the oven! 🙂

Here a sample command:

rsync -az -H --delete --numeric-ids /path/to/source server2:/path/to/dest/

1	rsync -az -H --delete --numeric-ids /path/to/source server2:/path/to/dest/

SSL PASSIVE FTP with virtual users on Raspberry Pi

I found this handy plugin to backup my blog: BackWPup
It has also an interesting feature which is the ability to backup remotely, for example on a FTP server.

So… here we go! 🙂

Few notes:

This uses vsftpd software
It will work ONLY over SSL
Due to SSL encryption, the FTP will also work ONLY in PASSIVE mode (ACTIVE mode is disabled)
This configuration has been made based of the fact that this raspberry pi is behind a router
This will use ONLY virtual users, chroot’ed, to increase the security (vsftpd will use a custom PAM auth file, which won’t lookup in /etc/passwd files – for this reason, any local user attempts to login will fail)
Virtual users usernames and credentials will be stored in a file
There is a workaround in place to avoid some common issues like “500 OOPS: Vsftpd: Refusing to Run With Writable Root Inside Chroot ()” – FYI, allow_writeable_chroot=yes does NOT work on vsftpd version 2.3.5.

Install required packets:

apt-get install vsftpd apache2-utils libpam-pwdfile

1	apt-get install vsftpd apache2-utils libpam-pwdfile

Create SSL certificate:

openssl req -x509 -nodes -days 7300 -newkey rsa:2048 -keyout /etc/ssl/certs/vsftpd.pem -out /etc/ssl/certs/vsftpd.pem
chmod 600 /etc/ssl/certs/vsftpd.pem

1 2	openssl req -x509 -nodes -days 7300 -newkey rsa:2048 -keyout /etc/ssl/certs/vsftpd.pem -out /etc/ssl/certs/vsftpd.pem chmod 600 /etc/ssl/certs/vsftpd.pem

Add a local user with limited access (like no console) that vsfpd will use to run virtual users:

useradd --home /home/vsftpd --gid nogroup -m --shell /bin/false vsftpd

1	useradd --home /home/vsftpd --gid nogroup -m --shell /bin/false vsftpd

Create directory structures for the virtual users:

mkdir -p /space/ftpusers/
chmod a-w /space/ftpusers/
mkdir -p /space/ftpusers/ftp01/rw
chmod a-w /space/ftpusers/ftp01
chown -R vsftpd:nogroup /space/ftpusers/ftp01

mkdir -p /space/ftpusers/

chmod a-w /space/ftpusers/

mkdir -p /space/ftpusers/ftp01/rw

chmod a-w /space/ftpusers/ftp01

chown -R vsftpd:nogroup /space/ftpusers/ftp01

Please note that all new virtual users added need its home directory manually created as per above. Also, due to the chroot option and the current limitation on vsftpd, if you want a user to be able to write in its home directory, you need to create an extra folder. Its root home folder has to be -w. This is a workaround that works 🙂

Setup PAM authentication

Create a new file /etc/pam.d/vsftpd.virtual and add the following:

auth required pam_pwdfile.so pwdfile /etc/vsftpd/vsftpd.users
account required pam_permit.so

1 2	auth required pam_pwdfile.so pwdfile /etc/vsftpd/vsftpd.users account required pam_permit.so

Now, let’s reorder a bit vsftp files in a directory:

mkdir -p /etc/vsftpd
cd /etc/
mv vsftpd.conf vsftpd
ln -s /etc/vsftpd/vsftpd.conf .

mkdir -p /etc/vsftpd

cd /etc/

mv vsftpd.conf vsftpd

ln -s /etc/vsftpd/vsftpd.conf .

Add new users (password max 8 characters):

htpasswd -c -d -b /etc/vsftpd/vsftpd.users ftp01 ftp01password

1	htpasswd -c -d -b /etc/vsftpd/vsftpd.users ftp01 ftp01password

Use the flag -c only the first time to create the file. If you re-use it, the file will be overwritten!
Also the -d flag is required because vsftpd is unable to read MD5 hashed password (default if -d is not used). The downside of this is a password limited to 8 characters.
Openssl could be used to produce a MD5 based BSD password with algorithm 1 using # openssl passwd -1 (not tested)

Let’s configure vsftpd

vi /etc/vsftpd.conf

# Main Settings
listen=YES
listen_port=21
connect_from_port_20=NO
ftpd_banner=Welcome to my FTP :-)
use_localtime=YES
force_dot_files=YES

# FTP Passive settings
pasv_enable=YES
pasv_min_port=1100
pasv_max_port=1150
pasv_addr_resolve=YES
pasv_enable=YES
pasv_addr_resolve=YES
pasv_address=<span style="color: #ff0000;">&lt;EXTERNAL IP or DYN DNS&gt;</span>

# Virtual user settings
local_enable=YES
chroot_local_user=YES
secure_chroot_dir=/var/run/vsftpd/empty
virtual_use_local_privs=YES
guest_enable=YES
guest_username=vsftpd
pam_service_name=vsftpd.virtual
user_sub_token=$USER
local_root=/space/ftpusers/$USER
hide_ids=YES

# Anonymous settings
anonymous_enable=NO
anon_upload_enable=NO
no_anon_password=NO
anon_other_write_enable=NO
anon_mkdir_write_enable=NO

# Write permissions
write_enable=YES
local_umask=022
async_abor_enable=YES

# SSL
ssl_enable=YES
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH
rsa_cert_file=/etc/ssl/certs/vsftpd.pem

# Logging
xferlog_enable=YES
log_ftp_protocol=NO
syslog_enable=NO
vsftpd_log_file=/var/log/vsftpd.log

vi /etc/vsftpd.conf

# Main Settings

listen=YES

listen_port=21

connect_from_port_20=NO

ftpd_banner=Welcome to my FTP :-)

use_localtime=YES

force_dot_files=YES

# FTP Passive settings

pasv_enable=YES

pasv_min_port=1100

pasv_max_port=1150

pasv_addr_resolve=YES

pasv_enable=YES

pasv_addr_resolve=YES

pasv_address=<EXTERNAL IP or DYN DNS>

# Virtual user settings

local_enable=YES

chroot_local_user=YES

secure_chroot_dir=/var/run/vsftpd/empty

virtual_use_local_privs=YES

guest_enable=YES

guest_username=vsftpd

pam_service_name=vsftpd.virtual

user_sub_token=$USER

local_root=/space/ftpusers/$USER

hide_ids=YES

# Anonymous settings

anonymous_enable=NO

anon_upload_enable=NO

no_anon_password=NO

anon_other_write_enable=NO

anon_mkdir_write_enable=NO

# Write permissions

write_enable=YES

local_umask=022

async_abor_enable=YES

# SSL

ssl_enable=YES

force_local_data_ssl=YES

force_local_logins_ssl=YES

ssl_tlsv1=YES

ssl_sslv2=NO

ssl_sslv3=NO

require_ssl_reuse=NO

ssl_ciphers=HIGH

rsa_cert_file=/etc/ssl/certs/vsftpd.pem

# Logging

xferlog_enable=YES

log_ftp_protocol=NO

syslog_enable=NO

vsftpd_log_file=/var/log/vsftpd.log

Now, on your router, make sure that the module ip_conntrack_ftp is loaded using lsmod command.
This is required for FTP PASSIVE mode to work.
I’ve realised that this can be called also nf_conntrack_ftp.
A good way to check all the alias associated to that netfilter module is using the following command:

# modinfo nf_conntrack_ftp
filename: /lib/modules/3.3.7/kernel/net/netfilter/nf_conntrack_ftp.ko
alias: nfct-helper-ftp
alias: <span style="color: #ff0000;">ip_conntrack_ftp</span>
description: ftp connection tracking helper
author: Rusty Russell &lt;rusty@rustcorp.com.au&gt;
license: GPL
depends: nf_conntrack
intree: Y
vermagic: 3.3.7 mod_unload MIPS32_R1 32BIT
parm: ports:array of ushort
parm: loose:bool

# modinfo nf_conntrack_ftp

filename: /lib/modules/3.3.7/kernel/net/netfilter/nf_conntrack_ftp.ko

alias: nfct-helper-ftp

alias: ip_conntrack_ftp

description: ftp connection tracking helper

author: Rusty Russell <rusty@rustcorp.com.au>

license: GPL

depends: nf_conntrack

intree: Y

vermagic: 3.3.7 mod_unload MIPS32_R1 32BIT

parm: ports:array of ushort

parm: loose:bool

Also, make sure to setup a port forwarding like as below:

$IPT -t nat -A PREROUTING -p tcp -i $EXTIF -d $EXTIP --dport 21 -j DNAT --to $FTPIP:21 # FTP connection port
$IPT -t nat -A PREROUTING -d $EXTIP -p tcp -m tcp --dport 1100:1150 -j DNAT --to-destination $FTPI # FTP PASS ports
$IPT -A FORWARD -i $EXTIF -d $FTPI -p tcp --dport 21 -j ACCEPT
$IPT -A FORWARD -i $EXTIF -d $FTPI -p tcp --dport 1100:1150 -j ACCEPT

$IPT -t nat -A PREROUTING -p tcp -i $EXTIF -d $EXTIP --dport 21 -j DNAT --to $FTPIP:21 # FTP connection port

$IPT -t nat -A PREROUTING -d $EXTIP -p tcp -m tcp --dport 1100:1150 -j DNAT --to-destination $FTPI # FTP PASS ports

$IPT -A FORWARD -i $EXTIF -d $FTPI -p tcp --dport 21 -j ACCEPT

$IPT -A FORWARD -i $EXTIF -d $FTPI -p tcp --dport 1100:1150 -j ACCEPT

Backup Raspberry Pi SD on your Mac… and restore.

Plug the SD in your Mac.

In the Terminal, as root, use diskutil to identify your SD.
Generally it’s the last in the list, if you’ve just plugged in.

root:~ # diskutil list

1	root:~ # diskutil list

You will see something like this:

In my case, the SD is /dev/disk4. For this reason, I run the following to unmount the whole disk.

root:~ # diskutil umountDisk /dev/disk4
Unmount of all volumes on disk4 was successful

1 2	root:~ # diskutil umountDisk /dev/disk4 Unmount of all volumes on disk4 was successful

Once done, you can create the backup using dd utility, but make sure to change the device from /dev/diskX to /dev/rdiskX, adding the “r“.

root:~ # dd if=/dev/<span style="color: #ff0000;">rdisk4</span> of=/path/to/mypibackup.img bs=1m

1	root:~ # dd if=/dev/<span style="color: #ff0000;">rdisk4</span> of=/path/to/mypibackup.img bs=1m

To restore, of course… invert if (input file) with of (output file)… 🙂

root:~ # dd if=/path/to/mypibackup.img of=/dev/rdisk4 bs=1m

1	root:~ # dd if=/path/to/mypibackup.img of=/dev/rdisk4 bs=1m

My notepad

Chris' IT notes…

Tag Archives: backup

Migrate Linux Subsystem from one PC to another

On your OLD PC

On your NEW PC

Backup Linux file/dir permissions

Holland backup setup

Rackspace Cloud – Remove old System IDs via command line

Rackspace Cloud Driveclient not working

Backup – rsnapshot and rdiff (multiple backups)

Initial problem:

So… I needed something that could:

Configuration files

PI configuration file (local backup)

The MAC configuration (remote backup).

Let’s clarify some bits and pieces

sync_first 1

TABs no spaces!

Test your configuration (-t)

Remote backups

Retain daily/weekly/monthly… sync… wtf?!

SSL PASSIVE FTP with virtual users on Raspberry Pi

Backup Raspberry Pi SD on your Mac… and restore.