Friday, April 25, 2014

Building a Raspberry Pi NAS - Part 3: Storing files encrypted in the cloud

This post is part of a series on building the ultimate NAS system with a raspberry pi, creating your own "cloud".
In this post I'll present a way to make sure you do not lose your important files.
For this, we'll setup a personal account on a cloud service provider and then mount the space they give us on the raspberry pi's file system. The files will be replicated remotely to this service to prevent data loss if local file systems fail.
This storage space will also be used for very important and sensitive data. For this reason, it will be fully encrypted so that not even the cloud sys admins could read it.
Alternative titles:
- Mount on raspberry pi
- Encrypt files with encFs on raspberry pi

Part 1: Getting a reliable remote storage space

First get an account on (this is a cloud storage provider). Personal accounts are free and give you 10GB.  I chose them because they support the webdav protocol, unlike google drive which would be my first option. Other possible provider could be HiDrive
. In theory you could use any remote storage space provider that supports some protocol that you can use for mounting.
Now let's create a mount point. In this series I am mounting the different filesystems on ~/mnt. For this create a ~/mnt/box1 folder.
cd ~/mnt
mkdir box1

Now install the necessary packages to mount a webdav fs.
sudo apt-get install davfs2

We want this to be fully automated and automounted on reboot. For this, we need to store the user and password you used for the account in a file.
Create folder ~/.davfs2
cd ~
mkdir .davfs2
Inside this folder, edit file called secrets
cd .davfs2
nano secrets
and insert a single line like this: username password
Where username / password are replaced by your own
Now we need to add the pi user to the davfs2 group so that he can mount these guys.
sudo usermod -aG davfs2 pi
Also do the reconfig:
sudo dpkg-reconfigure davfs2
Select yes on the screen that appears.
Edit /etc/fstab as root and insert the following line: /home/pi/mnt/box1  davfs  rw,noexec,noauto,user,async,_netdev,uid=pi,gid=pi  0  0
The _netdev specifies that this fs is only mounted if there is a network connection. You should test this. Logout and login again (so that your user gets the new group), and try to mount the file system with the following command:
mount /home/pi/mnt/box1
You will most likely get an error: /sbin/mount.davfs: file /home/pi/.davfs2/secrets has wrong permissions This is because the user pi is still not in the group above. Reboot the pi.
sudo shutdown -r now
Now that the pi is rebooted, go to the .davfs2 folder and change permissions of the secrets file. If you do not do it, you'll get the following error: "/sbin/mount.davfs: file /home/pi/.davfs2/secrets has wrong permissions"
cd ~/.davfs2
chmod 600 secrets
Now, try to mount the file system:
mount /home/pi/mnt/box1
if everything ok you can now copy files from/to /home/pi/mnt/box1 and they are being stored remotely.
Note: I have found that automounting this with the auto option in etc/fstab is not very reliable. For this reason, we'll use an init script below to deal with the auto mounting on boot.

Part 2: Encrypting the files

To encrypt the files we'll use EncFS. This is a filesystem that allows you to mount another existing mount point. When you create files on the EncFs file system, they are transparently encrypted with a password you have chosen and stored (encrypted) in the underlying file system. We'll use the file system created previously as the underlying storage.
Create a new mount point for the encrypted. I'll call it encbox1
cd ~/mnt
mkdir encbox1
Install encfs
sudo apt-get install encfs
To be able to mount fuse based file systems (like encfs) you need to be on the fuse group.
sudo gpasswd -a pi fuse
Now it is easy to create the encrypted virtual file system. Use the following command:
encfs ~/mnt/box1 ~/mnt/encbox1
This will use ~/mnt/box1 as an encrypted storage for whatever you create on encbox1.
Let's test by creating a file on encbox1 and then listing content of encbox1 and box1:
echo "test" > ~/mnt/encbox1/teste.txt
ls -lah ~/mnt/encbox1/
total 1,0K
drwxr-xr-x 3 pi pi 160 Abr 26 20:20 .
-rw-r--r-- 1 pi pi   5 Abr 26 20:11 teste.txt
ls -lah ~/mnt/box1/
total 2,5K
drwxr-xr-x 3 pi pi  160 Abr 26 20:20 .
-rw-r--r-- 1 pi pi 1,1K Abr 26 20:09 .encfs6.xml
drwx------ 2 pi pi    0 Abr 26 00:22 lost+found
-rw-r--r-- 1 pi pi   13 Abr 26 20:11 xbeDOz08kT4LmeJpXZmNAnky
Notice that the file is slightly larger after encryption (13Bytes) while the plain one is 5. This is because there is a overhead of +8Bytes when per file Initialization vectors are used. More with man encfs.

Part 3: Automounting encfs

As it turns out, automounting an encfs is not as simple as putting a line on /etc/fstab. We will create an init script for this and then set it to run automatically on boot. Create your script on /etc/init.d/boxencfs
Also, you need to put encfs password on a file only you can read.
cd ~
mkdir .encfs
cd .encfs
nano box1
(now insert your password and ctrl+o, ctrl+x)
chmod 700 box1
sudo nano /etc/init.d/boxencfs
And insert the following code:
#! /bin/bash
# /etc/init.d/boxencfs

# Provides:          boxencfs
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Simple script to mount via webdav and an encfs on top of it
# Description:       Mount via webdav and encfs on top of it.

# If you want a command to always run, put it here

# Carry out specific functions when asked to by the system
case "$1" in
    echo "Mounting"
    su pi -c "mount /home/pi/mnt/box1"
    echo "Mounting encfs"
    su pi -c "cat /home/pi/.encfs/box1 | encfs -S /home/pi/mnt/box1 /home/pi/mnt/encbox1"
    echo "Unmounting encfs"
    su pi -c "fusermount -u /home/pi/mnt/encbox1"
    echo "Unmounting"
    umount /home/pi/mnt/box1
    echo "Usage: /etc/init.d/boxencfs {start|stop}"
    exit 1

exit 0
This script, when passed the start argument, first mounts the box1 system on /home/pi/mnt/box1 and then mounts the encfs. The password for the encfs is stored on /home/pi/.encfs/box1 ( a file created previously ) and then piped to the encfs. -S argument to encfs makes it read the password from stdin. With the stop argument it does the opposite in opposite order. After saving this file, make it executable:
sudo chmod +x /etc/init.d/boxencfs
Test this file very well by starting it and stopping it. A problem here may halt your reboot process. Test that it stops
sudo /etc/init.d/boxencfs stop
Now test that it starts. After this, check your encbox1 and box1 folders to see if they have the files expected.
sudo /etc/init.d/boxencfs start
When you feel this is fine, set it to run on boot/shutdown
sudo update-rc.d boxencfs defaults
And now reboot your pi and pray that it doesn't hang on reboot :)
sudo shutdown -r now

Part 4: schedule an rsync to copy files to box1

At this moment we have a folder with 10GB capacity and anything we put there is encrypted on the fly and sent to servers. This folder will be where we will store a copy of our important files. To see this you can run df -h:
Filesystem               Size  Used Avail Use% Mounted on
rootfs                   7,1G  2,3G  4,5G  34% /
/dev/root                7,1G  2,3G  4,5G  34% /
devtmpfs                 211M     0  211M   0% /dev
tmpfs                     44M  416K   44M   1% /run
tmpfs                    5,0M     0  5,0M   0% /run/lock
tmpfs                     88M     0   88M   0% /run/shm
/dev/mmcblk0p1            56M   19M   38M  34% /boot
/dev/sda3                 12G  2,5G  8,5G  23% /home/pi/mnt/media1
/dev/sdb1                 30G  2,0G   26G   7% /home/pi/mnt/media2
/dev/sda2                9,8G  266M  9,0G   3% /home/pi/mnt/home   10G     0   10G   0% /home/pi/mnt/box1
encfs                     10G     0   10G   0% /home/pi/mnt/encbox1
In my case, I want to make a backup of /home/pi/mnt/home (which is a mount point for an external USB drive (see previous parts of this tutorial)) into /home/pi/mnt/encbox1. The command you want to run is:
cd ~/mnt
rsync -rtvuh --progress --delete home encbox1
This syncs whatever is in ~/mnt/home into ~/mnt/encbox1. -r means recursive, t to preserve modification times, v for verbose, h for human readable sizes. --delete makes files be deleted on encbox1 if they were deleted on home.
crontab -e

Now we setup cron so that this command is executed every day at 12:00. For this, type:
crontab -e
An editor appears, in my case it is nano. Insert a line like the following:
0 12 * * * rsync -rtvuhW --progress --delete --inplace /home/pi/mnt/home/ /home/pi/mnt/encbox1/ >> /home/pi/.log/boxbackups.log
This line has the following semantic: At minute 0 of hour 12, every day of the month, every month of the year, every day of the week. run the rsync command. The rsync flags are important. without --inplace, rsync creates all files with 0 bytes. This is a known issue when rsyncing to a network folder. Also, all paths are absolute, to avoid issues and the output is concatenated to a log file (/home/pi/.log/boxbackups.log) so that we can later see what was backed up.

References: :P

Sunday, April 20, 2014

Backup raspberry pi sd card

Raspberry Pi's SD card can easily become corrupted which prevents pi booting. It has happened to me on a power failure. Best way to avoid this is to back it up. Assuming you have an usb drive mounted on /home/pi/mnt/home/ you can create a bzip2 archive of the sd card with :
sudo dd if=/dev/mmcblk0 | bzip2 > /home/pi/mnt/home/sdimage.bz2
This will take a lot of time. It will create a single file with the sd card image. Later, you can restore the image to the SDcard with the following:
sudo bzip2 -dc /home/pi/mnt/home/sdimage.bz2 | dd bs=1M of=/dev/mmcblk0
-d means decompress
-c means output to standard output
Another way to back it up via tar
tar -cvpzf backup.tar.gz --exclude=/backup.tar.gz --one-file-system / 
sudo tar -xvpzf /path/to/backup.tar.gz -C /media/whatever --numeric-owner