Setting up server with encrypted LVM without KVM

or: how i moved my data from home


1.Introduction

Over the years, I stored all my data on home server. At some point I just got tired of maintaining all this hardware and I decided to move critical data to data center. I wanted to have a fully encrypted disk, so the search for cheap dedicated server begins..


2.Choosing hosting provider

And after some searching, I settled on the hosting provider called Kimsufi. For this money you get desktop hardware without IP-KVM and without option to install system on encrypted partition via web console. But this administration console provides the ability to boot from the debian rescue disk..


3.Installing Linux

The installation process is to boot into the rescue disk, manually partition the disk, setup encrypted LVM, and then install a debian-based system with debootstrap.

3.1. Disk

Once connected you need to create two primary partitions: a small one for /boot at the beginning (~200MB) and a second one with the remaining space.

Main partition on my system is “/dev/sda5”. Setup the LUKS header with:

cryptsetup -s 512 -c aes-xts-plain64 luksFormat /dev/sda5

Unlock it:

cryptsetup luksOpen /dev/sda5 sda5_crypt

Now we can create physical volume and volume group:

pvcreate /dev/mapper/sda5_crypt
vgcreate vg0 /dev/mapper/sda5_crypt

Create logical volumes:

lvcreate -L 20g -n root vg0
lvcreate -L 4g -n swap vg0
lvcreate -l 100%FREE -n storage vg0

Format the partitions, and mount it:

mkfs.ext4 /dev/vg0/root
mkfs.ext4 /dev/vg0/storage
mkswap /dev/vg0/swap
mkdir /target/
mount /dev/vg0/root /target
mkdir /target/boot
mount /dev/sda1 /target/boot
swapon /dev/mapper/vg0-swap

3.2.Debootstrap

I want to intall latest LTS version of Ubuntu, but debootstrap package doesn’t have script for “Trusty” in /usr/share/debootstrap/scripts/ directory, and it’s read-only directory. So:

cp -r /usr/share/debootstrap/scripts/ /mnt/
mount -t tmpfs tmpfs /usr/share/debootstrap/scripts/
cp -r /mnt/scripts/* /usr/share/debootstrap/scripts/
ln -s /usr/share/debootstrap/scripts/gutsy \ /usr/share/debootstrap/scripts/trusty

And finally — debootstrap!

debootstrap —arch amd64 \
—components=main,restricted,universe,multiverse \
trusty /target http://debian.mirrors.ovh.net/ubuntu

Copying SSH keys in the target system:

mkdir /target/root/.ssh/
cp ~/.ssh/authorized_keys /target/root/.ssh/authorized_keys

I highly recommend to write static ip address, because DHCP lease time at Kimsufi ~5 minutes.

/target/etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address <IP>
netmask <MASK>
gateway <GATEWAY>
dns-nameservers 8.8.8.8

Mount some file systems and enter the target system with chroot:

mount -o bind /dev /target/dev
mount -t proc proc /target/proc
mount -t sysfs sys /target/sys
chroot /target /bin/bash

3.3.Now we are in the target system

Adding information about patitions

echo 'sda5_crypt UUID=xxxxxxxx none luks,discard' > /etc/crypttab

/etc/fstab:

# path mp type opt dump pass
/dev/vg0/root / ext4 errors=remount-ro 0 1
/dev/sda1 /boot ext2 defaults 0 2
/dev/vg0/storage /storage ext4 rw,nosuid,nodev 0 2
/dev/vg0/swap none swap sw 0 0

Now we can install the necessary software for the OS and a couple of packages to make life easier:

apt-get install ubuntu-standard ubuntu-minimal locales software-properties-common aptitude ssh dropbear vim screen git htop linux-image-3.13.0-24-generic linux-image-extra-3.13.0-24-generic linux-image-generic cryptsetup lvm2 grub-pc

Copying SSH keys from system to initramfs.

cp ~/.ssh/authorized_keys /etc/initramfs-tools/root/.ssh/authorized_keys

Configuring over DHCP usually does not work in initramfs, so you need to specify the network settings manually.

echo 'export IP=<IP>::<GATEWAY>:<MASK>:<HOSTNAME>:eth0:off' \ >/etc/initramfs-tools/conf.d/network_config 

Adding modules

echo "aes" >> /etc/initramfs-tools/modules
echo "dm-crypt" >> /etc/initramfs-tools/modules
echo "sha256" >> /etc/initramfs-tools/modules

Now, edit one of the most important files
/etc/default/grub:

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=2
GRUB_RECORDFAIL_TIMEOUT=$GRUB_TIMEOUT
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
I want to pay attention to the parameter GRUB_RECORDFAIL_TIMEOUT.
By default, after failed boot server stays in the GRUB’s selection menu waiting for any entity. On headless server this behavior is not acceptable.

Updating grub:

update-grub2 

Updating initramfs:

update-initramfs -u

Create an user for yourself:

adduser <NAME>
adduser <NAME> sudo

Exit the chroot, exit rescue system, disable the netboot option, reboot, and wait..


4.First boot

If you are lucky, a few minutes later you should be able to log in as root to busybox and enter LUKS passphrase using this command:

/sbin/cryptsetup luksOpen /dev/sda5 sda5_crypt ; kill $(pidof cryptsetup) ; kill $(pidof plymouth) 

5.Last preparations

5.1.Unlock script

Adding script to initramfs for easy unlocking
/usr/share/initramfs-tools/hooks/unlock:

#!/bin/sh -e

PREREQS=""

prereqs() { echo "$PREREQS"; }

case "$1" in
prereqs)
prereqs
exit 0
;;
esac

## hook-functions provides copy_exec()
. /usr/share/initramfs-tools/hook-functions

echo "/sbin/cryptsetup luksOpen /dev/sda5 sda5_crypt">$DESTDIR/bin/unlock
echo 'kill $(pidof cryptsetup)'>>$DESTDIR/bin/unlock
echo 'kill $(pidof plymouth)'>>$DESTDIR/bin/unlock

chmod a+x $DESTDIR/bin/unlock

chmod +x /usr/share/initramfs-tools/hooks/unlock

Updating initramfs again

update-initramfs -u

5.2.Locale, Hostname

/etc/default/locale:

LANG="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

Hostname

echo "<HOSTNAME>" > /etc/hostname

5.3.SSH server

Change your SSH Port (Highly Recommended). And you can change Dropbear SSH port using this file — /usr/share/initramfs-tools/scripts/init-premount/dropbear

5.4.Firewall

I always wanted to try ufw:

sudo ufw allow <SSH_PORT>
sudo ufw enable

Turning off ipv6 if you don’t use it

echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf

5.5.VPN server

Openswan & XL2tpd

Installation

sudo aptitude install openswan xl2tpd ppp

/etc/ipsec.secrets:

<SERVER_IP> %any: PSK "<SHARED_SECRET>"

/etc/ipsec.conf :

version 2.0

config setup
nat_traversal=yes
oe=off
protostack=netkey

conn L2TP-PSK
authby=secret
pfs=no
rekey=no
type=tunnel
esp=aes128-sha1
ike=aes128-sha-modp1024
ikelifetime=8h
keylife=1h
left=<WAN_IP>
leftnexthop=%defaultroute
leftprotoport=17/1701
right=%any
rightprotoport=17/%any
rightsubnetwithin=0.0.0.0/0
auto=add
dpddelay=30
dpdtimeout=120
dpdaction=clear

/etc/xl2tpd/xl2tpd.conf :

 [global]
ipsec saref = yes
[lns default]
ip range = 192.168.0.2-192.168.0.100
local ip = 192.168.0.1
refuse chap = yes
refuse pap = yes
require authentication = yes
ppp debug = no
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes

/etc/ppp/options.xl2tpd:

refuse-mschap-v2
refuse-mschap
asyncmap 0
auth
crtscts
idle 1800
mtu 1200
mru 1200
lock
hide-password
local
debug
name l2tpd
proxyarp
lcp-echo-interval 30
lcp-echo-failure 4

/etc/ppp/chap-secrets:

# client server secret IP
<LOGIN> l2tpd <PASSWORD> *

/etc/sysctl.conf:

net.ipv4.ip_forward = 1
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0

5.5.1.Adding ufw rules

sudo ufw allow proto udp from any to any port 500
sudo ufw allow proto udp from any to any port 4500
sudo ufw allow from 192.168.0.0/24

Change in /etc/default/ufw:

DEFAULT_FORWARD_POLICY="ACCEPT" 

Add to the filter table in /etc/ufw/before.rules:

# Allow L2TP over IPSEC
-A ufw-before-input -m policy —dir in —pol ipsec -p udp —dport 1701 -j ACCEPT

And add the following to the top of the /etc/ufw/before.rules:

# nat Table rules
*nat
:POSTROUTING ACCEPT [0:0]
# Forward traffic from eth1 through eth0.
-A POSTROUTING -s 192.168.0.0/24 -o <OUT_INTERFACE>-j MASQUERADE
# don’t delete the ‘COMMIT’ line or these nat table rules won’t be processed
COMMIT

5.6.File sync services

5.6.1.BTsync

Btsync — it’s free proprietary software. So if you too paranoid you shouldn’t use it.

sudo add-apt-repository ppa:tuxpoldo/btsync
sudo aptitude update
sudo aptitude install btsync
sudo ufw allow <port>

5.6.2.Dropbox

Paranoia? Don’t use dropbox.

How to install daemon

http://www.dropboxwiki.com/tips-and-tricks/install-dropbox-as-a-service-onto-ubuntudebian-servers

5.7.Backup software

Installing

sudo aptitude install duplicity

Root

su

Gen GPG

gpg —gen-key

Testing

mkdir /storage/duplicity-backups/
mkdir /tmp/test/
touch /tmp/test/file{1..20}
PASSPHRASE="<YOUR_PASSPHRASE>" duplicity —encrypt-key <PUB_ID> —sign-key <PUB_ID> /tmp/test file:///storage/duplicity-backups/
echo PASSPHRASE=”<YOUR_PASSPHRASE>” > ~/.passphrase
chmod 700 ~/.passphrase

/etc/cron.daily/duplicity:

#!/bin/sh

test -x /usr/bin/duplicity || exit 0
. /root/.passphrase

export PASSPHRASE
/usr/bin/duplicity —full-if-older-than 1W —volsize 4096 —encrypt-key <PUB_ID> —sign-key <PUB_ID> /storage/important file:///storage/duplicity-backups/
/usr/bin/duplicity remove-older-than 3M —force file:///storage/duplicity-backups/
unset PASSPHRASE

Make it executable

chmod 755 /etc/cron.daily/duplicity
cd / && run-parts -v —report /etc/cron.daily
exit
sudo rm /root/.bash_history 

5.8.Hypervisor

Why not?

KVM

sudo aptitude install cpu-checker kvm libvirt-bin virtinst bridge-utils virt-manager

Check

kvm-ok

Create bridge /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto br0
iface br0 inet static
address <IP>
netmask <MASK>
gateway <GATEWAY>
dns-nameservers 8.8.8.8
bridge_ports eth0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off

Download image for new vm:

wget http://releases.ubuntu.com/14.04/ubuntu-14.04-server-amd64.iso

Create vm:

sudo virt-install —connect qemu:///system -n ubuntu-1 -r 1024 —vcpus=1 —disk path=/storage/kvm/ubuntu-1.img,size=12 -c /storage/kvm/ubuntu-14.04-server-amd64.iso —vnc —noautoconsole —os-type linux —accelerate —network=bridge:virbr0 —hvm

5.9.Monitoring

The last part and probably the most important part.

Nagios

sudo aptitude install nagios-nrpe-plugin nagios-nrpe-server nagios3 nagios-plugins-extra nagios-plugins-contrib 

You can change email address in /etc/nagios3/conf.d/contacts_nagios2.cfg

I prefer to make most of the check over NRPE, so /etc/nagios/nrpe.cfg:

log_facility=daemon
pid_file=/var/run/nagios/nrpe.pid
server_port=5666
server_address=127.0.0.1
nrpe_user=nagios
nrpe_group=nagios
allowed_hosts=127.0.0.1
dont_blame_nrpe=0
allow_bash_command_substitution=0
debug=0
command_timeout=60
connection_timeout=300
command[check_apt]=/usr/lib/nagios/plugins/check_apt
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_procs_disksleep]=/usr/lib/nagios/plugins/check_procs -w 7 -c 20 -s D
command[check_procs_total]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200
command[check_procs_zombie]=/usr/lib/nagios/plugins/check_procs -w 2 -c 10 -s Z
command[check_memory]=/usr/lib/nagios/plugins/check_memory -w 30% -c 10%
command[check_sda_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
command[check_sda_smart]=sudo /usr/lib/nagios/plugins/check_ide_smart -n -d /dev/sda
command[check_sda_storage]=/usr/lib/nagios/plugins/check_disk -w 15% -c 10% -p /storage
command[check_sda_swap]=/usr/lib/nagios/plugins/check_swap -w 20% -c 10%
command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
include=/etc/nagios/nrpe_local.cfg
include_dir=/etc/nagios/nrpe.d/

Now add one to /etc/nagios3/conf.d/services_nagios2.cfg

define service {
hostgroup_name *
service_description HW-sda_smart
check_command check_nrpe_1arg!check_sda_smart
use generic-service
notification_interval 0 ; set > 0 if you want to be renotified
}
define service {
hostgroup_name *
service_description OS-sda_storage
check_command check_nrpe_1arg!check_sda_storage
use generic-service
notification_interval 0 ; set > 0 if you want to be renotified
}

/etc/sudoers:

# Nagios
nagios ALL=(ALL) NOPASSWD:/usr/lib/nagios/plugins/check_ide_smart *

Restart services

sudo service nagios3 restart ; sudo service nagios-nrpe-server restart

P.S. This is probably my first post written in english, which is longer than 140 characters

Show your support

Clapping shows how much you appreciated Pavel’s story.