Guide: Installing OpenSolaris on a remote dedicated server

Published in

Moqume Blog

7 min readAug 24, 2009

There was an article at the OVH Wiki that outlined how to install OpenSolaris on a dedicated server with the help of Linux and Xen. This article is of special interest to those who have a dedicated server at a hosting company, but do not have the option to install anything but a Linux distribution.

Unfortunately, that particular article attracted a number of Wiki spammers and it disappeared in a sea of links to unscrupulous websites. I am reviving that article here, for those interested.

Scenario: You have a dedicated server (not a mere Xen guest) with a hard disk and a network connection, but no real CD/DVD drive and no real KVM. You have a Linux distribution and you would like to install OpenSolaris. It is possible by following the steps in this guide.

Part 1. Install Xen/Linux

Create the base installation

Have Linux with Xen support (dom0, a.k.a. host) installed on your server. The detailed steps for doing this depend on your hosting provider; here’s how you do it on OVH:

Open the “Manager” web application, select your desired server, click on “Services”.
Click on “Reinstall / Change OS”.
Pick “Linux”, “Base distributions”, “Debian 5.0 (Lenny)”, “64 bits”.
Create a main ext3 partition of 2–4 GB. 2 GB should be enough; you may make that larger if you want to carry out additional steps/customizations, not just those outlined here.
If you have to, create a swap partition of any size you see fit. This will be deleted at the end of the procedure.

Customize and add Xen

distroconst.tar.gz contains files needed to alter the build process for the modified live OpenSolaris CD, while linuxroot.tar.gz contains support files used in Linux.

If you want, copy your SSH key on the remote server to enable password-less operation:

REMIP=94.11.22.33
# Copy SSH key
scp .ssh/usual.pub root@$REMIP:.ssh/authorized_keys

Then, use SSH to log into your Linux remote server, install Xen and reconfigure as needed:

wget http://cr.opensolaris.org/~e.p/remote-install/distroconst.tar.gz
wget http://cr.opensolaris.org/~e.p/remote-install/linuxroot.tar.gz
tar xvfz linuxroot.tar.gz
apt-get update
mkdir /boot/grub
# We switch to Grub and remove named; we'll need the dns port for dnsmasq.
apt-get install lilo- bind9- grub xen-linux-system-2.6.26-2-xen-amd64 wget libvirt-bin qemu psmisc less dnsmasq shorewall-perl bridge-utils parted ethtool xauth

Wait for the installation to finish, answering any questions in the process, and continue:

# Configure the internal network for virtual machines.
cat >>/etc/network/interfaces <<EOFauto virbr0
iface virbr0 inet static
    address 192.168.122.1
    netmask 255.255.255.0
    network 192.168.122.0
    broadcast 192.168.122.255
    pre-up brctl addbr virbr0
    pre-up ethtool -K virbr0 tx off
    post-down brctl delbr virbr0
EOF# Configure shorewall so that virtual machines can talk to the Internet using NAT.
SHW=/etc/shorewall
echo startup=1 >> /etc/default/shorewall
echo IP_FORWARDING=On >> /etc/shorewall/shorewall.conf
cat > $SHW/zones <<EOF
fw firewall
net ipv4
vir ipv4
EOFcat > $SHW/interfaces <<EOF
net eth0
vir virbr0
EOFcat > $SHW/policy <<EOF
fw all ACCEPT
vir all ACCEPT
all fw ACCEPT
all all REJECT
EOFcat > $SHW/masq <<EOF
eth0 virbr0
EOF# Reconfigure dnsmasq
cat >>/etc/dnsmasq.conf <<EOF
interface=virbr0
dhcp-range=192.168.122.64,192.168.122.127,12h
EOF# Is everything OK? Go ahead, then.
grub-install /dev/sda
reboot

You should be able to ping the server and enter via SSH again. The command “uname -a” must show that you’re running the Xen-ified kernel.

Part 2. Add partitions, install the OpenSolaris domU

SSH -X into your Linux server. Remember to use -X to forward X11 connections; you will need that for the OpenSolaris installer.

If you have created a swap partition at install time, disable it:

swapoff /dev/sda2

Launch your favorite partitioning tool (cfdisk, for example). Delete the swap partition, if any.

Create a small sda2 partition that will contain the modified live image; 800MB is a reasonable size. Its type is irrelevant, and it can remain Linux; only make sure that the type is not Solaris, since there can be only one Solaris partition in each hard disk.

Allocate the remaining free space to sda3, making it a large Solaris partition.
You will end up with something similar to this:

sda1 Boot Primary Linux ext3 [/] 4194.90
sda2 Logical Linux 797.86
sda3 Primary Solaris2 230003.51

Make sure you use partition type BF for Solaris and not the obsolete 82 (would trigger a bug in the installer: http://defect.opensolaris.org/bz/show_bug.cgi?id=4872)

Download and launch the OpenSolaris Guest

Type the following commands:

# Download OpenSolaris
wget -o osol-0906-x86.iso http://dlc.sun.com/osol/opensolaris/2009/06/osol-0906-x86.iso# Reconfigure Xen for libvirt-bin
echo '(xend-unix-server yes)' >> /etc/xen/xend-config.sxp
/etc/init.d/xend restart# Start VMvirsh create linuxroot/cd-osol.xml
virsh console osol

Answer the initial questions (the defaults are fine) and go to the login prompt.
Remember that only the user “jack” can log into the live CD, with password “jack”.

Enable the ssh server

svcadm enable ssh

Wait for the virtual machine to be assigned its own IP address via DHCP by dnsmasq. Find out if that has happened with “/sbin/ifconfig xnf0” inside the VM, or quit the console by pressing Ctrl-] and look at /var/log/syslog in the Linux host machine. When the address has been assigned successfully, you will see a DHCPACK line.

Make sure you’re at the prompt of the Linux machine and issue the following commands, correcting as needed the IP address:

VMIP=192.168.122.84
ssh -X jack@$VMIP pfexec gui-install

Answer the questions and start the installation. When it finishes, do not press “Reboot”. Instead, close the installer and issue the following command at the Linux shell prompt:

ssh jack@$VMIP ‘pfexec sh -c “set -x;
 /usr/sbin/zpool import -f rpool || true;
 /usr/sbin/zfs set mountpoint=/newroot rpool/ROOT/opensolaris;
 /usr/sbin/zfs mount rpool/ROOT/opensolaris;
 (cd /newroot/platform;
 tar cf — `find i86xpv -name unix` `find i86pc -name boot_archive`);
 /usr/sbin/zfs set mountpoint=/ rpool/ROOT/opensolaris;
 /usr/sbin/zpool export rpool”’
 | tar xf -

This will copy the kernel and the boot archive from the newly-installed VM onto your Linux system, so that Xen can boot it.

Finish by issuing:

ssh jack@$VMIP “pfexec /sbin/init 6 &”

(as the VM definition is set up so that, on reboot, Xen does not restart the domain, but rather destroys it). Check that the VM has really turned off: issuing

virsh list

should only display “Dom-0”, and not “osol”.

Part 3. Create the live CD

Note: we suppose that you want to do this on your VM, saving you the hassle of uploading almost a gigabyte. If your Internet connection doesn’t have a reasonably fast upstream link, this is the way to go. Look below for the alternative.

When the VM has turned off, boot it off the rpool on the hard disk:

virsh create linuxroot/hd-osol.xml

When it has booted up and got is IP (see above how to check it), we need to tell OpenSolaris to build a new filesystem, unpack distroconst.tar.gz there, and start building the CD.

# Remove old SSH public key for server
rm -f .ssh/known_hosts
VMUSER=ep
cat distroconst.tar.gz | ssh $VMUSER@$VMIP ‘pfexec sh -c “
 set -x;
 export PKG_CACHEDIR=/var/pkg/download;
 export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/gnu/bin;
 /usr/bin/pkg install SUNWdistro-const;
 /usr/sbin/zfs create rpool/constructor || true;
 /usr/sbin/zfs set mountpoint=/constructor rpool/constructor;
 cd /constructor;
 /usr/gnu/bin/tar xfz — ;
 /usr/bin/distro_const build ./slim_cd_x86.xml”’

What’s in distroconst.tar.gz?

The file contains a modified version of the manifest that builds OpenSolaris’s standard live CD. The modifications are the following:

it runs the add_driver_aliases.sh script after installing the packages
then, it runs the extend-live-fs-root.sh script

add_driver_aliases.sh is where you want to tweak your drivers so that the systems boots OK. In this case it does four things, all meant to improve compatibility with the OVH’s Kimsufi XL hardware platform:

replaces the ‘nge’ driver in 111b with the version in 116, extracted from SXCE — needed to work correctly on the Kimsufi XL’s motherboard;
enables the nge driver for the MCP78S;
enables the ahci driver for the MCP78S;
disable MSI as a workaround for a bug in MCP78S’s AHCI support.

extend-live-fs-root.sh, on the other hand:

puts the listhd utility into the image;
patches the live-fs-root service method so that it looks in HD partitions for live images.

You might want to do different things; for instance, replace “jack”’s password with a custom one, etc.

Copying the CD to its final destination

At the end of the process, you will have an OpenSolaris.iso image in /constructor/media inside the VM. If you don’t want to go through the build process and you already have such a modified image (it is NOT just the stock live CD!), you can download it there.

Then push it onto its partition:

ssh $VMUSER@$VMIP ‘pfexec sh -c “set -x;
/usr/bin/dd if=/constructor/media/OpenSolaris.iso
of=/dev/rdsk/c7t0d0p2 bs=8192k”’

Part 4. Setting up the boot loader

We need to extract the kernel and the microroot from this live CD and copy them onto the ZFS root pool.

ISADIR=/amd64
(cat <<EOF
default 0
timeout 5
title Live image boot
findroot (pool_rpool,2,a)
bootfs rpool/liveboot
kernel$ /platform/i86pc/kernel${ISADIR}/unix -B livemode=text,livessh=enable
module$ /boot${ISADIR}/x86.microroot
EOF
) | ssh $VMUSER@$VMIP ‘ISADIR=amd64/
ISO=/dev/dsk/c7t0d0p2
MOUNTPOINT=/tmp/iso.$$
DISK=/dev/rdsk/c7t0d0s0
pfexec sh -c “set -x;
mkdir -p $MOUNTPOINT;
/sbin/mount -F hsfs -o ro $ISO $MOUNTPOINT;
/usr/sbin/zfs create rpool/liveboot || true;
/usr/sbin/zfs set mountpoint=/liveboot rpool/liveboot;
( cd $MOUNTPOINT;
tar cf — `find . -name unix` `find . -name x86.microroot` )
| ( cd /liveboot; tar xf — );
/bin/sync;
/sbin/umount $MOUNTPOINT;
/bin/rmdir $MOUNTPOINT;
cat > /rpool/boot/grub/menu.lst;
/sbin/installgrub -fm /boot/grub/stage1 /boot/grub/stage2 $DISK “‘

What we’re doing here on the OpenSolaris VM is:

we prepare a mountpoint and mount the ISO image;
we create a new filesystem, called “liveboot”, on the root pool;
we extract the kernels and the microroots onto the new filesystem;
we unmount the ISO;
we overwrite grub’s configuration with a single-option menu that boots this kernel and microroot;
we run installgrub and overwrite the MBR.

Finally, shut down the VM again:

ssh $VMUSER@$VMIP ‘pfexec /sbin/init 6 &’

Part 5. Reboot the Linux/Xen machine and install

Reboot your Linux machine. In a while you should be able to ping the server, and ssh -X into it as “jack”.

From there, go through the usual “pfexec gui-install” routine. Just make sure not to touch the partition where you dumped the live CD (because the installer needs it, and because it might turn out to be useful later if something goes afoul).