Upgrading from Ubuntu Bionic Beaver (18.04 LTS) to Focal Fossa (20.04 LTS)

Raghuram Bharathan
Nov 2, 2020 · 4 min read

We use an E2E public cloud system to run our website, some of our internal applications as well staging instances of the applications that we develop.

The applications use various technologies — html, php, ruby, python, nodejs, java as well as docker container. We use nginx and run each application on dedicated subdomains. We also use MySQL and postgresql databases for many of the apps.

Our server has been running 18.04 LTS with most of the programming tools/technologies locked to the LTS version. In some cases, we have done manual upgrades.

For python, we use pyenv and pyenv-virtualenv thereby making us independent of OS python version. Likewise rbenv for ruby and nvm for nodejs. However, our php apps have been using the version provided by the OS (7.2.x). Ditto for our databases — mysql and postgresql.

We keep regularly updating the system to the latest version. This is often an issue since the driver for the backup tool provided by E2E is linked to the kernel version, and the driver sometimes lags behind the kernel update.

Ubuntu 20.04 came out in May first week, but the option to upgrade from 18.04 to 20.04, usually happens only after the first point update is done, which was on August 6. But the update prompt didn’t come for a while, courtesy some upgrade-blocking bugs.

Finally, we got the prompt that a new LTS version was available for our system. The questions was should we update or not? One of the determining factors was that most of our new applications would run on the newer OS. So, it was imperative that our environment where it was staged was kept updated. Yes, we had the option to dockerize some of the apps, but we decided that it was best to keep our server updated.

The first barrier to upgrade is any pending upgrades on the existing version. Unless there are no updates left, you cannot upgrade the OS. In our case, we had a couple of applications which refused to upgrade.

$ apt-get update && apt-get upgrade
...
The following packages have been kept back:
...

Searching for an explanation, we found a few ways to overcome it. We chose

apt-get install <list of packages kept back>

In one case, we discovered one of the repo sources (in /etc/apt/sources.list) was incorrectly pointing to Trusty (14.04!) instead of Bionic and correcting it, fixed the issue.

We were now ready to upgrade. Since our system was backed up, we weren’t too worried about anything going wrong. (Just the hazzle of needing to re-install/restore).

However, as we ran sudo do-release-upgrade we came up with the following

This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.

If you continue, an additional ssh daemon will be started at port
'9004'.
Do you want to continue?

We had no other option. So, having noted down the relevant details (in case of failure) and having added the port number where the additional daemon was running to ufw, we continued. One typical failure scenario is power failure that can sever the ssh connection to the server. Luckily for me, this didn’t happen and the upgrade went through successfully.

Now, we needed to check if all our apps were running fine on the updated OS. It appeared that most were not running!

To start with, php was not present! Looks like our upgrade uninstalled the older version, but didn’t install the newer one.

So, we manually installed php, the version being 7.4.x on 20.04. It was not enough just to install php, but also a few extensions required by the various web apps (like php7.4-fpm for instance). We also needed to update nginx configuration file

fastcgi_pass unix:/var/run/php/php-fpm.sock;

(Previously, it was /var/run/php/php7.2-fpm.sock)

We also needed to install php7.4-msyql.

With that, our php apps were now running.

Shouldn’t our python apps be fine, since we weren’t relying on the OS version python? No!

ImportError: libffi.so.6: cannot open shared object file: No such file or directory

The upgrade had messed up the native library required by python. The correct fix for this was to reinstall the python in our pyenv.

pyenv uninstall 3.6.5
pyenv install 3.6.5

We also needed to recreate the virtualenvs for each of our apps.

Since the upgrade had also upgraded mysql server from 5.x to 8.x, the python-mysql package also needed to be upgraded to support the newer python version.

After doing these changes, we got our python apps running again.

Now for the ruby (on rails) apps. Here we only had issue with the mysql package, which was fixed by reinstalling the package.

Our postgresql configuration had gone missing, again, due to the postgresql update from 10.x to 12.x. We had to add relevant entries for our docker container to connect to the postgresql running on our host.

Our applications were now running on Focal Fossa. Our upgrade from Bionic Beaver to Focal Fossa was complete and successful.

Learnings:

  • Backup everything before update
  • Make a list of applications and their dependencies
  • Check one application after another to ensure that they work after the upgrade
  • If you see a problem, chances are someone has seen it before! Stackoverflow is always there to help.

innoventes