Installing packages and modules for Decision Optimization projects in Watson Studio Local

Victor Terpstra
Mar 5, 2019 · 3 min read

Installing a Python package in IBM’s Watson Studio Local (WSL) should be targeted to the project’s packages/python directory by default. This allows the package to be available for any user of the project, in the Decision Optimization (DO) Model Builder and in deployments, without the need for repeated installs.

Introduction

It is now possible to use packages and modules in your DO for WSL optimization model that are not included in the WSL default installation or which you have custom developed. I’ll discuss how to best install and use them.

The just released DO for WSL v1.2.3 (and ICPd v1.2.0) has an important new feature: the Model Builder run-time environment mounts the WSL project directory. This means that the DO Python model can now import custom packages and modules, as long as they are installed in the project directory.

Although not exposed in the WSL user interface, any WSL project contains a folder packages/python. The purpose of this folder is to install custom packages here, instead of doing a regular pip install. In this way, any deployment environment will have immediate access to the same installed packages. This avoids doing potentially complex pip installs in the deployment environment.

But for DO there is a second reason, namely the packages are now directly accessible from the DO run-time environment and from deployed DO models.

Installation steps

  1. Install the Python package using the option:
    --target='../packages/python'
    For instance:
    !pip install mypackage --target='../packages/python'
  2. If you have any custom developed Python modules, add them to the scripts folder.
  3. Add the packages/python and scripts folders to the Python path.
    By default they are not included, so you have to add it explicitly by adding the following to your code:
import sys, os
for folder in ['scripts', 'packages/python']:
path = os.path.join(os.environ['DSX_PROJECT_DIR'], folder)
if path not in sys.path:
sys.path.insert(0, path)

This little code snippet does two things over a more simple sys.path.insert(0,'../packages/python'):

  • It will avoid adding the same path each time this code is run. Which would have made the Python path unnecessarily long and inefficient.
  • It uses the os.environ['DSX_PROJECT_DIR'] instead of the parent of the current directory ../. The ../ works fine in notebooks (where the current directory is the notebooks folder), but doesn't apply in the DO Model Builder run-time environment.

Also note that the fact that we insert the path at position 0 means that these packages could potentially override packages that are installed by default in WSL. This could be important if your code requires a different version of a package.

Un-installing a package

Normally, un-installing a package that has been installed with a regular pip install, can be done with !pip uninstall — yes mypackage. However, that doesn't work if the package is installed in a target directory. You will have to delete the package directories yourself. For instance, you can use WinSCP to access the packages folder and manually delete the mypackage folder.
Alternatively, you can delete them from a Jupyter cell by using bash commands.

To get a listing of all folders in the packages/python directory use:
!ls '../packages/python'
Then delete the folders with:
!rm -r '../packages/python/mypackage'
!rm -r '../packages/python/mypackage-0.0.1.dist-info'

(Note that a pip install typically will have created a second directory with the wheel file that you want to delete.)

Package available to all users

Since the package is now a part of the WSL project, it is included in the project GitHub repository and in the project zip-file. This means that other users, after installing the project from GitHub or zip-file, will be able to use the project immediately and do not need to install the required packages separately.

Comparison with regular pip install

Alternatively, the more ‘traditional’ way to install a package in WSL is:
!pip install mypackage --user

The advantages are:

  • The --user option will make the package available for all your projects in this cluster.
  • The path to the package is included in the default Python path. No need to use the sys.path.insert in your code.

But the disadvantages are:

  • Each user will have to install all required packages of a project separately.
  • This installation is not available in deployed environments.
  • This installation is not available in the DO Model Builder run-time environment.

As a side note, if you are not using the --target option, installing without the --user is not advised. Some packages will not install properly, and the installation gets deleted once the environment is restarted.

Summary

By default, install custom packages in WSL using:
!pip install mypackage --target='../packages/python'
This makes the package available for all WSL users of your project, the DO Model Builder, and deployments.

Victor Terpstra

Written by

Senior Data Scientist — Prescriptive Analytics, IBM Data Science Elite Team. The opinions expressed are my own and don’t necessarily represent those of IBM.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade