Python 2 Remediation and Problems at Sea

After Python 2.7 Sunsets, Will Your App Be Cast Adrift?

Steven F. Lott
Nov 5 · 13 min read
sailboat floating on a lake
sailboat floating on a lake

I Used to Live on a Sailboat. ⛵️

Sounds Romantic, Right? 💌

Sunsets. 🌄

Tropical Islands.🏝

Life on a boat also includes dealing with a litany of things which are actively wearing out. The ocean is beautiful. The gentle rocking at anchor is pleasant. But it’s also cyclic loading on components that will — eventually — fatigue. Some good features of life at sea also have monsters lurking just below the surface. Going to sea means you enter the food chain, and you’re not the apex predator. The worst of the sea monsters are the barnacles and algae attacking your infrastructure — your boat — from the moment it enters the water.

What does this have to do with Python?

Python 2 is a lot like the ratty old charging system for my boat’s batteries. It worked. Mostly. The shore-power charger hummed, loudly, and got really hot. It used a big diode bank to avoid shorting out the alternator, a technique which leads to inefficiencies. It all had to go because it was very slow to charge the boat, and built on ancient technology.

For many users, Python 2 remediation is well under way. I want to address some of the edge cases. Specifically, I want to lift up the problem of “orphaned” Python software as a potential problem when Python 2 is no longer being actively supported.

On a sailboat, the diesel engine is called an auxiliary propulsion system. If you only use it briefly to get in and out of the slip, problems can be ignored. On a well-made boat most things have backups and if one system fails the other still works. When we lived in the Bahamas, for example, an oyster had grown so large the forward head sink didn’t work. The problem is minor, but meant using the galley sink for hand-washing.

Python 2, however, is a little more nuanced than a battery charger filled with heavy transformers and other 1980’s era technology. Currently, the problems with Python 2 are all potential problems in some possible future. When a technology stack doesn’t have active management, then no one is looking out for those potential problems. The mentality of “it isn’t broken, so let’s not try to fix it” mindset can take over, and create longer-term problems.

Fixing a boat’s battery charger was a series of problem-solving exercises. I mean simple problem-solving. Asking questions like “What’s that god-awful hum?” and “Why didn’t the batteries charge while we were motoring?” forced us to replace the charger, the alternator, the diodes, the wires, the safety switches, and then upgrade by adding solar panels.

Python 2, however, isn’t — currently — as broken as my vintage battery charger was. There’s no simple test that will reveal a glaring problem with Python 2 that makes it unusable. It’s slow and quirky. But come January 2020, the state of Python 2 could rapidly escalate from worrying to a potential security nightmare, also.

Sometimes I think of Python 2 the way I think of the fuel line for the diesel engine. It was a short bronze tube from the fuel pump to the secondary filters. It worked safely, and reliably for decades. Then one night, my crew woke me up to announce they smelled burning diesel. The change from safe and reliable to hazardous disaster was abrupt and frightening.

Python 2 has a broad spectrum of applications:

  • You might have a handy script someone gave you to help automate some boring stuff.
  • You might be using a small departmental application in an enterprise context. Maybe it’s running on an on-premises server, and has been working fine for years. Maybe the author left the company years ago, and you know someone who can restart the app when there’s a problem.
  • You might have an enterprise application deployed in the cloud with a large support organization.
  • You might be part of a business which depends on services written in Python 2.

Obstacles Navigating from Python 2 to Python 3

For some Python 2 use cases, it is clear where it is being used and by whom. Larger enterprises — particularly tech-focused companies that build their own software — will have application support people who can report on Python 2 use and be tasked with making remediation plans.

Using this clock in a big organization can feel like sailboat racing: there’s a defined course to follow, and you need to get your boat around all the marks as quickly and efficiently as possible. Managing the switch to Python 2 is like being the skipper of a racing boat watching the crew prepare the sails for a maneuver; when everything’s ready, she calls “Helm’s a-lee,” and the boat turns smoothly onto the new course. The whole operation is a glorious coordination between people, technology, wind, and water.

This works because the Python code is actively being managed as a strategic asset.

I’d like to call this Rule 27 code. This follows the international collision regulations (COLREGS) rule for “Vessels Not Under Command or Restricted in Their Ability to Maneuver.” There’s no skipper in charge of the technology stack. No one is actively steering the application toward the safe waters of Python 3. This leads to a badly mixed metaphor, though. I started out likening Python to sails and engines, now I’m likening it to safe navigation. My editor won’t approve, so I’m going to stick with “orphaned.”

Our application is built from a stack of parts, some of which aren’t being actively supported by knowledgeable engineers. What do we do to avoid problems when we didn’t build the boat? How can we be sure we’ll remain safe and ship-shape? I’ve got some suggestions for folks who aren’t a software (or naval) architect.

The App Inventory

I’ve sailed with people who win race after race on their own tiny boat but struggle to tie some of the heavy-weight knots we use on a bigger live-aboard boat. They understood sailing in a deep and visceral way, but didn’t know much about handling a 21 ton vessel with five sails. In a similar pattern, we might be users of software we didn’t build, but need to understand and use.

  • “Get technology details from your web vendors, including their Python remediation plan.”

Finding Python 2

Finding Python 2 code in an orphaned library, application, or web server can be difficult. The code is not under active maintenance. There’s no diesel engine mechanic we can rely on to change the oil, charge the batteries, and replace fuel filters. Often this is because we’re the diesel engine mechanic tasked with keeping some web server running in spite of Python 2.

1. Figure out how to run the application.

Some apps include a README or other documentation telling you to run a python the_app.py command. In other cases, there may be a shell script that runs the app. Apps deployed in the cloud may have scripts to build the cloud server and start the application.

2. Figure out which Python it uses.

Some digging may be required to find the python command that starts the app. There may be layers of shell scripts. The context in which the command gets run may be difficult to discern. It may be necessary to include echo $PATH and which python commands in a shell script to figure out what — exactly — is running the Python application.

3. Figure out how to test.

This can be difficult. Some applications have no tests of any kind. Other applications will have a tests directory, and will use the tox (or nox) tool to execute tests in a variety of environments. This may devolve into a more serious engineering effort to figure out how to mock a database in order to test a complex, orphaned application.

Remediation

Once we’re sure of the Python version being used, we’re ready for the remediation steps:

1. If necessary, write tests.

This may be as small as a single, tiny test to import the module for an application and run one of the functions in the module. Or create an instance of one of the classes defined in the module. A test to force the module to be imported is a good place to start.

2. Test with Python 3.

Ideally, this means going all the way down the road to using tox (or nox) to run a test suite in Python 3 only. In some cases, using pytest or python3 -m unittest is sufficient to confirm things will work.

3. Fix what’s broken.

The fixes tend to fall into a number of scenarios:

  • In some cases, an app will depend on the six package and will run in both Python2 and Python3.
  • In the most common cases, the 2to3 tool can be used to create a revised version. This tool is a first-class part of Python 2.7, and covers many, many common cases elegantly.
  • In some cases, the 2to3 tool will work, but there are other changes required, and it’s time for a rewrite. This is the opportunity you’ve been waiting for.
  • In rare cases, the rewritten application won’t pass a comprehensive suite of unit tests. While frustrating, it is good to find this. Failing to pass unit tests after a conversion often means some nuanced, version-specific code that shouldn’t have been there in the first place. One source for differences can be the change in semantics of the division operator. In Python 2, 355/113 == 3. This is no longer true. In Python 3, 355/113 == approx(3.141593), using the pytest.approx() function.
  • In really rare cases, the 2to3 tool can’t deal with the code. If the code is too weirdly obscure for common tools to work with, it should probably be rewritten.

Related

Bashing the Bash — Replacing Shell Scripts with Python
Spec to Gherkin to Code — A Relay Play based on Swagger and OAS
Heat Death of the Universe and Faster Algorithms Using Python Dict and Set
The Database Design Change Blues — Performing in the Key of Python


Capital One Tech

The low down on our high tech from the engineering experts at Capital One. Learn about the solutions, ideas and stories driving our tech transformation.

Steven F. Lott

Written by

Programmer. Writer. Whitby 42 Sailor.

Capital One Tech

The low down on our high tech from the engineering experts at Capital One. Learn about the solutions, ideas and stories driving our tech transformation.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade