Python 2 Remediation and Problems at Sea

After Python 2.7 Sunsets, Will Your App Be Cast Adrift?

I Used to Live on a Sailboat. ⛵️

Sounds Romantic, Right? 💌

Sunsets. 🌄

Tropical Islands.🏝

Life on a boat also includes dealing with a litany of things which are actively wearing out. The ocean is beautiful. The gentle rocking at anchor is pleasant. But it’s also cyclic loading on components that will — eventually — fatigue. Some good features of life at sea also have monsters lurking just below the surface. Going to sea means you enter the food chain, and you’re not the apex predator. The worst of the sea monsters are the barnacles and algae attacking your infrastructure — your boat — from the moment it enters the water.

What does this have to do with Python?

Python 2 is a lot like the ratty old charging system for my boat’s batteries. It worked. Mostly. The shore-power charger hummed, loudly, and got really hot. It used a big diode bank to avoid shorting out the alternator, a technique which leads to inefficiencies. It all had to go because it was very slow to charge the boat, and built on ancient technology.

For many users, Python 2 remediation is well under way. I want to address some of the edge cases. Specifically, I want to lift up the problem of “orphaned” Python software as a potential problem when Python 2 is no longer being actively supported.

On a sailboat, the diesel engine is called an auxiliary propulsion system. If you only use it briefly to get in and out of the slip, problems can be ignored. On a well-made boat most things have backups and if one system fails the other still works. When we lived in the Bahamas, for example, an oyster had grown so large the forward head sink didn’t work. The problem is minor, but meant using the galley sink for hand-washing.

Python 2, however, is a little more nuanced than a battery charger filled with heavy transformers and other 1980’s era technology. Currently, the problems with Python 2 are all potential problems in some possible future. When a technology stack doesn’t have active management, then no one is looking out for those potential problems. The mentality of “it isn’t broken, so let’s not try to fix it” mindset can take over, and create longer-term problems.

Fixing a boat’s battery charger was a series of problem-solving exercises. I mean simple problem-solving. Asking questions like “What’s that god-awful hum?” and “Why didn’t the batteries charge while we were motoring?” forced us to replace the charger, the alternator, the diodes, the wires, the safety switches, and then upgrade by adding solar panels.

Python 2, however, isn’t — currently — as broken as my vintage battery charger was. There’s no simple test that will reveal a glaring problem with Python 2 that makes it unusable. It’s slow and quirky. But come January 2020, the state of Python 2 could rapidly escalate from worrying to a potential security nightmare, also.

Sometimes I think of Python 2 the way I think of the fuel line for the diesel engine. It was a short bronze tube from the fuel pump to the secondary filters. It worked safely, and reliably for decades. Then one night, my crew woke me up to announce they smelled burning diesel. The change from safe and reliable to hazardous disaster was abrupt and frightening.

Python 2 has a broad spectrum of applications:

• It’s part of Mac OS X and some Linux distributions.
• You might have a handy script someone gave you to help automate some boring stuff.
• You might be using a small departmental application in an enterprise context. Maybe it’s running on an on-premises server, and has been working fine for years. Maybe the author left the company years ago, and you know someone who can restart the app when there’s a problem.
• You might have an enterprise application deployed in the cloud with a large support organization.
• You might be part of a business which depends on services written in Python 2.

These are only a few suggestions for places you might find Python in use. Python is very widely used, and there are tons of nuanced distinctions among the variety of places it appears.

Obstacles Navigating from Python 2 to Python 3

For some Python 2 use cases, it is clear where it is being used and by whom. Larger enterprises — particularly tech-focused companies that build their own software — will have application support people who can report on Python 2 use and be tasked with making remediation plans.

As of this writing, there are only about 7 weeks left in the life of Python 2. Depending on when you read it, there may be even less time left. The Python 2 Countdown Clock (https://pythonclock.org/) helps to remind everyone of the looming deadline.

Using this clock in a big organization can feel like sailboat racing: there’s a defined course to follow, and you need to get your boat around all the marks as quickly and efficiently as possible. Managing the switch to Python 2 is like being the skipper of a racing boat watching the crew prepare the sails for a maneuver; when everything’s ready, she calls “Helm’s a-lee,” and the boat turns smoothly onto the new course. The whole operation is a glorious coordination between people, technology, wind, and water.

This works because the Python code is actively being managed as a strategic asset.

But…

What about small-boat sailors? What about a group that’s inherited an application that isn’t obviously using Python 2? What about an application that’s not obviously using Python at all? After all, web services should conceal all of their implementation details and a desktop application program can contain a “frozen” Python 2 image.

Also…

What about large, large tech companies with tons of code all over the place? What about fleets of Git repositories packed with code that may — or may not — rely on Python 2?

The problems are similar: they both involve code we might call “orphaned.” The developers aren’t taking an active role in the life of the code. The application is in the hands of end-users, or operations folks who are tasked with keeping it running, not fixing it. This parallels boat owners who aren’t sure if their stainless steel rigging components are grade 316 stainless or the less reliable 304 grade. When the builder is out of the picture, how can the owner be *sure* of what’s installed? In some cases, they can’t be sure, and parts need to be replaced.

I’d like to call this Rule 27 code. This follows the international collision regulations (COLREGS) rule for “Vessels Not Under Command or Restricted in Their Ability to Maneuver.” There’s no skipper in charge of the technology stack. No one is actively steering the application toward the safe waters of Python 3. This leads to a badly mixed metaphor, though. I started out likening Python to sails and engines, now I’m likening it to safe navigation. My editor won’t approve, so I’m going to stick with “orphaned.”

Our application is built from a stack of parts, some of which aren’t being actively supported by knowledgeable engineers. What do we do to avoid problems when we didn’t build the boat? How can we be sure we’ll remain safe and ship-shape? I’ve got some suggestions for folks who aren’t a software (or naval) architect.

The App Inventory

I’ve sailed with people who win race after race on their own tiny boat but struggle to tie some of the heavy-weight knots we use on a bigger live-aboard boat. They understood sailing in a deep and visceral way, but didn’t know much about handling a 21 ton vessel with five sails. In a similar pattern, we might be users of software we didn’t build, but need to understand and use.

Creating an application inventory is a potentially confusing step. In some cases, a person knows all the apps they’re using — spreadsheet, web service, desktop application — the various tools are distinct, and they’re comfortable enumerating a list. Other folks, however, don’t have their heads buried in the technology, and a click here or a click there is all the same to them. These folks will often have to look for help in completing a useful inventory.

Now, I’m going to make a sweeping stereotypical distinction between folks with some knowledge of their tech stack and folks who aren’t focused on the computer-oriented details.

If you’re not computer-obsessed, the two steps are:

• “Refresh all your desktop apps,” and
• “Get technology details from your web vendors, including their Python remediation plan.”

When you’re not sure where a desktop app came from, or — perhaps — there’s a complicated backstory about the desktop app you’re using, this should raise hackles of suspicion. There are storm clouds on the horizon. You’ll need a good explanation from people you can trust.

There’s not much time to get answers and formulate concrete plans to upgrade software. It’s best to take steps to prevent problems. Moving quickly is essential.

The two steps listed above might be too superficial for someone more deeply involved in their technology stack.

Finding Python 2

Finding Python 2 code in an orphaned library, application, or web server can be difficult. The code is not under active maintenance. There’s no diesel engine mechanic we can rely on to change the oil, charge the batteries, and replace fuel filters. Often this is because we’re the diesel engine mechanic tasked with keeping some web server running in spite of Python 2.

There are some — minor — syntax differences between Python 2 and Python 3. The one that gets the most press is the print statement. However. An app can use from __future__ import print_function and replace Python 2 print statements with Python 3 print() functions.

There are a few other syntax distinctions, but many applications might not have any of these problems. For example, the statement except Exception, name: syntax is unique to Python 2, but may never appear. A number of functions in the standard library are changed, but an application may not use any of the unique-to-Python-2 functions.

The Python 2 and Python 3 similarities mean there’s no obvious, simplistic “run this and figure out what Python the code requires” kind of tool. There are some indications, but the only real test for Python 2 is — well — doing the hard work of running the tests with Python 3 and seeing if it breaks. Remember this rule: It’s possible to write code that works in both Python 2 and Python 3.

The general approach is this:

1. Figure out how to run the application.

Some apps include a README or other documentation telling you to run a python the_app.py command. In other cases, there may be a shell script that runs the app. Apps deployed in the cloud may have scripts to build the cloud server and start the application.

AWS Lambdas and some double-clickable desktop apps can have Python bundled in. The way the app runs isn’t as important as the Python image it’s built with.

In some cases, double-clicking an icon may rely on some OS-level bindings between extensions (like .py) and applications (like C:\python\python.exe.) In other cases, there can be a “shebang line” in the file. It might say #!/bin/env python2.7.

2. Figure out which Python it uses.

Some digging may be required to find the python command that starts the app. There may be layers of shell scripts. The context in which the command gets run may be difficult to discern. It may be necessary to include echo $PATH and which python commands in a shell script to figure out what — exactly — is running the Python application.

When an app is bundled with a copy of Python, the script to build the app will be a more important resource than any shell command.

3. Figure out how to test.

This can be difficult. Some applications have no tests of any kind. Other applications will have a tests directory, and will use the tox (or nox) tool to execute tests in a variety of environments. This may devolve into a more serious engineering effort to figure out how to mock a database in order to test a complex, orphaned application.

These three steps feel a lot like yanking a line on a sailboat to see what it’s connected to. In a well-managed racing boat, the lines are often color-coded. In a boat where the racing budget isn’t quite so generous, the lines may be all the same color, making it difficult to figure out what the line does.

Remediation

Once we’re sure of the Python version being used, we’re ready for the remediation steps:

1. If necessary, write tests.

This may be as small as a single, tiny test to import the module for an application and run one of the functions in the module. Or create an instance of one of the classes defined in the module. A test to force the module to be imported is a good place to start.

2. Test with Python 3.

Ideally, this means going all the way down the road to using tox (or nox) to run a test suite in Python 3 only. In some cases, using pytest or python3 -m unittest is sufficient to confirm things will work.

3. Fix what’s broken.

The fixes tend to fall into a number of scenarios:

• In some rare cases, a well-written app or library will have __future__ imports and will run in Python 3 with no further changes.
• In some cases, an app will depend on the six package and will run in both Python2 and Python3.
• In the most common cases, the 2to3 tool can be used to create a revised version. This tool is a first-class part of Python 2.7, and covers many, many common cases elegantly.
• In some cases, the 2to3 tool will work, but there are other changes required, and it’s time for a rewrite. This is the opportunity you’ve been waiting for.
• In rare cases, the rewritten application won’t pass a comprehensive suite of unit tests. While frustrating, it is good to find this. Failing to pass unit tests after a conversion often means some nuanced, version-specific code that shouldn’t have been there in the first place. One source for differences can be the change in semantics of the division operator. In Python 2, 355/113 == 3. This is no longer true. In Python 3, 355/113 == approx(3.141593), using the pytest.approx() function.
• In really rare cases, the 2to3 tool can’t deal with the code. If the code is too weirdly obscure for common tools to work with, it should probably be rewritten.

Yes. This is work. The changes between Python 2 and 3 aren’t trivial. After all, this is a profound change to the underlying power-plant of your application. For me, the switch to Python 3 has been like getting the two new headsails made for my boat: we had more power and more control over our sailing. The original sails weren’t exactly falling into shreds, but they never would have lasted through a storm. Part of getting new sails included changes to the rigging, which also made the boat easier to handle when conditions transitioned from “fun” to “sporty” and started to trend toward “scary.”

Much of the work for the big, visible, actively-maintained enterprise-scale applications is either done or well under way. It’s the “orphaned” technology that presents a bunch of smaller hurdles.

Orphaned technology is always frustrating. Apps that require old operating systems, or networks that are still based on out-of-date hardware are the root cause of many potential problems. The old Python 2.7 interpreter could become an unpatched security hole in an otherwise useful application.

The old charging system on my boat? All replaced. Charging from shore power, solar panels, and the new alternator works beautifully. The tattered head sails? Replaced before we had a problem. The stainless-steel chainplates? All replaced with 316-grade stainless steel.

Which brings us to the broken refrigerator: we threw that away. It was 300 pounds of useless with a large number of complex parts for keeping ice cream frozen. The process of upgrading to Python 3 may involve a bit of triage. Some applications may have seemed important in the past, but aren’t really creating significant value. It may be more effective to delete old software, and change manual procedures, do some integration, or make some enhancements as a way to replace it.

And the leaking fuel line? We simply failed to foresee the problem. It had been in place for decades. Since everything else was ship-shape, we managed to get to safety when the line chafed through. Later, from a comfortable anchorage, we had a funny argument with the supplier of our boat’s engine parts — they didn’t think there were any of the 1980’s engines with rigid fuel lines still in service. They were sure someone had already replaced the old tube with the new hose.

An application or web service still running Python 2 is like my old engine with a bronze fuel line, waiting to be chafed through. You can hope there won’t be a problem. And wake up to a scary smell in the middle of the ocean. Or you can replace the Python 2, and sail away happy and successful toward your next tropical island destination.

Related

Bashing the Bash — Replacing Shell Scripts with Python
Spec to Gherkin to Code — A Relay Play based on Swagger and OAS
Heat Death of the Universe and Faster Algorithms Using Python Dict and Set
The Database Design Change Blues — Performing in the Key of Python

DISCLOSURE STATEMENT: © 2019 Capital One. Opinions are those of the individual author. Unless noted otherwise in this post, Capital One is not affiliated with, nor endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are property of their respective owners.