Upskill tutorial for plugin architecture

Day 9: Plugin. Importlib. Protocol.

Hi 👋 I am Hud, a postdoc for engineering data science at the AI Manufacturing Center in Laramie, Wyoming. My funding is running out (AaAaaA !), so while I am actively looking for a new job, instead of doing the 205th coding certificate to prove my worthiness — I thought I’d do design challenges and document how I spend my time upskiling so other engineers can do the same.

Nowadays, certificates are everywhere. Documenting small upskill projects that you can later show off is the best way to get recognition as a professional engineer.

This is day 9 of a 30-day design challenge. Follow along and let me know if you get stuck!

TL;DR tasks

Download the provided code for the challenge.

Task 1: Understand the current codebase

• Review the existing codebase and understand how the payment methods are currently implemented and used in the main.py file.
• Identify the parts of the code that need modification to support the plugin mechanism for adding new payment methods.

Task 2: Design the plugin mechanism

• Plan the design for the plugin mechanism that allows adding new payment methods without modifying the main.py file.
• Determine the structure and format of the plugin scripts.
• Decide on a suitable location (e.g., plugins folder) to store the plugin scripts.

Task 3: Create a new payment method plugin

• Create a new Python script containing the code for the payment method that needs to be added.
• Ensure that the script follows the required structure and format for the plugin.

Task 4: Implement the plugin loading mechanism

• Modify the main.py file to include a plugin loading mechanism.
• Use importlib and os.walk to dynamically load the payment method plugins from the plugins folder.

Task 5: Test the plugin functionality

• Run the main.py file and verify that the newly added payment method plugin is recognized and available to the user.
• Test the functionality of the new payment method and ensure it works as expected.

Task 6: Refactor the code structure

• Consider splitting the code into separate files to improve code organization and maintainability.
• Create a separate file to manage the loading and accessing of plugins.

Task 7: Test and verify code decoupling

• Add additional payment method plugins and verify that they can be used without modifying the main.py file.
• Ensure that the main.py file remains relatively small and only handles the core functionality.

Task 8: Document the changes made

• Document the modifications made to the codebase to support the plugin mechanism for adding payment methods.
• Explain the design approach used and the benefits of decoupling the code from the main.py file.
• Provide instructions on how to add new payment method plugins in the future.

The problem

We’ve seen in the previous challenge the strategy pattern used for various processing payment types. Using this strategy pattern with separate functions for each payment type can become problematic when more payment types are needed. As the number of payment types grows, the number of functions needed to handle each payment type will also grow, leading to a large and unwieldy codebase.

In addition, adding a new payment type would require defining a new function and modifying the code that maps payment types to functions, which can be error-prone and time-consuming.

A plugin architecture can solve this problem by allowing new payment types to be added dynamically at runtime without modifying the existing codebase. In a plugin architecture, each payment type would be implemented as a separate plugin that can be loaded and unloaded at runtime.

This approach would make it easier to add new payment types in the future without modifying existing code. It would also allow for more modular and maintainable code, as the payment processing logic for each payment type would be encapsulated in a separate plugin.

To implement a plugin architecture, you could define a common interface for payment processors and create a plugin manager that loads and manages the plugins. Each plugin would implement the payment processor interface and provide its own implementation of the payment processing logic for its payment type.

This approach would allow for a more flexible and extensible payment processing system that can easily accommodate new payment types as they become available.

The plugin architecture

The solution I’ve come up with is pretty simple to follow. Rather than having things written in some config file, we need a few things:

• plugins folder, containing a file for each payment processing service i.e. apple, credit card and paypal
• plugin manager file

Inside the plugins folder

All processing payment service files have the same structure. For example, the code below defines a plugin for processing Apple Pay payments. The get_payment_method function returns the name of the payment method handled by this plugin, which is "apple". The process_payment_apple_pay is taken exactly from the main.py code. This function takes a total amount as a Decimal and prompts the user to enter their Apple Pay device ID. It then masks the device ID and prints a message indicating that the payment is being processed.

This plugin can be loaded and used by a payment processing system that supports plugins. When a payment request is received for an Apple Pay payment, the system can use this plugin to handle the payment processing logic for that payment type. We will repeat this for paypal and credit card payment types, and can also extend this to any other payment services as needed without ever touching the main code.

The plugin manager, importlib, Protocol

In the plugin manager we have a Plugin protocol as a class that defines the common interface for payment processing plugins. The get_payment_method and process_payment methods of the Plugin class are defined as static methods using the @staticmethod decorator.

Static methods are methods that belong to the class itself, rather than to instances of the class. This means that they can be called on the class itself, rather than on an instance of the class.

In this case, the get_payment_method and process_payment methods are defined as static methods because they do not depend on any instance-specific state or behavior. They simply define the interface that all payment processing plugins must implement.

The PLUGINS dictionary is also defined as a class-level variable using the dict type. This dictionary stores the available plugins, with the keys being the names of the payment methods and the values being the plugin instances.

By defining the Plugin class and the PLUGINS dictionary as class-level objects, the code avoids the need to create instances of the plugins. Instead, the plugins are defined as module-level objects that can be accessed directly by the payment processing system.

We can then load all modules from a specified folder and add them to a dictionary of available plugins using the function load_plugins_from_folder .

The function uses the os.walk method to recursively traverse the specified folder and find all Python files with a .py extension. For each file, it extracts the module name and path, and uses the spec_from_file_location function from the importlib.util module to create a module specification.

Finally, we add some helper functions to deal with low-level data structures and typing by using type annotations (see day 2 challenge) to specify the types of the function arguments and return values, and by using built-in data structures such as dictionaries and lists to store and manipulate data. This will help us handle the payments in the main code.

Add more payment services

Try it for yourself! Create a separate payment service file e.g. Google Pay and run the main code. You should see something like this:

# before
What payment method would you like to use? (cc, apple, paypal)apple
Please enter your Apple Pay device ID: 123
Processing Apple Pay payment of $70.65 with device ID 123...

# after
What payment method would you like to use? (cc, google, apple, paypal)google
Please enter your Google Pay device ID: 123
Processing Google Pay payment of $70.65 with device ID 123...

Conclusion

Congratulations! You finished Day 9 from the 30-day design challenge.

If you have reached this far, you know how to:

• Extract strategy pattern functions to plugins
• Use importlib and protocol abstraction to import plugins
• Add payment services via plugins

Check out the day 10 challenge!

Also, you can access the full 30-day GitHub repository here.

💡 My goal here is to help engineering data scientists upskill in design. I’d like to hear from you! Was this helpful? Anything I can improve? Connect with me on LinkedIn | Medium