Best Practices for Using Open Source Code in Python Applications
Open source software (OSS) is released under a license where the author grants users permission to use, modify, build features of, and distribute the software to anyone and anywhere. The license dictates how the software can be used, modified, and distributed.
There are different licenses under the open source initiative. Some of the popular ones are MIT, BSD 3-Clause, Apache 2.0, GNU General Public License, and GNU Lesser General Public. It is important to note that each license has a degree of freedom on how the software is to be utilized.
There are many OSS projects that have gained great popularity and are widely used:
- TensorFlow: This is an open-source library for machine learning.
- Apache Spark: This is an open-source framework for big data analytics.
- Linux: Linux is an open-source unix-like operating system.
Clearly OSS is great. So, why is it important to follow certain practices while using OSS in building our applications?
- Security: One in sixteen OSS projects contains security vulnerabilities, which puts your whole application at risk.
- Maintenance and Support: Frequent updates to OSS requires constant maintenance and support from developers. Failing to address do so leads to unusual behavior and build failures in the future.
- Flexibility: With so many different software development tools, it is crucial to check OSS compatibility with existing tools and how an open source component affects the current system.
Let’s deep dive into some of the best practices to consider while building Python applications with open source tools.
Policy/Regulation
Before building enterprise applications, it is vital to create a policy on OSS usage for developers and system architects. If developers start using OSS without understanding the licensing agreement, it may lead to organizations losing their intellectual property or even losing monetary value.
There are 1,400+ licenses under the open source initiative, each having different restrictions and obligations. Thus, it is important to have an organization-level policy to which all your developers must adhere.
Keeping Track and Updating
Due to the need for rapid development and innovation, developers are increasingly turning to open source frameworks and libraries to accelerate software development lifecycles (SDLCs). Some of the popular open source projects have tens of thousands of contributors and have frequent updates with new features.
Every developer knows how easy it is to lose track of these updates, especially when there are many open source components
From my personal experience, I know how cumbersome manually updating components can be. When I was working on an OCR (optical character recognition) project, I was experimenting with multiple open source OCR tools. I finally decided to go with Easyocr.
When we finalized the tool for the project, the latest version, which is what we were using, was 1.2.2. By the time we finished building the infrastructure and testing framework around the use case, at least six major updates to the OCR had already been released since v1.2.2. As you can imagine, this led to inconsistent results by the OCR and the inference time and model size had been reduced drastically in these updates.
So the next time, we created a pipeline to check for OCR updates and how they affect the application’s performance.
Thus, we learnt our lesson: Keep Track of open source components in the application.
If you don’t want to make the effort of creating a pipeline, but want a similar tool that can check updates and do more, then there are multiple software available for this.
Let’s check out few tools in the market for the open source management
- WhiteSource Renovate
WhiteSource Renovate is an open-source management tool that provides real-time tracking for OSS updates. It keeps track of the logs and commits of each update, and runs and verifies test suites with the updated OSS.
Renovate provides wide support in various languages including Python, Go, Javascript, Java etc., and has an extensive configurability. It integrates into the existing workflow and schedulers in the system like Docker, Git, GitLab, etc., without any hassle.
2. Fossa
Fossa is also an open-source management tool which ensures the security, compliance and quality of the open source code. It provides real-time tracking of open source dependencies on large scale systems with auditing & removing the potential risk in the system.
With continuous compliance, Fossa increases developer’s agility by automating license scanning and compliance. Like Renovate, it supports different languages and workflow integration tools but at the expense of the customer.
3. Black Duck
Black Duck is a commercial tool that manages the security, quality, and license compliance risks that arise from using open source components and third-party applications.
Along with key functionalities, the tool provides modules for analysis like dependency analysis of open source components, snippet analysis for checking proprietary code and other metrics. Similar to other tools, Black Duck integrates well with existing workflow management tools but at the expense of customers.
Just like you have the MLOps pipeline or DevOps pipeline, it is crucial to have a separate pipeline to track OSS tools in the environment.
Quality
While discussing the benefits of open source tools, you should never get carried away by the ease or familiarity of the tool. You should always look for high-quality standards in your open source components. You should check details like how quickly do bug fixes come out, how many contributors are there, how active is the community and so on. It reflects the relevance of the open source tool in the current market.
As we see in SDLC, when a feature is released, the whole system undergoes testing to check for breakage in the system because of the updates. Similarly, when an open source code is updated with a new feature, we can check how the updates affect the open source project by using tools like Pytest. Naturally, we cannot perform testing on all the features updates. Still, you should track application-specific metrics that are directly affected by open source updates.
What happens when quality takes a backseat? When quality isn’t actively chased, vulnerabilities get introduced. For example, the Heartbleed Bug was a serious vulnerability found in OpenSSL, a cryptographic open-source software library used by an estimated two-thirds of web servers worldwide at the time of the bug’s discovery.
Automated Control Over Open Source
An excellent way to secure the use of OSS is by embedding the open source component as regular software with CI/CD processes. There are tons of potential vulnerabilities in the software when they are developed and released. Integrating OSS into the automated pipeline will result in all the OSS dependencies in a project being analyzed automatically.
During the automatic build process, potential vulnerabilities related to an open source component will be detected. This will help the developer fix the unstable build for deployment. Anyway, the idea is to improve the process of identifying vulnerable components in the system and provide instantaneous feedback to the developer, making the system secure and functional from end to end.
Conclusion
Open source tools are part of every software development system and a good open source tool has a large user community and contributors. Though open source tools save the organization’s time and resources, it should be noted that open source tools differ widely in terms of their security vulnerabilities, licensing, and maintenance. Thus, it becomes important to do a complete evaluation of any open source tool before incorporating it into your project.
