Enhancing Support for Multiple Platforms: A Comprehensive Analysis
by Ryan Mack, Senior Software Engineer at Cisco ThousandEyes
At ThousandEyes, we integrate most of our software offerings into our cloud-based Software as a Service (SaaS) solutions. However, some of our software applications are deployed directly onto client systems. A prime example of this is our Enterprise Agent.
As a Senior Software Engineer on the Enterprise Agents team, I am tasked with the ongoing responsibility of updating and optimizing our software’s compatibility with the most recent Operating Systems (OSs) and platforms. This article provides insight into our processes and methodologies for ensuring quality across a variety of deployment types.
The Evolution of Deployments
The Enterprise Agent represents one of the most fundamental software components within the ThousandEyes portfolio. Over time, we have continually modified and updated it to support the OSs and platforms that our clients need. Primarily, the software conducts network probing and transmits data to the backend for processing and visualization. Given our software’s objective of providing a comprehensive network overview, we understand that clients may need to deploy the software in various ways to capture diverse vantage points within their network. Thus, we have endeavored to simplify deployment by incorporating support for installation on Cisco switches, Meraki devices, virtual machines, docker containers, and beyond.
For the purposes of this communication, I will categorize our types of support into two segments: OS support, which includes support for building and running on a specific OS (such as Ubuntu, RHEL, etc.), and platform support, which pertains to support for building and running on a distinct platform (like Docker, a Cisco switch, etc.).
OS Support
The Rationale for Incorporating New OS Support
When discussing OS support for the Enterprise Agent, you may be acquainted with the OSs that we enable native Linux installs on. Historically, we have provided support for both Ubuntu and RHEL (Red Hat Enterprise Linux), as well as free RHEL alternatives like CentOS. We also accommodate Alpine Linux for specific platforms such as Meraki, which clients may not be aware of since they do not need to select their OS in that situation explicitly.
The main rationale we consider is threefold when adding support for a new OS. They are:
- To support a new version of a currently supported Linux “flavor”
- To accommodate new customer requirements
- To support a new platform
Let’s explore these rationales in depth.
Firstly, it is necessary to add support for an OS to keep pace with the latest versions of an existing “flavor.” An example of this is when we included support for Ubuntu 20.04 (focal). We aimed to ensure that clients using Ubuntu 18.04 (bionic) could continue using ThousandEyes within the Ubuntu framework. Therefore, we incorporated support for the most recent version at the time to maintain optimal support for these clients.
We add this type of support frequently whenever a new version is released for a distribution we support. It is best to support an OS as soon after its release as possible, so it is essential to keep track of release announcements for every OS and plan accordingly. As an OS family releases new versions, the latest security updates may only be available in the most recent versions. Developers such as Canonical (who develop Ubuntu) will often support security fixes in older versions for a while, but all versions will eventually have an “end of life” date. Customers know this and often want to move to newer versions sooner rather than later to avoid hitting that end-of-life date.
Customers do not always have complete control over the OS on their systems for regulatory or corporate reasons. We aim to reduce these barriers by supporting our agents on a wide range of OSs.
A case in point is our RHEL support. It may seem simpler to support only one Linux distribution for native Linux installs. For instance, we could eliminate RHEL altogether and exclusively support Ubuntu. However, this is not feasible as many of our clients are mandated to use RHEL due to their organizational regulations. RHEL is an enterprise-grade Linux distribution that includes advanced security features (like SELinux and FIPS support) required by many regulated industries (such as banks) to pass compliance checks during audits. By supporting RHEL, we enable these clients to use ThousandEyes in compliance with their regulatory requirements.
Every OS that we support will be covering some use-case deemed necessary by customers, therefore making it a requirement for us to support.
Lastly, we may add support for an OS to support a specific platform. For example, we needed to incorporate support for the ARM CPU architecture to support Meraki. We resolved this by introducing support for Alpine Linux, which offers both ARM and x86 versions within a lightweight OS. Although we don’t anticipate clients installing the Alpine Linux version of the agent on native Linux deployments, supporting this OS allows us to accommodate better Meraki and other platforms that utilize the ARM architecture.
Considerations for a New OS
Having discussed the reasons for adding support for a particular OS, let’s delve into the considerations for implementing support. Our Enterprise Agent is a C++ application, which means our agent code must be separately compiled on each supported OS to create a binary compatible with that particular OS/CPU setup. At ThousandEyes, we utilize CI/CD pipelines to build and publish our code, so the first task when adding a new OS is to define a CI builder in Jenkins for our build jobs. When adding a new builder, our main considerations are package version differences and packaging standards for the specific OS.
In terms of system package version changes, it’s crucial to compare the versions that come standard on the new OS with those on OSs we already support. Having a C++ application, a major factor is the GCC compiler we use to build our code, which we often update between major OS releases. Whenever we upgrade to a newer version of GCC, the rules enforced during compilation time may change. We must research the latest GCC versions to find what issues we may encounter.
If they anticipate major changes, the GCC developers release documentation for common issues developers may experience when porting code to the latest version. Here is an example of one such document for developers migrating to GCC 11:
https://gcc.gnu.org/gcc-11/porting_to.html
GCC is just one instance of a package that we depend on, but we follow a similar process for all the packages we utilize to build and run our application.
Aside from managing library updates, we also need to consider how to package the software on each OS. We distribute all our Enterprise Agent software as a Linux package using standard Linux package managers such as APT, YUM, and APK. When we added support for Alpine Linux, it was our first distribution to utilize the APK package manager. We had to familiarize ourselves with the apk package manager and configure our code into an APK package to align with our existing YUM and APT packages. It also required setting up an APK package repo which would be publicly available for installing these new packages.
Platform Support
The Rationale for Incorporating New Platform Support
Platforms represent another primary category of support for the Enterprise Agent. While platforms are not independent from OSs, they complement each other. For instance, our Docker container runs on Ubuntu, and our Meraki container runs on Alpine Linux. We have to decide which OS to use for each platform. First, though, I want to acknowledge how we decide to support a new platform.
The ThousandEyes Enterprise Agent aims to provide clients with a way to monitor network conditions from various vantage points. Therefore, we aspire to support a range of deployment methods so clients can view their network in the ways they desire.
A prime example of this is our support for Cisco Cat9k switches. By introducing a ThousandEyes Enterprise Agent container that runs on a Cisco Cat9k switch, we empower clients to test their entire network from the viewpoint of their switch. If their network relies on Cat9k devices, it can be more realistic to see the network from this perspective instead of requiring them to deploy a separate server or Docker container, which inherently would have a different view than running on the switch directly.
Considerations for a New Platform
When adding support for a brand-new platform, we consider several factors. The most important one is the OS required by the platform. If we can use an OS we already support, we won’t need to consider all the previously mentioned points for new OS support.
We generally determine OS compatibility with a platform by analyzing its capabilities, such as its CPU architecture and hardware specs, to determine what would make the most sense. We also compare the platform with other existing platforms we support and try to keep OS support consistent between similar platforms. For instance, we use the same OS (Ubuntu) on our Raspberry Pi image and our ThousandEyes Virtual Appliance (TEVA).
Apart from the OS, we must determine how we will run our software on the platform. For Cisco routers and switches, this often means creating a Cisco Application Hosting container for the new device type. As we support many Cisco devices using Cisco Application Hosting, we can add a new platform without needing to do much additional engineering work. Our main concern is to validate our existing container against the new platform to ensure it runs as well as it does on other platforms. We also need to assess the new platform’s resources to account for its capabilities (i.e., if it can run extra software like our browser test extension, BrowserBot).
Different platforms can present unique challenges. A case in point is our support for Meraki MX. Though still Cisco routers, these devices are quite different from other supported devices like the Nexus or Catalyst series. Software running on the MX operates in a read-only environment and does not use the same Cisco Application Hosting platform as the other Cisco devices we support. For this reason, we had to create a separate container type for our Meraki deployment.
That was just one example of the challenges we face. Still, our team of engineers will need to evaluate every new platform to understand what is similar and what is different compared to the platforms we already support.
Ensuring Quality
Establishing functionality is only part of the challenge when adding support for new platforms and OSs. Once we confirm that the code compiles and runs, we shift our focus to validating the quality of our product on the new deployment.
Quality Considerations
At ThousandEyes, we set the bar high for quality assurance in all our software. The initial level of this rigorous process involves unit and integration tests. We design these tests to verify the functionality of our features, and we automatically execute on them with every code modification. The scope of these tests spans across all operating systems for which we develop.
We prioritize identifying bugs during this unit and integration testing phase as it allows for quicker and more efficient rectifications. Despite this, it doesn’t diminish the importance of our comprehensive system validation. This manual process is essential for confirming that our software meets our stringent quality standards. It provides a critical final check to ensure that the user experience seamlessly aligns with our commitment to excellence.
To ensure quality, we perform a series of manual validations on any new platform or OS. We use our existing OSs and platforms as the benchmark against which we compare the new software. It’s crucial to make fair comparisons. Therefore, if we are releasing support for a new version of Ubuntu, we will benchmark that against a previous version of Ubuntu that we already support. If we release support for a new model of Cisco switch, we compare the performance against a similar switch that we already support. We consistently run the same verification tests on every platform we support to exercise all of the features that our software supports.
Consistency is key, enabling engineers to familiarize themselves with the expected results for each test and identify when something deviates from the norm. We utilize ThousandEyes dashboards and alerts to benchmark our new platforms against known constants and alert us to potential problems, as shown in Figure 1 below.
We aim for our clients to upgrade to the latest platforms and OSs without encountering surprises. Therefore, we meticulously vet each one to detect any significant changes. Sometimes, these changes are inevitable if the platform has major differences. For example, we cannot support our browser test extension, BrowserBot, on the Cisco N9k platform even though we support it on the Cisco Cat9k platform. This boundary is due to the resource limitations on the platform.
When there are differences, we strive to document them and provide detailed documentation to our customers. This step ensures they are informed about the impact of different deployment decisions in their fleet.
Closing Thoughts
In summary, implementing support for the Enterprise Agent on a new platform or OS is an intricate process. However, these steps are not specific to ThousandEyes. Across industries, desktop applications, like our agent, must be continually updated as OSs and devices continue to evolve. The considerations for platform and OS support I have outlined here could prove beneficial to anyone looking to incorporate new support.
On that note, ThousandEyes is hiring! Please see our Careers page for open roles!
