Scaling SDLC Applications

Shawn Stafford
Jan 9 · 5 min read

If you work in a software development organization, you probably deal with various software development applications. These include core applications like source control, bug tracking, and continuous integration. In a small organization (less than 500 engineers), scalability is not typically a concern. Most applications in this area perform quite well when the number of users or the amount of concurrent activity remains small. This article is for the other end of the organizational spectrum, the ones who experience the challenges of using SDLC applications which were not designed to meet the needs of a large organization.

The term “commodity hardware” is nearly synonymous with horizontal scalability. When the hardware is inexpensive and seldom fault tolerant, you need to design your software to provide fault tolerance. It needs to be capable of distributing the workload across multiple systems and expanding to include more servers as usage demands. The idea of “commodity software” is similar: if the application is not designed to scale horizontally with increased load, you must distribute the projects and users across multiple instances.

Organizations commonly make the mistake of trying to deploy single monolithic application instances and scale them vertically to support the entire company. Unfortunately most applications are not designed for large scale deployments, so performance begins to degrade as the amount of concurrent activity increases. This is usually a result of application design limitations, and no amount of optimization or tuning is going to change that.

Of course these problems rarely present themselves during initial deployment because the application data is relatively small and the activity on the system is low. If performance degradation is gradual, it may be possible to project the resource usage forward to anticipate future requirements, but sometimes the degradation is severe or exponential. The best way to mitigate the risk of performance problems in the future is to decompose the application into smaller instances. But how do you know when an application needs to be decomposed for horizontal scaling? The answer is easy: always. Assume that an application can’t scale and you won’t be disappointed.

Here are some common factors that lead to scalability problems:

  • Storing application data on the filesystem
  • Loading large amounts of data in memory to render in the user interface
  • Executing poorly tuned, highly relational database queries

In order to decompose the application, you must determine if it is possible to decompose its usage. This requires an understanding of how users, projects, and teams are structured within your organization. The diagram below shows the comparison between team-based organization in a single application instance compared to an alternative deployment with breaks the application into three instances.

There are several things to consider when breaking up an application:

  • Logical grouping of projects and teams
  • Reporting and communication across instances
  • Licensing and operational costs

Projects and Teams

Project and Team Distribution

In order to ensure consistency across instances, you may need to develop internal tools which can be used to deploy new application instances and create new teams, projects, and users within each instance. By using abstractions to model the administrative activities across application instances, you can establish an infrastructure which can manage an arbitrary number of application instances or application types.

Examples of this internal tooling include:

  • Containerization (Kubernetes)
  • Configuration Management (Ansible, Puppet, Chef)
  • Administrative tools for invoking application REST APIs

With these tools in place, it should be possible to deploy a new application instance in minutes. Once the instance is running, the internally developed administration tools can be used to configure the application. As usage increases, it may be necessary to create a self-service user interface to allow users to create new teams and projects on their own.

Cross-application Reporting and Communication

Licensing and Operational Cost

Although open source software may not have a licensing cost associated with it, there could be a higher operational cost due to a lack of paid support or quality documentation. Depending on the software project, your operational staff may need to invest more time becoming experts in the software or investigating problems. The same is also true for custom tools and software developed internally.

In addition to the application licensing, there may be licensing costs associated with the operating systems or hardware as well. Management and administration of the hosting environment caries a cost as well, so ensure that you choose an environment that you have the knowledge and ability to efficiently manage at scale.

Summary

Shawn Stafford

Written by

Release Engineer with an interest in pipeline traceability and observability.