Open-Source Software Architecture Management - Why you can profit from it
Joint work with Tim Cruse
An outstanding software architecture does not develop by itself, but rather good architecture management is needed. Especially if you are working on large projects, you will need to have set up a good software architecture management process. This includes on the one hand a competent software architect and project manager and on the other hand a well-designed project plan and good structures in the organization itself. In the following, we will try to determine, which skills a perfect software architect needs, what the most important parts of the project planning are, which structures need to be implemented in the organization and how development teams can thrive in their development tasks. Finally, we will look at open-source software projects and try to find out, how software architecture management is ensured.
Key Players in Software Architecture Management
To achieve well-documented software, your company needs a person who is in charge of the whole system. A person, who knows enough about the software and also is a good communicator to the other colleagues. This person is called a software architect. Unlike the name would suggest, his main job is to perform tasks in the background to ensure the good shape of the software architecture. He is not the one who directly creates the software.
Another key player in the software architecture workflow is the project manager. This person is in charge of the new features of the system and prioritizing the tasks. This is why the software architect must have an understanding of the problems of the project manager and find solutions together for concurrent tasks. You can imagine that the project manager is in charge of the external aspects of the project. The software architect is the opposite because his responsibility is the internal side of the software project.
If these domains are not well allocated and there are conflicts of interest between the software architect and the project manager some business goals could be suffering and won’t be realized.
The Perfect Software Architect
In the wide descriptions of the scope of duties in the software architect’s function, many different skills are required… or at least helpful. Because of the many different tasks, it is impossible to say this is the perfect software architect with this and that skills. However, we made a list with the most frequently used skills in the job of the software architect:
🗣 Communicator
Your main work as a software architect will be with other people. Therefore it is important that you can communicate pretty well with them. On the one side, you need to express your ideas clearly for anyone to understand them and their usefulness. On the other side, you need to be able to find compromises with other stakeholders, which can apply to your software architecture. Furthermore, the people you are working with must be willing to listen to you. So it’s an advantage when you have a well entertaining presentation style.
Also, you will work as a kind of translator. You translate the technical stuff to the customer without a technical background (mostly). So don’t speak from that 3-dimensional nested left joined SQL query over 10 different database tables — just say a database query.
🤵🏼 Manager
A big part of the life of a software architect is management. You need to manage the workload which is limited by time. Therefore you need a good understanding of the work itself and how long it will take to perform such tasks for you or your colleagues. So here are your analytical skills on the spot.
Furthermore, you will be confronted with problems which you will need to solve. Also sometimes you need to take risks to get the stakeholders satisfied and make their wishes true. Also, you must have a mindset, that allows you to see and understand the other side of the table and be open for changes.
To get your colleagues on the same track as you, you will need to show your leadership abilities. If the software architecture can’t make the usefulness of the documentation obvious, why should I even try to do this? Yeah you can say just do it or I will tell your supervisor, but this is not good leadership, this is 4th-grade-snitch-behavior.
🤝 Team Player
Together with your leadership skills, you will need to be a team player. You must be able to build and keep the teams around you in a good mood. They must be thrilled to listen to another exciting story of the software architecture world…Okay maybe the last one is a little bit too much, but I think you get the point.
Your technical help is sometimes required to talk on a specific issue and you need to act as a kind of mentor for new changes, which you had enabled.
🔭 Visionary
It is helpful to have your feet on the ground but your mind on the moon. So you imagine what the future brings and which new technologies or developments are significant in the upcoming month and years. So you can elaborate on how this will affect the software architecture process and which switches must be enabled to still assure the workflow.
📚 Knowledge
Another key aspect of the software architect’s job is his knowledge in technical areas. As mentioned before he must contain up-to-date knowledge in many various areas because all of them could influence the software architecture process. It is not required that he is an expert in each of these fields, it is far more necessary that he has a broad knowledge about this topic, how it works, and why it is important. For example, these fields contain patterns of software development, database structures, modeling languages, or web services and technologies.
But you don’t need to know why an artificial intelligence system uses six hidden layers instead of eight, but you should understand that an AI System generates predictions based on the training data.
Project Planning
If you are carrying out a software project, you are probably going to encounter that the planning of your project is going to proceed over time. At first, you may have an initial “top-down” plan for convincing the management to build the system and provide an estimated cost and schedule. Though, as soon as you were able to convince management of the planned system and you are ready to go, this initial plan is likely going to change. As soon as management is convinced by system and budget the architecture team will set up a first architecture design. In project planning two budget cases can be distinguished:
- Either there is a budget planned for the whole project which includes a schedule as well. This kind of budgeting is called “top-down”.
- Or the budget is solely for the architecture design phase and the overall project budget will be determined by this phase. This kind of budgeting is termed “bottom-up”.
🔄 Description of the Merged Process
There are several pros and cons for both kinds of budgets and ways of planning, yet adopting a combination of both, is ideal to achieve a significant improvement. At first, the architecture team should produce an initial architecture design and a release plan for the system including the kind and time of features being published. After the architecture team has provided an initial design, the assignment of leaders and team building for various parts of the projects should follow. At this stage, an estimation of the cost and an overall schedule can be provided to the management. Usually, this overall schedule provided by the teams will be much more accurate than the top-down schedule. Once the teams have been formed, the design of the software development plan showing the first initial release as the architectural skeleton including the following feature-oriented incremental development can be created. When the software development plan has been provided, the single teams can decide on time and groups for integration as well as the coordination needs for various projects.
💰Economic Guidelines for Architectural Decisions
While planning a project and designing a software architecture there are going to be some major architectural decisions that need to be made. These decisions will be, among other things, led by economic circumstances. The economic discussion leading to architectural decisions is separated into two parts:
- The need of considering the costs of building the system in the first place.
- The long-term costs caused by cycles of maintenance.
These two fields of thought play into the general decision-making context, which consists of the following factors: First of all, you need to consider that major architectural decisions lead to technical and economic implications. Therefore it is always advisable to use your business goals to guide your decisions. While the business goals are always important for guidance, you need to be aware of the interplay between costs and benefits. Because by knowing the costs and benefits of your decisions, you are in the position of making a reasoned selection among competing alternatives. Nevertheless, it is important to mention that you should not conduct an economic analysis for every single architectural decision but only for the most basic ones, like an architectural strategy.
The economic analysis is based on understanding the benefits of decisions in multiple scenarios and transferring them into a comparable format for you. Reaching comparability between different scenarios and benefits can be an especially tricky job, but the effort is worth it. This is why: People who can use the appropriate tools to guide their analysis and decision-making process are an enormous asset to the structured development of a sophisticated software system.
Inside the Organization
But it is not only in the software architect’s responsibility that the software architecture management process is working in a company. Additionally, the organization itself needs to enable directives.
💡 Benefits
It is in their favor especially if you are about to acquire another company. If your software architecture is in good shape, it saves time, energy, and cost to integrate your software to another. Also if your company is going to be acquired, if your software architecture is dependable, it will raise your price tag in comparison to other companies without good software architecture.
🧾 Conditions
But to ensure this, you will need to adequately fund the software architecture management branch. This contains the salary of the software architect himself, but also the tools for the work. Furthermore, you will need to get good people with the appropriate skill set for your company. This requires a qualified human resource management department. To assure that your Software Architect stays for the next years you should be willing to contain a good work environment, with admirable rewards and enough prestige to the position.
☝️Responsibilities
In a project, you should assure clear lines of responsibilities so that you don’t create conflicts of interest between people with the same authorities. Also, try to get the software architect included in every step in the project life cycle so that he can influence it or prepare changes in the architecture and is not surprised with the final results.
🎯Meetings
If you are a large company with many software architects in different projects, it’s useful that you bring them together in a meeting or monthly mentoring sessions, so that they can communicate with each other about their projects and their best practices. For example, if such a meeting doesn’t exist it’s likely to happen that one of the architects does double work, which costs time and wouldn’t be necessary if the software architects have a professional exchange between them. Further, it is helpful if you arrange an external workshop, to maintain their knowledge up to date.
🛠Tools
Last but not least it is important, that everybody in your company works with the same architecture tools. To assure this you can implement a central resource catalog of tools, where the software architects can choose their tools from. Additionally, it is useful to have sufficient documentation for reusable software architecture modules. This minimizes double work.
🧭 Team Organization
As soon as the architecture design is in place, it can be used to determine the project organization. Members of the team that designed the software architecture can become team leaders for the implementation of one part of the architecture. Common roles within software development teams are:
- Team leader: Managing tasks within the team.
- Developer: Constructing and writing subsystem code.
- Configuration manager: Responsible for builds and integrations tests.
- System test manager: Testing of system and acceptance.
- Product manager: Representing marketing; defines system requirements.
🌍 Global Development
Software projects today are oftentimes developed in distributed teams, many of them allocated around the globe for several reasons:
- Cost: It is hoped that the overall project cost can be reduced through development in low-cost countries. Though, this might not be true due to the time-intensive adaptation of domain expertise by the developers and management practices that suit the difficulties of distributed development.
- Skill sets and labor availability: There may not be enough skilled developers in a single location. Therefore distributed development may be the easiest solution rather than relocating developers to the work location.
- Local knowledge of markets: Developers with intense knowledge of local markets can evaluate the need for specific features and cultural issues that may occur.
Consequences which arise through performing globally distributed development is the need for an improved way of allocating responsibilities to teams. If your team is working in the same building, coordination can happen easily through informal contacts like meeting in the coffee room or hallway for example. However, this form of contact is not possible if your team is working distributed. To ensure that the development is coordinated a few options of coordination and exchange need to be considered. First of all, the process must be documented. Therefore, the documentation is a method to coordinate teams in the same office as well as distributed in different locations. Additionally, members of a team need to be able to meet and talk directly to each other since this is sometimes the easiest way of communicating. This can be done remotely through appropriate tools. While personal meetings are important, electronic communication still should be used to communicate quick questions and changes.
To be able to thrive on the advantages of distributed development teams, people and organizations need to learn to communicate effectively with each other in several ways. Until these skills of coordination have been learned, misunderstandings among the teams maybe cause of setbacks and defects in projects.
Open Source Projects
How is software architecture management implemented in large open-source projects with many contributors? We looked and several big projects and how they ensure the quality of their software architecture. These projects are Visual Studio Code, Google Flutter, Apache Spark, Kubernetes, Pandas, TensorFlow, PyTorch, node.js, vue.js. All of the projects we’ve looked into are maintaining the source code and documentation on GitHub.
Although the task of contribution is not just coding. It also includes helping new users, updating the mailings lists, testing and approving code changes, and improving the software documentation. Or as the node.js Community-Committee describes: “code commits !== the only means to contributions.”
The proposal to contribute to a project follows in most cases the same steps:
☝️ Getting Started
First of all, you need an overview of the project, which you will get in the main README.md file in the repository. Here you will find a description of how you can collaborate into the project. Sometimes you will get linked to the contributing page or guidelines. Also, you will find an installation description, which tools are helpful to use, and how to set up the development environment of the project.
👨👩👦👦 Community Management
“Kubernetes is a community project. Consequently, it is wholly dependent on its community to provide a productive, friendly and collaborative environment.” — Kubernetes
The key to a solid open-source project is a healthy community around the project. All the projects we’ve looked at are ensuring a friendly environment and platform to communicate and collaborate together, which invites you to contribute.
“All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.” — Pandas
For communicating there are several tools in use. GitHub is always used for coding tasks. But for structural or organizational issues other chats are being used. For example, TensorFlow uses a Google Group discussion and collects issues e.g. bugs over Stackoverflow. Many projects have their own Slack (PyTorch, node.js, Kubernetes) or discord (vue.js, Google Flutter) channel. Additionally, often projects have regular meetings, which are performed with zoom.
In these communities are some people, who are in charge of new changes and have permission to merge pull requests to the source code. Often these people are the lead developer, who first created the project. For example Evan You in vue.js. But sometimes these are people, who are in the community for a long time period and contribute great work to it. In large projects, these project heads are also often working for large companies, which have an interest in maintaining and developing the project further. For example Adam Paszke (Google), Sam Gross, Soumith Chintala (both Facebook AI) and Gregory Chanan (Facebook) at PyTorch or Greg Van Liew, Uche Nkadi, Daniel Imms (all Microsoft) at Microsofts Visual Studio Code.
Furthermore, some very large projects have organized groups for special segments of the project. At Kubernetes, these groups are called Special Interest Groups (SIGs), which are described as “time bounded Working Groups”. Examples for SIGs are Architecture, Testing, Autoscaling, or Usability. In these groups, chairs are in charge of holding meetings to talk about new changes or issues.
Node.js has similar structures with the implementation of the technical steering committee(TSC) and the community committee(CC).
⛑ Issues Management
“The issue list of this repo is exclusively for bug reports and feature requests.” — PyTorch
To keep track of all open tasks in a project, it is necessary to maintain a management system. Many open-source projects use the GitHub issue management, like Kubernetes, node.js, Google Flutter or TensorFlow. Although some others like PyTorch are using their own forums. Apache Spark uses JIRA for tracking tickets and Pandas also maintaining code triage. Additionally, the project managers are labeling the issues so it is faster to get an overview of a topic.
As a collaborator, you are welcome to create your own issues. Therefore you should always write down which system and software you are using when editing the project source code. Allowing to reproducible the error is. The open-source projects are also wishing for new feature ideas.
📝 Contribute to the Code Base
“If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions to the core, please first open an issue and discuss the feature with us” — PyTorch
If you want to contribute to the source code of the project, you certainly need to obligate some basic values of the project. This contains in all of the projects the code of conduct, which sets the rules for working together.
In most projects, you will also need to sign a Contributor License Agreement(CLA). This CLA defines the terms for using the intellectual property. There are two types, one for individual contributors and one for corporations.
Therefore many projects like Google Flutter have their own values, which all contributors committed to. Also, you should be aware of the style guide of the code, for example. Apache Spark:
“For Python code, Apache Spark follows PEP 8 with one exception: lines can be up to 100 characters in length, not 79.”
TensorFlow:
“Changes to TensorFlow Python code should conform to Google Python Style Guide”
This contains also elements like variable naming. The visual Studio code uses PascalCase for type names & enum values and camelCase for function and method names. Also, they’re describing to “use whole words in names when possible” at their Coding Guidelines.
If you are familiar with this, you can now create a pull request. It’s important to fork from the appropriate branch. For example, the master branch in the vue.js project contains the last stable release, but the dev branch is the relevant branch for merging against. Furthermore, it’s useful to communicate that you are working on a bugfix or issue, to reduce double work.
When you finished the issue, you push your changes to the repository and ask for a review. It’s important to reference in your commit to the issue number you worked on. Then one or two reviewers check your changes and approve them (hopefully). Next, a project director is merging your branch to the repository.
The Corona Warn App
About the Project
The Corona-Warn-App project originates to develop an official COVID-19 exposure notification app for Germany. For the app itself a decentralized approach has been chosen, meaning that the corresponding data will be stored locally on each device to prevent access and control over the data by authorities or any other interested party. As the app is the target of an intense public discussion regarding data protection and privacy, it is entirely open-source to guarantee as much transparency as possible. Officially the German government has assigned SAP and the Deutsche Telekom with the development, however, all interested parties are encouraged to contribute and become part of the developer community. By design, the app has been developed for Germany, though the development should be as open and transparent as possible, also to interested parties around the globe. Therefore, all content is primarily in English.
How to contribute
First of all, all members of the project must abide by the Contributor Covenant Code of Conduct, where basic rules of behavior, like no harassment, polite interaction, and unacceptable behavior have been defined. Additionally, all community leaders have the right to reject comments, commits, code, wiki edits, issues, and other contributions. Also, your contributions must be licensed under the Apache 2.0 License.
Steps to contribute
In the case of the Corona-Warn-App GitHub issues are used to track bugs and improvement requests. When opening up an issue in the collaboration, as much context as possible should be provided and suitable labels should be applied to ensure that the community members can cluster the issues better. Labels that are used in the development of the Corona-Warn-App are bug, feature request or enhancement for example. If one wishes to work on an issue, one needs to first claim it by commenting on the issue and then, after developing a solution, creating a pull request by addressing a suitable maintainer of the repository. The Corona-Warn-App GitHub project has multiple repositories each representing different areas of the whole project, e.g. cwa-app-ios (IOS app), cwa-app-android (Android app), cwa-website (public website) and so on. Each repository has code owners, who are in charge of the area. For the repository cwa-app-ios this is Marc-Peter Eisinger, Christian Kienle and Marc Bometh (all SAP). So far 30 people have been active in the contribution to the development and documentation. For contributing code solutions and fixes, one needs to open up a pull request in which a few criteria need to be met. First of all, commits should be as small as possible, while ensuring that each commit is correct independently. When changes are made they should be tested thoroughly, preferably automated. In the case of manual testing, developers should provide necessary information about the test scope. Work in Progress pull requests should only be created if in need of clarification. In case your pull request requires changes to a commit, you should test the changes and then commit again for another review.
Reviewing Pull Requests
Each pull request needs to be reviewed by exactly two reviewers to ensure that possible mistakes are found. This rule prevents pull requests from being reviewed by more than two people, but still grants the possibility for more reviewers if something is wrong or about to break. Code reviewers are obliged to comment on things they are in the opinion of being wrong but in the end, the pull request creator is responsible for the code. After receiving two approvals, the pull request can be merged.
Upshot
Overall, it can be seen that proper software architecture management is needed to ensure a high-quality outcome. However, as seen in the open-source examples, there are multiple ways to achieve this. This leads to the insight, that most importantly for a project to be successful, adequate structures need to be in place. This includes a well-managed community, in which everybody feels comfortable and welcome, a management of the occurring issues and a mutual framework, which declares rules and guidelines for contribution.
Further Reading
Bass, Len, Paul Clements, and Rick Kazman. Software architecture in practice. Addison- Wesley Professional, 3rd Edition, 2013.
Bass, Len, Paul Clements, Rick Kazman, and Mark Klein. Models for Evaluating and Improving Architecture Competence. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, Technical Report CMU/SEI-2008-TR-006, 2008.
Boehm, Barry & Supannika Koolmanojwong, Jo Lane, and Richard Turner. Principles for Successful Systems Engineering. Procedia Computer Science. 8. 297–302. 10.1016/j.procs.2012.01.063, 2012.
Britto, Ricardo, Darja Šmite, and Lars-Ola Damm. Software Architects in Large-Scale Distributed Projects: An Ericsson Case Study. IEEE Software, Bd. 33, Nr. 6, S. 48–55, 2016.
Clements, Paul, and Len Bass. Relating Business Goals to Architecturally Significant Requirements for Software Systems. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, Technical Note CMU/SEI-2010-TN-018, 2010.
