Everything as a code: Part 1
Disclaimer
It’s just a concept. We can take some useful techniques from it but it’s not mandatory to follow all of them in the one project.
What the benefits do we have using code?
- Engineers are lazy (isn’t?)
- Engineers write code a lot of years already.
- During these years we implemented a lot useful tools that help us to write and deliver our code faster and easier.
How many of you use git with all the benefits of having different versions of code, parallel work and comparing functional?
How may of you use CI pipelines that checks your code style, run tests, search vulnerabilities?
The question is: why we can’t use all these useful tools not only with our production code, but with all the possible aspects of our software development responsibilities?
The “Everything as a code” philosophy aims the goal to make it real. We will review different approaches and tools that will support us to be closer to this philosophy, probably will save your time and will give you ability to continue be lazy =)
Processes as a code
Everything starts from processes building. You have it in any case (even if in your company processes are not formalised — they still exist).
On the high level we can describe our sofrtware development process in the next way:
Just three simple steps, everyone is able to remember it.
But in reality it usually looks a bit more complex
And each if these rectangles are separate sub-processes, for example architecture sub-process may look like this:
or development sub-process may looks like this:
Now it’s going to be complicated to remember all these steps and especially follow them all the time without any mistakes.
This is where workflow as a code and CI tools are taking their place.
This is how the architecture workflow may be automated described as code:
name: "Check HLD documents content"
on:
pull_request:
types: [opened,edited,synchronize]
jobs:
changed_files:
runs-on: ubuntu-latest
name: Test changed-files
permissions:
pull-requests: read
steps:
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v40
- name: Checkout
uses: actions/checkout@v3
- name: Verify all the HLD files
run: |
search_words=("Overview" "Functional requirements" "Non-functional requirements" "Solution" "Involved flows")
for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
if [[ "$file" =~ \.MD$ ]]; then
echo "$file verification..."
all_present=true
for word in "${search_words[@]}"; do
if ! grep -q "$word" "${{ github.workspace }}/$file"; then
all_present=false
break
fi
done
if [ "$all_present" = true ]; then
echo "File '$file' passes validation."
else
echo "File '$file' does not pass validation."
exit 1
fi
fi
done
name: Notify Teams
on:
pull_request:
types: [closed]
jobs:
notify_slack_push:
name: Notify Slack
runs-on: ubuntu-latest
steps:
- uses: abinoda/slack-action@master
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
with:
# message to send to slack
args: '{\"channel\":\"${{ secrets.SLACK_CHANNEL_ID }}\",\"blocks\":[{\"type\":\"section\",\"text\":{\"type\":\"mrkdwn\",\"text\":\"*HLD file was populated. PLease start the LLD [process}\"}},\{\"type\":\"mrkdwn\",\"text\":\"<${{ github.event.repository.html_url }}|View Repository>\"}}]}'
if: success()
We have preconfigured branches, that will not let you avoid design review. The verification steps will be performed automatically, all the notification will be sent as soon as your design review will be done.
If our HLD doesn’t follow the structure — our pipeline will be failed. According to a repository rules — new HLD can be merged only after the review from architecture squad.
The example above was written using GitHub actions.
Some alternatives
GitHub actions: https://github.blog/2022-02-02-build-ci-cd-pipeline-github-actions-four-steps/
GitLab CICD: https://docs.gitlab.com/ee/ci/
Jenkins pipelines: https://www.jenkins.io/doc/book/pipeline/
Teamcity pipelines: https://www.jetbrains.com/teamcity/features/configuration-as-code/
The advantages of having this processes as code
- You cannot skip a part of process (at less it not so easy)
- A lot of action may be automated
- All the process changes can be easily reviewed and the history of this changes is available
Architecture as a code
The idea of “architecture as code” approach has next bullet points:
- Store your code documentation as close to your code as possible
- Use tools that will let you write documentation as a code (text and diagrams)
Mermaid + Markdown
Let’s review a pretty old but still useful combination.
Markdown:
Markdown is a lightweight markup language that is commonly used to format and structure plain text documents. It was created by John Gruber and Aaron Swartz in 2004 and is designed to be easy to read and write, while also being easy to convert into HTML. Markdown is often used for creating documents that can be displayed on the web or in various text editors (including all the popular git repository implementation). Official website: https://www.markdownguide.org/
Mermaid:
Mermaid is a popular and versatile open-source JavaScript library that allows you to create diagrams and flowcharts directly in your web-based documents and applications. It is often used in Markdown documents, such as README files, and in various web development contexts to generate visually appealing diagrams and charts. Official website: https://mermaid.js.org/
Here is an example of how the HLD can look like using markdown+mermaid combination:
# Payment processing system design
## Overview
_System processing payments.
Takes care about durable and fast payment processing.
Service built to replace a legacy payment processing component._
## Functional requirements
Service should support a payment processing feature which contains next steps:
- Payment creation
- Bank verification of payment
- Financial monitoring of payment
- Tracing all the payment statuses changes and errors that may occur during the process
## Non-functional requirements
As a part of requirements service should support next non-functional features:
- **performance:** the latency of the service should be 1000ms (at least 50% smaller than in legacy systems). We should be notified about all the cases that cross this threshold.
- **durability:** payment process should be highly durable. Each payment status should be a save point from where the system should be able to continue the process. All the errors and unusual cases are reported.
## Solution
The service is called by payment processing web application,
makes calls to bank data system, financial monitoring system,
kafka message broker and its own database...
```mermaid
C4Context
title System Context diagram for Payment processing system
Person(BankAnalyst, "Bank analyst", "Bank employee responsible for payment analyses")
Rel(BankAnalyst, PaymentReportingApp, "makes payments reconciling", "HTTP")
Person(PersonalBankCustomer, "Personal bank customer", "A customer of the bank that has a personal bank account")
Rel(PersonalBankCustomer, PaymentProcessingApp, "makes payments checks payments results", "HTTP")
System_Boundary(SystemBoundary, "Bank payment processing system") {
Container(PaymentReportingApp, "Payment reporting web app", "TypeScript, Angular", "Web application providing payment reports")
%% FE Apps:
Rel(PaymentReportingApp, PaymentReportingService, "takes payments reports", "HTTP")
Container(PaymentProcessingApp, "Payment processing web app", "TypeScript, Angular", "Web application providing payment processing functionality")
Rel(PaymentProcessingApp, PaymentProcessingService, "starts payment processing
reads payment statuses", "HTTP")
%% BE Services:
Container(PaymentReportingService, "Payment reporting service", "Java, Spring boot", "Service provides reports for analysts.
Reads and agregates data from processing database + external bank database")
Rel(PaymentReportingService, PaymentProcessingDatabase, "reads payments data", "JDBC")
Rel(PaymentReportingService, BankDataSystem, "reads payments data", "HTTP")
Container(PaymentProcessingService, "Payment processing service", "Java, Spring boot", "Service processing payments. Takes care about durable and fast payment processing")
Rel(PaymentProcessingService, PaymentProcessingDatabase, "saves/reads payment processing details", "JDBC")
Rel(PaymentProcessingService, PaymentsStatusesTopics, "puts/read payment ids with statuses", "TCP")
Rel(PaymentProcessingService, FinancialMonitoringSystem, "send payments to check", "HTTP")
Rel(PaymentProcessingService, BankDataSystem, "saves and retrieves payments
with final statuses", "HTTP")
%% Storages:
ContainerDb(PaymentProcessingDatabase, "Payment processing database", "Oracle Database 12", "Stores payment processing details, errors and statuses")
ContainerQueue(PaymentsStatusesTopics, "Payments statuses topics", "Apache Kafka", "Provides event messaging for payment processing")
}
System_Boundary(ExternalSystemBoundary, "External systems") {
System(FinancialMonitoringSystem, "Financial monitoring system", "External financial monitoring system")
UpdateElementStyle(FinancialMonitoringSystem, $bgColor="grey")
System(BankDataSystem, "Bank data system", "Core legacy bank system. Stores the bank data.")
UpdateElementStyle(BankDataSystem, $bgColor="grey")
}
UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="1")
```
_The service is called by payment processing web application, makes calls to bank data system, financial monitoring system, kafka message broker and its own database._
## Involved flows
### Payment processing sequence:
```mermaid
sequenceDiagram;
Payment processing web app->>+Payment processing service: send payment
Payment processing service->>+Payment processing database: create new payment
Payment processing database-)-Payment processing service: created payment
Payment processing service->>Payments statuses topics: add to payment "created" topic
Payment processing service->>+Bank data system: verify payment by bank
Bank data system-)-Payment processing service: verification results
Payment processing service->>+Payment processing database: save processing results
Payment processing database-)-Payment processing service: updated payment
Payment processing service->>Payments statuses topics: add to payment "verified by bank" topic
Payment processing service->>+Financial monitoring system: verify payment by fin. mon
Financial monitoring system-)-Payment processing service: verification results
Payment processing service->>+Payment processing database: save processing results
Payment processing database-)-Payment processing service: updated payment
Payment processing service->>Payments statuses topics: add to payment "processed" topic
Payment processing service->>+Bank data system: save to bank system
Bank data system-)-Payment processing service: saving results
Payment processing service-)-Payment processing web app: processing results
```
1. User starts payments processing using web application
2. Web application calls a processing method in payment processing service (PPS)
3. PPS saves the payment data in its database \
3.1. If some error occur - PPS sends payment id, status and error message to error kafka topic and stops the flow. \
3.2. If saving was successful - PPS sends payment id to a “created payments” kafka topic and go to “4.”
4. PPS calls “verify payment” jdbc metod of “bank payment system” \
4.1. If some error occur - PPS sends payment id, status and error message to error kafka topic and stops the flow. \
4.2. If verification was successful - PPS sends payment id to a “verified by bank payments” kafka topic and go to “5.”
5. PPS calls “verify payment” restmetod of “finantial monitoring system” \
5.1. If some error occur - PPS sends payment id, status and error message to error kafka topic and stops the flow. \
5.2. If verification was successful - PPS sends payment id to a “verified by financial monitoring payments” kafka topic and go to “6.” \
**TBD:** _to increase the process speed - we can make 4 and 5 in parallel. 6 in that case should wait until payment will be moved to “verified by bank and fin. mon.” status. #After_MVP_
6. PPS saves the payments in bank payment system using “save payment” jdbc method. \
6.1. If some error occur - PPS sends payment id, status and error message to error kafka topic and stops the flow. \
6.2. If savingwas successful - PPS stops the flow. \
**TBD:** _implement an “error listener” which will listen “error kafka topic” and based on the error message and payment status will make some actions on them (retry/ error notification/cancel etc.) #After_MVP__
### Payment processing service flow:
```mermaid
flowchart TD;
S((Start)) --> CP(Create payment) --> EC1{error occur?}
EC1 -- Yes --> SE(Send error) --> E((Finish))
EC1 -- No --> SSS1(send successful status) --> VB(verify payment by bank)
VB --> SPD1(save process details) --> EC2{error occur?}
EC2 -- Yes --> SE(Send error)
EC2 -- No --> SSS2(send successful status) --> VF(verify payment by fin. mon)
VF --> SPD2(save process details) --> EC3{error occur?}
EC3 -- Yes --> SE(Send error)
EC3 -- No --> SSS3(send successful status) --> SB(save to bank system)
SB --> EC4{error occur?}
EC4 -- Yes --> SE(Send error)
EC4 -- No --> E
```
_Text description..._And this is how the same file looks like in github (or in preview mode of your IDE):
Looks pretty similar to what we can find in our confluence pages ;)
Some alternatives:
AsciiDoc is a plain text markup language for writing technical content. It’s packed with semantic elements and equipped with features to modularize and reuse content. AsciiDoc content can be composed using a text editor, managed in a version control system, and published to multiple output formats. Markdown alternative. https://asciidoc.org/
PlantUML is a text-based diagram generation tool that allows users to generate various types of diagrams using a simple syntax. It is open-source, and supports a wide range of diagram types, including sequence diagrams, flowcharts, activity diagrams, and class diagrams. Mermaid alternative. https://plantuml.com/
The advantages of having this HLD as a code
- We are able to automate some parts of our architecture review
- The human review can be more productive (you are able to leave a comment on the specific shape or specific relation in diagram and the author doesn’t need to store some list of open questions separately or rewatch the “review meeting recording” to remind himself what was discussed.
Next part: https://medium.com/@svosh2/everything-as-a-code-part-2-11f849a43c6c
