103rd Monthly Technical Session
Hello fellow technologists!
Here at HENNGE, we are passionate about sharing knowledge, especially about the latest and greatest in the world of tech. This time around, we are graced with the presence of two interns at our office, Arung from Indonesia and Nikhil from India. Together with a few other HENNGE-ers, they talked about the latest technologies that they found interesting and useful. Some of these topics include AWS Graviton for cheaper and more efficient computing on AWS, common pitfalls and workaround in carrying-out schema change in production databases, programming slack messages and bots, and many others.
Ownership and Borrowing in Rust | Nikhil
In this session, Nikhil talks about how memory management works in the Rust programming language with the concept of ownership and borrowing. This is in contrast with other programming languages, which either have automatic memory management in the form of a garbage collector or manual memory management, where the user of the language must carefully manage the allocation and deallocation of memory of their programs.
Ownership in Rust consists of three rules:
1. Each value has a variable called its owner
2. There can only be one owner at a single time
3. When the owner goes out of scope, the value is dropped
Ownership of a value is transferred when:
- assigned to a different variable
- passed into a function
- returned from a function
As a consequence of the ownership rules, a value that is passed to a function will have to be passed back via function return if it needs to be used after the function is finished. This is where borrowing comes in, it gives function read-only access (a reference) to a value, where that value will not be dropped when the function ends and will still be available thereafter.
Reference (aka borrowing) in Rust consists of two rules:
1. At any given time, you can have either one mutable reference or any
number of immutable references.
2. References must always be valid.
There are a lot more topics around ownership and borrowing which are the flagship feature of the Rust language. This talk only gives a primer on them and Nikhil suggested the audience to dive into it to learn more.
A better way to not misconfigure terraform codebase: Terramate | Arung
Nowadays, engineers use Infrastructure-as-code tools to manage their infrastructure on the cloud, usually using terraform. If you see terraform for the first time, you might immediately recognize how cool it is to be able to provision any cloud resources imaginable via declarative code and have them created in a single apply command.
However, as with any other codebase, the bigger the project becomes, the harder it is to manage the increasingly larger amount of terraform code. This gets exacerbated when your project requires multiple deployments in different environments, for example, or when each deployment requires multiple modes, etc. This results in multiple long-term problems, such as:
1. Too much repetition
2. Change to all/most resources would be a big pain (eg. adding tags to every resource, adding prefix, etc)
3. “locals” block is tiring to see
4. Misconfiguration
So how to tackle this problem? Arung offers two solutions available in the market.
Terragrunt is much more popular than terramate and has been here longer. However, its main functionality is limited to a templating engine/code generation tool for terraform code. Terramate, on the other hand, tries more to become an orchestration tool for terraform, which also includes code generation capability. For example, it enables you to perform terraform commands recursively (init, apply, etc.) in many subdirectories in a single time, without needing you to visit each subdirectory one by one.
In terms of code generation, one of the coolest things that terramate allows is specifying global values that get inherited to every subdirectory below.
This allows us to focus only on the values specific to a subdirectory and let other values be inherited from parent directories. This will minimize the chance of misconfiguration in the deepest subdirectories of you terraform codebase.
As with anything, terramate also has its own downsides:
- There’s still a need to create a configuration file on each directory (although it’s minimized and tidier).
- There’s still repetition, just in a different form (e.g. need to regenerate code on each new change).
- No intellisense (vscode extension is still on the way).
- A bit counterproductive since you need to add new files when what you really want is to make the code cleaner.
Graviton: Moving from x64 to ARM on AWS | Hans
In this session, Hans talked about graviton processors offered by AWS in their cloud services offering. Currently, AWS has released three generations of graviton processors family, and they are available to use on EC2, ECS, and AWS Lambda. AWS claimed that utilizing the latest generation of graviton processors (Graviton3) can lead to up to 40% better price performance over comparable x64 instances.
AWS customers can easily use graviton just by selecting it on each service. On EC2, the user needs to select 64-bit (Arm) architecture and corresponding graviton instance types such as t4g or c6g, where the g at the end stands for graviton. On ECS, user need to select ARM64 for cpuArchitecture and LINUX for operatingSystemFamily. On AWS Lambda, user need to select arm64 architecture.
Building your code for ARM, however, is a whole different beast. Basically, if your build machine (such as CI runner) runs on the same ARM architecture, then the building is a breeze; you don’t need to consider anything. However, if your build machine is not an ARM machine (which is most likely), you will need to resort to a different workaround to build your code to run on graviton. Such is the case, for example, on Github Actions where it only supports x64 architecture.
Hans shared some of his experience of building for graviton using these workarounds:
- Docker build needs to be performed using docker buildx + QEMU emulator
- With Alpine Linux base image, the build hung for 20 minutes. Workaround: use slim base image.
- Building a Python project with poetry takes about 20% longer
- Building react static file with Yarn; the build hung for 30 minutes. Workaround: build outside docker, and copy in the static file.
In terms of pricing, running on graviton will cost about 20% cheaper. However, AWS claimed it could save up to 40%, which turns out also considers graviton’s faster performance. The 20% cheaper pricing, though, is still beaten by another low-cost offering of AWS in the form of Spot instance.
In conclusion, switching to graviton on AWS is relatively straightforward, but building for it takes some effort. Saving 20% in cost is nice, but still not as cheap as spot instance. If you have the machine to build for Arm natively, then it’s a pretty easy recommendation to switch from x64 to graviton. If not, another alternative like Spot is probably a better option.
Programmatically Sending Slack Messages | Bagus
In this session, Bagus talked about the many ways of sending messages programmatically. There are a lot of reasons one would like to program for sending Slack messages, but one of them is Chatops. Chatops, or conversation-driven DevOps, is the use of chat clients, chatbots, and other real-time communication tools to facilitate software development and IT operations tasks. In this case, the real-time communication tool is Slack.
There are many ways of programming for sending Slack messages, which Bagus has summarized in the following table.
This time, Bagus shared his experience using two of these methods, chat.PostMessage and incoming webhooks. Firstly, both of them require the creation of Slack Apps, which will provide the token required for our program to access Slack’s API.
The chat.PostMessage method then requires at least two things:
- The slack app oauth token, with chat: write authorization
- A channel id, which the Slack app is a member of
The incoming webhooks method only requires the following:
- The webhook url, which can be created in the Slack app configuration. Here we need to select the target channel the webhook will post into.
The next topic Bagus touched on is how the message looks in Slack, and the message layout. There are two ways of defining the layout for slack messages, blocks, or secondary attachments.
With blocks, the text objects are rendered in a compact format that allows for two columns of text side-by-side. It’s pretty clean, and there are no colored borders on the left side of the message. Secondary attachment, on the other hand, is a legacy part of Slack messaging functionality. It acts as a secondary part of messages or additional content that adds context to the main message. As such, it conforms to a very strict order of visual elements, and it is not possible to change where a certain field is rendered in the message. Bagus recommended the use of blocks instead of secondary attachments.
Pair Programming | Olivia
In this session, Olivia talked about pair programming, a software development technique where two programmers simultaneously work on the same tasks on the same machine. Although a lot of us have heard of the term before, it remains a question of why the practice is not widely adopted in the industry. Some might consider it a waste of resources because making two programmers work on two different tasks will be doubly efficient. Others view the practice as time-consuming and energy-draining because a lot of communication and discussion are involved, which some people might feel uncomfortable about. However, the advantages of pair programming don’t immediately show up. It gets more beneficial in the medium and long run. It’s proven to be an essential practice to boost communication within the team and also boost cooperation. It eventually leads to more quality programs and fewer bugs.
There are a few styles of doing pair programming. The classic and most common style is the so-called driver and navigator style. The driver is the person who has control of the keyboard and the mouse and is in charge of writing code. The navigator guides and helps the driver to code while at the same time actively reviewing each line of code that the driver wrote. While the driver needs to have tactical thinking and focus on tiny goals, such as how to solve this small task through a programming language, the navigator needs to focus on the logical process and observe the driver with a strategic mindset. They also need to take note of potential bugs and has to keep the big picture in mind.
The ping pong style is a task-driven development technique. It consists of three steps red, green and refactor. Red means programmer A need to write a test case that will fail, and green means programmer B builds the feature to pass the test. Step three, which is optional, is to refactor together. After that, we start the next red-green-refactor, like playing ping-pong.
Another style of pair programming is strong style pairing. It’s very useful for knowledge transfer. The rule is for an idea to go from your head into the computer; it must go through someone else’s hands. The driver usually is someone who’s new to the knowledge, the language, the tool, or the codebase. The navigator, who usually is the more experienced person, gives clear instructions at the highest level of abstraction. The driver can understand for the driver to implement what you want them to while learning the language or knowledge you want to transfer.
Some benefits of pair programming are:
- knowledge sharing
- collective code ownership
- combined two modes of mindset: tactical and strategic
- reflection
- instant feedback and code review on the go
- better quality code and team flow
Challenges of pair programming are:
- can be exhausting and energy consuming
- constantly observed by someone sitting next to you
Handling Postgres Schema Changes Without Downtime | Liam
In this session, Liam talked about a case study on how to change Postgres schema without causing downtime. A few months back, the Kumamushi team had to perform a schema change for a table in their production database. It consists of a few columns, one of which is page_views, which currently has a type of integer and needs to be changed into bigint. The table itself has about 20 million rows of data. The change itself is really simple and can be executed via the following query:
ALTER TABLE product ALTER COLUMN page_views TYPE BIGINT;So what happens when we execute such a query to the table? Yup, it causes a lock on the table, and no operations can be done while waiting for the change, and because 20 million rows are quite a lot of data, it takes quite a bit of time. So every query from users fetching up or deleting from the records gets queued up, and nothing really moves until it finishes.
So what is table lock in posgres? There are mainly three types of locks in postgres:
- ACCESS EXCLUSIVE: blocks all usage of the locked table
- SHARE ROW EXCLUSIVE: blocks concurrent row modifications and DDL operations against the table but still allows reads.
- SHARE UPDATE EXCLUSIVE: blocks concurrent DDL operations against the locked table.
And as it turns out, the previous query does cause ACCESS EXCLUSIVE lock, which prevents any other operations from taking place. So what should we do if we really want to make this change?
Similar to the steps shown above for renaming a table, altering a column type safely will consist of several steps:
- add a new column with the new type
- backfill the new column with the values from the old column
- add triggers on update insert delete to copy any new additions to the old column into the new columns
- when the backfill is done, we can switch to the new column and drop the old column
And that was all the sessions of this edition of MTS. We learned a lot of different topics, from infra related to databases, programming language, slack messages, and even the practice of pair programming. And as usual, we end the day with a big Kanpai, and we enjoy each other’s company while continuing the discussion about technology.
