We live in a Digital Transformed world, where technology allows new forms of work in a high speed changing environment. Traditional businesses are challenged by start-ups and tech companies, with innovative and disrupting business models. New apps and services are created and become obsolete in a blink of an eye. This new generation of consumers has the power of choice on their hand since they are kids. COVID-19 brought a new normal to the way we live putting companies under pressure to change or die, and the technology is the great lever, even though at the same time create new challenges.

In this scenario, every day new ideas and insights come from innovators and are almost immediately copied by followers to become the new reality with innovations on top of it, creating a virtuous cycle of continuous improvement also new ways of doing things. Mobile, apps, tablets, and the internet of things are just the tools to live in the new normal.

The time between coming up with ideas or insights and its materialisation has to be shorter than ever before. Otherwise, the momentum is lost while competitors are winning space. When consumers face errors and problems using the new technologies, the users don’t come back, and that can bring a company to lose its lead

Challenges and solutions to the new tech

In order to respond to this fast speed, connected world, the traditional development, test, production and operation models does not fit anymore, creating bottlenecks and friction when speaking of the relationship among teams. Each of the technology areas ends up becoming a silo with strict interaction rules.

At one side of the ring, we got development, trying to answer in its best, in a faster way, the business insights, using agile methodologies, modern architectures and languages. On the other corner, IT operations, in a quest for stability and control for the production environments, creating processes and procedures to ensure that every piece of code released to production would be stable in order to avoid incidents, not forgetting to protect what is already running.

This enormous abyss between Development and Operations brings titanous clashes, slowing down the delivery time and problem resolution.
To reduce the friction also allow business ideas to become features to service consumers, the DevOps concept was forged around 2010; a concept that grew and, during the last years, are helping to change the IT landscape.

DevOps: concepts advantages

It is hard to find a unique definition for DevOps, as sometimes the market is looking at this matter as a group of blind men trying to see an elephant with their hands. Each one touches a different part and have a different view of it.

First of all, DevOps is a work culture, bringing software development near IT operations, closing the gap between those areas and harvesting the fruits of this gathering.

DevOps is not a methodology or a tool, but it’s a set of practices, built on top of automation, communication and shared objectives, changing organisational cultures to bringing to life a new way to deliver IT. DevOps includes the whole Design, Build and Operate IT lifecycle, unifying these processes with governance and security serving as its basis, sewed up with automation, on an agile way of work.

In order to ensure that the business is aligned to this new industrial revolution, with today’s fast information consumption society and, consequently, to the Digital Transformation, there is a pressing need for organisational changes; a big change, of course, that embraces much more than the IT department in itself and even more than the Operations and Development relationship, creating an arc from the business to the consumer.
Therefore, IT tends to be seen not only as a support area and more like a part of the business.

This concept disruption is enormous, as silos must be broken, communication plans must be clear, concise and need to be followed and it is necessary to stop looking just to task executors and starting to see the whole service. When the business starts a more holistic approach to the development, test, production and operation cycle, without barriers between them, they will be, almost automatically, a DevOps business.

At DNX Solutions, we work to bring a better cloud and application experience for digital-native startups in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery and Service Mesh and Data Solutions (movement, transformation, lakes, warehouses and analytics). We are constantly hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin

What DevOps is and how its contributes to the Digital Transformation was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Amazon MWAA (Managed Workflow for Apache Airflow) was released by AWS at the end of 2020. This brand new service provides a managed solution to deploy Apache Airflow in the cloud, making it easy to build and manage data processing workflows in AWS.

MWAA enables automatic deployment of all the infrastructure and configuration for Airflow Web Server, Scheduler, Workers, Metadata Database and also the Celery executor combined with SQS, to manage jobs dispatching. Users just need to setup an S3 bucket for DAGs, plugins and Python dependencies (via requirements.txt) and associate its content with the MWAA environment.

Authentication is also managed by AWS — native integration with IAM and resources can be deployed inside a private VPC for additional security.

Since all the resources are deployed by AWS, developers don’t have access to the underlying infrastructure. Consequently, the main interface used by Data Engineers is the Airflow UI, which is available via public URL or VPC endpoint, depending on the deployment type selected (public or private network).

However, Airflow UI is not the only option for interacting with your environment; MWAA also provides support to the Airflow CLI. This is a useful option if you want to automate operations to monitor or trigger your DAGs, and in this post I explain how you can best make use of Airflow CLI from an MWAA environment.

The following content is suitable for those already familiar with the benefits and functionality of Apache Airflow. If you are new to Airflow, or you’re searching for more insights about the advantages of using Amazon MWAA compared to hosting your own environment, I recommend you explore earlier posts on this subject before reading on.

The Apache Airflow CLI and its use with Amazon MWAA

Airflow has a very rich command-line interface that allows for many types of operation on DAGs, starting services, and support for development and testing.

Airflow CLI is an interesting maintenance alternative within MWAA, since it allows Data Engineers to create scripts to automate otherwise manual/ repetitive tasks.

In MWAA, not all commands are supported because as developers we cannot perform operations that might impact server resources or user management (e.g. webserver, scheduler, worker, etc), but all commands related to monitoring, processing and testing DAGs are supported in the current version.

To check the full list of supported and unsupported commands, refer to the official User Guide. At the time of writing, this is the status of different commands:

List of supported commands

• backfill
• clear
• dag_state
• delete_dag
• list_dag_runs
• list_dags
• list_tasks
• next_execution
• pause
• pool
• render
• run
• show_dag
• task_failed_deps
• task_state
• test
• trigger_dag
• unpause
• variables
• version

List of unsupported commands

• checkdb
• connections
• create_user
• delete_user
• flower
• initdb
• kerberos
• list_users
• resetdb
• rotate_fernet_key
• scheduler
• serve_logs
• shell
• sync_perm
• upgradedb
• webserver
• worker

How to trigger a CLI command from Amazon MWAA

To access the Airflow CLI from MWAA, there are four basic steps:

1. Authenticate your AWS account via AWS CLI;
2. Get a CLI token and the MWAA web server hostname via AWS CLI;
3. Send a post request to your MWAA web server forwarding the CLI token and Airflow CLI command;
4. Check the response, parse the results and decode the output.

This sounds complicated but is actually a fairly straightforward process. Let’s deep dive and investigate each step in detail.

1. Authenticate to your AWS account

To access your MWAA cluster, you must install and configure AWS CLI, granting access to the account where your environment is deployed.

If you are not used to this process, read the AWS CLI User Guide, which explains how you can configure a profile in your AWS CLI and grant access to your accounts.

2. Get CLI token and MWAA web server hostname via AWS CLI

The next step is to collect a CLI Token which is a Bearer token used for authentication in your MWAA environment. When first authenticating in the AWS account we can also authenticate to the MWAA environment and collect the token which grants access to perform Airflow CLI commands, by entering the following command :

aws mwaa create-cli-token --name $MWAA_ENVIRONMENT

Remember to assign the name of your MWAA environment by exporting the environment variable $MWAA_ENVIRONMENT.

export MWAA_ENVIRONMENT=my_environment_name

If the command is successfully executed, you should receive a JSON response with two attributes:

{
  "CliToken" : "",
  "WebServerHostname" : ""
}

Parse the results and store them in other environment variables for later use. I suggest the following names:

• $CLI_TOKEN
• $WEB_SERVER_HOSTNAME

3. Send a post request to your MWAA web server forwarding the CLI token and Airflow CLI command

Finally, using the CLI token and the web server hostname, you can trigger your Airflow CLI command via curl request by following the example below:

curl \
  --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" \
  --header "Authorization: Bearer $CLI_TOKEN" \
  --header "Content-Type: text/plain" \
  --data-raw "$AIRFLOW_CLI_COMMAND"

Notice we assigned the environment variables acquired from the previous step ($CLI_TOKEN and $WEB_SERVER_HOSTNAME) and also published a third variable with the name $AIRFLOW_CLI_COMMAND. In this variable we send the Airflow command to be performed by the CLI, for example, if you want to execute the following command:

airflow list_dags

The variable $AIRFLOW_CLI_COMMAND should be filled with:

list_dags

Important note: if your MWAA environment is published in a private network you can’t perform the curl request via public internet; a VPN must be used to establish the connection between your local machine and the VPC endpoint, or you may need to execute this command from another computing resource placed inside of the same VPC.

4. Check the response, parse the results and decode the output

Finally, the last step is to parse and decode the output of the curl request. If everything went well you should have received a JSON response with the following attributes:

{
  "stderr" : "",
  "stdout" : ""
}

Notice both attribute values are encoded in Base64. Remember to decode the results to collect the final output from Airflow CLI. A simple way to achieve that is by using the command:

base64 -d

Automating all the steps with a single script

In the previous sections we discussed all the steps needed to run a CLI command in MWAA, but now I’ll describe how to combine everything in a single script which enables you to quickly and easily perform CLI calls.

This is a shell script created for Unix based operational systems (e.g. Linux, MacOS); if you are running Windows you may need to adapt this content or run the script through Windows WSL (Windows Subsystem for Linux).

Notice I am using jq to parse the JSON responses from AWS CLI and the curl request to MWAA, but feel free to adapt the code if you prefer another approach.

The script above collects all the arguments and send it to the curl request by using the variable $*. So, if we name this script as airflow-cli.sh and you type the following command in your terminal:

airflow-cli.sh list_dag_runs my_dag_name

The MWAA environment will perform the following CLI command:

airflow list_dag_runs my_dag_name

An interesting trick to improve the user experience is to rename this script as airflow and copy it to one of the folders mapped in the local $PATH (e.g. /usr/local/bin/airflow). In this way you can call the commands in the Airflow CLI by typing:

airflow <arguments>

Just ensure you don’t have the real Airflow CLI installed, to avoid conflicts.

If you already have Airflow CLI installed, another option is to run this script from a Docker image and map it to the container local path.

Conclusion

Amazon MWAA is an incredible service which reduces the complexity of managing an Apache Airflow cluster, thus enabling Data Engineers to focus on DAGs and the data workflow instead of spending endless time on infrastructure.

The Airflow UI continues to be the primary means of interaction with MWAA, but the additional option of using Airflow CLI allows advanced users to maximise the benefits of MWAA, and take advantage of all the features of a hosted solution.

I hope you enjoyed this content and make good use of this script in your Amazon MWAA environment!

References:

• Amazon Managed Workflows for Apache Airflow — User Guide: https://docs.aws.amazon.com/mwaa/latest/userguide/amazon-mwaa-user-guide.pdf
• Using a CLI token: https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#CreateCliToken

At DNX Solutions, we work to bring a better cloud and application experience for digital-native companies in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, and Service Mesh and Data Solutions (movement, transformation, lakes, warehouses and analytics).

We are always hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at https://github.com/DNXLabs and follow us on Linkedin or Facebook.

How to use Apache Airflow CLI with Amazon MWAA was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

The challenge:

If you don’t know Cpanel, think of it as an all-in-one open-source box. Cpanel is a powerful hosting server with built-in services such as Apache, PHP, Mysql, and Named for DNS. It allows you to create customised hosting packages and host customers over them. Usually, it’s well suited to small customers. The biggest challenge starts when you put more and more load within the same box.

What about a scenario where you don’t have a single hosting package although you have built your solution years ago over Cpanel APIs? These APIs are used to provision email accounts, FTP users, DNS records, and so on.

You would end up with an infrastructure design like this:

Even though you have a big server to handle all the workloads, what happens if the server goes down? And if you have that server hosted in a third-party data center? You would have to raise a ticket and wait for their answer. Meaning a big downtime ahead.

Solution:

Since decoupling services and applications into micro-services is a complex and long process, the fastest solution was to lift and shift the server to a similar Cpanel server on an EC2 instance.

That choice gave time to the developer teamwork on the overwhelming applications while allowing some standard applications to be pushed to the Cloud.

Previously, all the PHP Laravel APIs running on that box were modernised to docker images and deployed on ECS.

Check it out our open-source terraform ECS cluster module:

DNXLabs/terraform-aws-ecs

And our module to deploy ECS applications such as PHP Laravel:

DNXLabs/terraform-aws-ecs-app

If you want to find out more about application modernisation for PHP, Node, Ruby, etc, check out our website:

application modernisation - DNX Solutions

Phase 1 - Decoupling the most overwhelming Cpanel services

DNS to Route 53:

Dumping all Route 53 records from a Cpanel server is a pain in the neck. Why’s that? Because Cpanel usually creates a lot of ‘useless’ records if you’re creating a single domain or subdomain.

We managed to have all the DNS records and values and we’ve created all of them on Route 53.

There is no simple way to dump the DNS records on Cpanel. Be aware of that.

Mysql to RDS:

The databases on Cpanel are created with a strange prefix to avoid duplication. We’ve got a dump of all databases, and using the python script below, we ran a 40 or so parallel syncs to restore all databases:

DNXLabs/tools-box

Phase 1 final design:

Now we have a server running on AWS and at least some ways to handle the forthcoming problems.

Right now, the solution is not yet finalised, so stay tuned for our next posts where we will update you with the next stages of this migration.

To Be Continued…

At DNX Solutions, we work to bring a better cloud and application experience for digital-native companies in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, and Service Mesh and Data Solutions (movement, transformation, lakes, warehouses and analytics).

We are always hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at https://github.com/DNXLabs and follow us on Linkedin or Facebook.

Migrating an on-premise Cpanel to AWS was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

On December 1, 2020, Amazon announced ECR Public and ECR Public Gallery.

We at DNX Solutions are always looking to give support with the most updated AWS features to our clients, so we came up with the idea to create a mirror between Docker Hub and Amazon Public ECR.

The fastest way we found was creating a simple python script to automate the pull and push of a list of docker images declared. To make sure we were on the right path, we tested migrating all our images.

Running from an EC2 instance, this took around 10 hours, 26 repositories, 188 images, and more than 39 GB of data.

We are very passionate about the open-source culture and being mindful of the people who follow us, on the link below you can find the script with instructions on how to create the mirror.

DNXLabs/tools-box

At DNX Solutions, we work to bring a better cloud and application experience for digital-native companies in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, and Service Mesh and Data Solutions (movement, transformation, lakes, warehouses and analytics).

We are always hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at github.com/DNXLabs and follow us on Linkedin or Facebook.

Mirroring Docker Hub with AWS Public ECR was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let’s talk about NodeJS Promises, I promise it won't be long

Serverless architectures are becoming very popular nowadays and NodeJS runtime is an interesting option on AWS Lambda. However, an asynchronous runtime is not always a suitable solution. So, understanding promises in Javascript is essential to determine when to use NodeJS Lambda Functions on AWS.

In this article, we are going to wall through the characteristics of javascript in chronological order. As the title says, it won’t be long so bear with me!

:)

First things first, non-blocking.

It is a huge topic and can be considered a subject for another article. However is essential to keep in mind that NodeJS is a non-blocking language and uses a single Thread to run your operations.

What do you expect before running this snippet NodeJS code? Most people would expect something like this:

1.Starting up
2.Two seconds!
3.Zero seconds!
4.Finishing up

But, remember that NodeJS is asynchronous by default so the result of this code is going to be this:

1.Starting up
4.Finishing up
3.Zero seconds!
2.Two seconds!

Please notice that everything happens at the same time, go ahead and execute this code in your terminal, start to play around. You will start to understand what happens under the hood or optionally you can stay tuned a new article about Single Thread in details is coming soon. 👍

Callbacks are like promises’ ancestors

So, we had Callbacks before that are the foundation of NodeJS, using it you have the feeling that things are happening in sequence.

A callback is a function called at the completion of a given task, this prevents any blocking, and allows other code to be run in the meantime.

To illustrate callbacks let’s use the following example creating “doWorkCallback” function. This function simply waits 2 seconds and return either an error message or an array, so you can run in your terminal and alternate the return to see both results displayed in your display.

If you are not familiar with this ES6 syntax, using arrow functions. I also created this snippet with a simpler code with the same functions:

Here we can clearly see that the “synchronous” feeling is generated by the callback that is the functions passed to our myFunction as an argument.

🤯

Let’s go to the official definition of promise

The Promise object represents the eventual completion (or failure) of an asynchronous operation, and its resulting value.

When you have an asynchronous runtime by default, fetching operations will eventually require you to know the result of an operation to determine the next events in your application or script. It is one example where using promise is a suitable approach, so let’s move forward and investigate its origins.

This need for knowing the result of an asynchronous operation is not new and has appeared since the beginning. They are designed to manage asynchronous code, like callbacks enhancement.

Here is an example:

Using the same example before, but now using promises we can see their 3 states that are pending, fulfilled and rejected. So, you can monitor and take actions in your application or script based on those states.

Don’t be shy, go ahead and run this code on your computer! :)

I said it would not be long, so I am stopping here but we could extend this subject to callback hell, promise chaining and async-await topics. But the aim of this text is to explain promise in its pure form. It is essential to know the problem that these technologies solves and consider these aspects when choosing your AWS Lambda runtime.

Stay tuned for more content!

At DNX Solutions, we work to bring a better cloud and application experience for digital-native startups in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery and Service Mesh and Data Solutions (movement, transformation, lakes, warehouses and analytics).

We are constantly hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at https://github.com/DNXLabs and follow us on our Linkedin.

Let’s talk about NodeJS Promises, I promise it won't be long was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

As every solution comes from a challenge or problem. Engineers love to solve problems and problem-solving requires creativity and knowledge of the tools you have available to you, very similar to an artist that want to express their art with different materials and tools that they have. So, here is the challenge/requirement solved:

Challenge

Database secrets rotation can be a compliance requirement or simply to enhance the security of your environments. Rotate RDS passwords in AWS is facilitated using the Secrets Manager service. However, you may have a fully automated deployment in your infrastructure as ECS blue-green deployments and want to refresh secrets also in your application when a new secret is created.

So, it is a challenge that can be tackled using AWS Lambda functions and here is how you can do it in an efficient and secure way on AWS.

Solution

Every solution starts with an efficient and solid architecture, the language chosen was Python due to its synchronous nature and vast source of libraries as for example boto3, a very popular and well-maintained set of functions to interact with AWS services. Below you can check a High-Level Diagram of the solution with its sequential steps.

1. The Secret Rotation Lambda function connects to the RDS using the current secret stored in SSM parameters store.
2. Generates a 32 characters random password including ascii_letters, digits and 4. punctuation (excluding “:/@“\’\“) as those are separators characters as required by PostgreSQL
3. If current secret successfully connects, then rotates the secret in the database
4. Test new secret before update application parameters
5. If success, then the lambda builds the database URL and RDS secret.
6. Then, finally restarts the application containers on ECS to refresh secret

Disclaimer: Keep in mind that this solution has some downtime and can be triggered during long term basis outside of business hours for example. If you cannot afford any downtime at all, you can use your application to manage the database connection pool and grab the secrets from SSM or Secrets Manager instead of using environment variables into ECS containers.

So, let’s get our hands dirty and deep drive into our Lambda function!

Let’s check some code ⌨️

For start, let’s import our dependencies, this solution imports the PyGreSQL classic interface and pgdb compliant module for PygreSQL written by Ch. Zwerschke and D’Arcy J.M. Cain respectively. Also, it imports typical libraries for string manipulation, timing and logging, for example.

And then our lambda function handler, here you can associate the steps 1 to 6 described before. Due to the synchronous process required for this solution, we can clearly see step by step how the python script will execute safely the secret rotation.

The generate_secret function has some particulates, as PostgreSQL does not accept certain punctuations in the password, be aware when generating a new secret, here is how the current function works:

As in this scenario, the ECS service leverages SSM parameters store to build the tasks, restarting the application containers is necessary.

Don’t forget to create a Cloudwatch trigger for your Lambda function and keep in mind that it is not a zero-downtime approach, chose the cron expression wisely.

🧙

The End

If you read until this point and want to implement this solution on your AWS environment, I do not want to take more of your time and let you know that you can refer to the Github repository below, feel free to contribute or fork it anytime!

wvxavier/ecs_rds_secret_rotation

At DNX Solutions we bring a better cloud experience for digital-native companies in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, Service Mesh, Data Engineering & Analytics, and Managed Services.

Check our open-source projects at https://github.com/DNXLabs and follow us on Linkedin.

RDS secrets rotation and ECS update was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since Amazon released the “t2” instance class for EC2, a problem I noticed that happens more frequently than it should is lack of an understanding of how the credit system works.

One of the most common issues is when an EC2 instance runs out of CPU credits and becomes throttled, causing an impact on users.

This article covers how to leverage cost savings of bursting services while keeping high-availability.

EC2: t2 and t3 instances

The family of t2 and t3 instances are around 30% cheaper than their equivalent in other instance type families.

To achieve such reduction, AWS introduced “burstable” CPU credits. These credits are consumed every time the CPU is used and is replenished at a fixed rate, depending on the instance type.

As an example, if you create a t2.micro instance (very popular as it’s part of the free tier), wait until your credit balance is full and then start using 100% of CPU, it will run out of credits in 2 hours and 24 minutes. After that the instance will be limited to use only 10% of the CPU.

But how to calculate that?

Each instance type has a rate of credits earned and a maximum value. As per July/2020, the values are:

For t2.micro from our example. The max credits it holds is 144.

A CPU credit provides for 100% utilization of a full CPU core for one minute.

The t2.micro has 144 credits and 1 CPU core, which means 144 minutes of 100% CPU, or 2h24m.

How about t3?

T3 is a newer generation. Overall the instances gain more credits for the same or cheaper cost, so the only reason to continue using T2 would be because of the free tier or if your region/AZ doesn’t support this type.

Unlimited

To prevent throttling when credits run out, you can enable T2/T3 Unlimited.

With this feature enabled, instead of limiting your CPU to the baseline, AWS will charge you an extra to keep your credits flowing.

Enabling this can be a lifesaver if your workload is known to not use all the credits balance very often. Otherwise, it’s cheaper to use another instance type.

Example: If your instance works on 100% CPU for a full month on a t2.large, you will pay:

• $85.26 for the instance + $72 for the extra CPU credits = $157.26

Comparing to an m4.large, which has the same specs, you would pay:

• $91.25

In summary, T2/T3 instances are only cheaper when the amount of CPU credits provided is enough for your workloads. If your workload needs to sustain a high CPU over a long period of time, using an M4 or C5 is more economical.

RDS

Similar to EC2, RDS also supports T2/T3 instances, but with a catch.

RDS does not give the option to enable/disable T2/T3 Unlimited, which as we saw before, gives insurance against long CPU spikes that might exhaust the credits.

Instead, for RDS T2 instances have this option off by default and T3 instances have it on by default.

Conclusion

T2/T3 instances are great in terms of cost-efficiency, but it comes with an operational cost.

Using this instance class requires more planning and monitoring to make sure the type of workload running there is adequate and that you will be alerted before credits run out.

As an example, one of our Terraform modules creates an alarm for low credit balance on RDS instances automatically if the instance type starts with a “db.t”:

DNXLabs/terraform-aws-db-monitoring

At DNX Solutions we bring a better cloud experience for digital-native startups in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, Service Mesh, and Data Architecture and Services.

Check our open-source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin.

Understanding Credit balances on AWS services: EC2 and RDS was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is our second update. Check out our first update here:

DNX Tech Update: April-May

As part of our routine, we invest heavily in improving our open source stack and making sure knowledge is captured in the form of code.

DNX One

Last month our main focuses were on compliance and data.

Compliance

terraform-aws-security-baseline — This module setup alerts for common controls defined on the CIS AWS Foundations Benchmark.

terraform-aws-ecs — Our ECS Cluster module now supports Fargate. Why is this included as a compliance update, you ask?

By being a managed service from AWS that is compliant with most industry regulations, Fargates reduces the burden for customers of having to manage compliance internally. More info at https://aws.amazon.com/fargate/faqs/#Security_and_Compliance.

Backups

terraform-aws-backup — A module to automatically backup data from multiple resources using Tags and AWS Backup.

Logging

We created are releasing a module that helps to ship logs to an ElasticSearch cluster: terraform-aws-kinesis-stream-es.

It works by creating a subscription filter in Cloudwatch Logs which collects and puts these logs in a Kinesis Data Firehose stream. The Kinesis stream then will process and deliver to an ElasticSearch domain.

Kinesis was added to the mixture to act as a buffer between the logs collected and ElasticSearch, so in case of a spike above the capacity of ES or if it goes down for maintenance, logs entries are not lost.

Blog

Three new blog posts this months from our engineers:

• AWS ECR — Improving container security by using Docker image scanning
• Upgrade Elasticsearch from 2.3 to 7.4
• AWS Amplify: Create Custom Resolvers Programmatically (to fetch data from AWS RDS databases)

For the full blog, check https://medium.com/dnx-labs.

Under development

one-cli

We are still working on improving the developer experience with AWS. Our CLI is one of the pillars of this strategy.

You can track the progress at https://github.com/DNXLabs/one-cli

For a snapshot of all our open-source code can be found at https://modules.dnx.one/

At DNX Solutions we bring a better cloud and application experience for digital-native startups in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, Service Mesh, and Data Architecture and Services.

We are continually hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin.

DNX Tech Update: June-July was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

It has been three months since I decided to completely reshape my career moving from an SAP senior consultant role to an AWS cloud solutions architect role. I knew this was an unconventional move, and several risks were involved in the process, but it has been a marvellous experience so far, and today I can talk with confidence this was the right choice for me.

A lot of my colleagues reacted in different ways when I announced this update, and I believe most of them still don’t get the point of transferring almost 12 years of solid experience in a particular field and restarting fresh in a completely new environment.

The idea behind this personal insight that I am sharing today is to bring enlightenment to some of the factors involved in this decision and also to advise other people that are facing a similar dilemma.

Discovering the Cloud

My first contact with Cloud was through SCP (SAP Cloud Platform) back in 2017, SCP was released in the market in 2012, but it was still a very young solution compared to the major players (AWS, GCP and Azure).

As soon SAP started to establish partnerships with the major cloud providers to deploy their PaaS on their data centres my curiosity increased and consequently I, slowly, moved my focus towards their environments, services and offers.

Finally, in between 2018 and 2019, I had the opportunity to work in a significant S/4HANA implementation deployed on AWS. Initially, I didn’t have too much exposure to the AWS cloud since I was more focused on development, data and reporting, the only advantage I could see at that point in time was the migration workload that is usually related with the SAP Basis team.

The turnover happened when this customer decided to transfer their reporting strategy from SAP proprietary solution to AWS data lake approach. Testifying this customer’s decision gave me a clear view of the potential while combining the best of both worlds, and my curiosity continuously grew over time.

As an SAP consultant participating in this migration project, I remember how confusing communications were between both teams and during the process, I realised how important it was to have an architect with experience in both ends.

At that particular moment, my unexpected journey began.

Moving to the Cloud

Changing to a new career path is not an easy task, especially if you are in your comfort zone for more than a decade. So, at that point, I was questioning myself, how should I start this journey?

The first thing I did was to enter in contact with my network. I scheduled several catch-ups around the city with some of the most talented colleagues in my connections list.

Based on their inputs, I noticed it was necessary to focus on two main items:

• Select a cloud provider only, don’t try to cover all of them at the same time. My choice was AWS because of market share in Australia and the list of services available.
• Study and prepare yourself for the associate level certifications, having a cloud certification is an excellent entry-point for the market and also provides the knowledge base to start as a consultant in the area.

I started this certification path focused on AWS, and in three months, I got precisely three AWS certifications:

1. AWS Certified Cloud Practitioner
2. AWS Certified Solutions Architect — Associate
3. AWS Certified Developer — Associate

With the exposure of the certifications in my online profile, I received an invite to participate in an internal program of a well known AWS consultancy in Sydney. They offered me a mentorship/training program, I was learning even more during this process, and in parallel, I was able to demonstrate my value as a consultant during each phase of the training.

At the end of this program, I received an invitation to join the company, and the rest is history, three months of exciting work with incredible customers, lots of learnings and many plans for the future.

Adapting my SAP experience to the Cloud

So, what about all the experience I had in the SAP space? Is it all gone?

I’m not programming in ABAP or using SAP-related technologies anymore. Still, several skills that I constructed during this journey were adapted quickly to the Cloud Architect role and provided me advantages compared to other consultants, for example:

• Soft-skills
• Customer-facing skills
• Enterprise architecture experience
• Business processes and communication

Also, the experience as Software engineer and as Data analyst / BI consultant that I acquired while working with SAP contributed a lot to the transition process providing me enough know-how to design solutions for applications or data strategies in the Cloud.

Finally, notice that SAP creates their own technologies in house, but they are usually looking for external references in the market, consequently, since I had a vast background in multiple technologies I was able to grasp new content in a faster way.

The future of SAP and AWS

From my point of view, there are three major areas to explore when talking about SAP and AWS:

1. Migration workload: This is one of the most consolidated areas when talking about SAP and AWS. A lot of customers are already running their workloads in AWS and taking advantage of the cloud proposition. A few companies are specialising in this kind of activity, and this market continues growing over time.
2. Application extensibility: With SAP releasing their software in a SaaS model, the application extensibility was reshaped, and side-by-side extensibility became an important topic. Different initiatives are running via SAP Cloud Platform with Cloud-foundry for S/4HANA and also with Kyma and Kubernetes for C/4HANA, but in case the SAP PaaS is not the preferable option of the customer, the application extensibility can be done directly in the cloud provider of choice, using their own offerings of PaaS, with custom integrations between the apps and core solutions (e.g. S/4HANA or C/4HANA).
3. Data and reporting strategy: Actually, this is one of those areas that I share most of my interest and excitement. Data and reporting strategies outside of the SAP in-house software offerings are becoming more and more common. Since there aren’t any impacts with read-only data in the SAP indirect licensing model, a lot of customers are exploring alternative solutions moving their data and reporting strategies to data lakes and analysing and visualising their insights through different cloud services.

Finally, what about my future?

At the moment, I don’t see a return to the SAP world in the short or long term. However, this doesn’t mean that I abandoned the area for good.

As I mentioned before, I believe there is a lot of potential to combine both worlds and create incredible solutions for the customers, but at this stage, I am fully committed with my cloud journey, and I have a lot of plans in progress.

I hope this blog gave you some ideas of how to start if you are facing a similar transition and I also wish I was able to bring you some kind of motivation to leave your comfort zone and try something new.

If you are interested in some of my previous SAP content, I advise you to have a look in my SAP People profile. I have some exciting blogs focused on different technologies that will give you an overview of the kind of work I was doing before this new role.

SAP People: https://people.sap.com/felipedemello.rodrigues

If you are interested in Data and AWS or more content combining SAP and AWS, stay tuned because I have more insights to share along the way. :)

At DNX Solutions we bring a better cloud and application experience for digital-native startups in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery, Service Mesh, and Data Architecture and Services.

We are continually hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin.

An unexpected journey from SAP consultant to AWS cloud architect was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

AWS Amplify is an amazing open-source project from AWS that helps you build secure, scalable mobile and web applications.

In this post, I will help you to create custom resolvers programmatically in AWS Amplify without relying on the AWS Console and keeping everything as code that in the past, you either needed to use the AWS AppSync console or edit AWS CloudFormation templates to implement this logic.

Why Custom Resolvers?

If you’re not familiar with AWS AppSync or GraphQL, a “resolver” is essentially a function that’s responsible for fetching data from a location to fulfil a request. For instance, a resolver might query from a database to get a stored record, or it could just compute a value directly. Resolvers are attached to fields on a “type” in a GraphQL schema. These are executed at runtime, depending on the request that comes from a client.
When you’re developing GraphQL APIs, you often have to customize the resolvers to perform custom logic and implement things like data manipulation, authorisation, or fine-grained access control.

Reference: https://aws.amazon.com/blogs/mobile/amplify-adds-support-for-multiple-environments-custom-resolvers-larger-data-models-and-iam-roles-including-mfa/

In our example here, We want to fetch the data from an AWS RDS PostgreSQL database instead of a Dynamo DB table.

Before you start!

You must have your Amplify project initialised (amplify init), and create a GraphQL API (amplify API add), I won’t focus in this initialising part. If you are here, you should have done this as well. Now We Can start modifying the default code created by Amplify.

Starting to code

It is our example schema:

schema.graphql example

We should create our lambda that will be our Custom Resolver:
Run the command: amplify add function
We will call it lambdaResolver, as the output bellow.

Now we can start editing our Lambda Resolver:

index.js for lambdaResolver

This Custom Resolver will be responsible for fetching the data that you want, and you can get the data from any source. You need to return the exact object that your schema is expecting, for example, to mock some data you can do the following:

index.js for lambdaResolver mocking data

Make sure that you lambda has the right permission/access and/or It’s in the correct VPC and has the necessary execution role. In our case, We are fetching the data from an AWS RDS PostgreSQL located in a secure subnet. The lambda resolver should be able to reach the RDS.

VLT Templates

For each method, you should create a VLT template inside the resolvers folder for the response and the request.

Query.getTodo.req.vtl

Query.getTodo.res.vtl

Custom Resources (DataSource and Resolver)

We need to create custom resources for our lambda resolver, and We will add a DataSource, Resolver and a Role.
The DataSource will be our lambda resolver. Resolver will map our VTL templates and the method. The Role will permit the DataSource to execute the lambda.

CustomResources.json responsible for creating the DataSource, Resolver and Lambda Role

These resources should be included inside the CustomResources.json file inside the stacks folder.

After all the change We should push/sync our project. You can push your changes to your code repository if you are using the Amplify pipelines, or call the CLI command amplify push to run locally.

It will create all resources that We included in CustomResources.json file and will link each other. We can open the AppSync console (amplify API console), and We can see all the code that We added are there.

Now We can test our Query if it’s working and calling our lambda.

Conclusion

You can programmatically create all your custom resolvers, without relying on the AWS Console. Also, you can connect to any data source you want, making it a powerful tool, even connect more data sources using Pipeline Resolvers.

We work at DNX Solutions and help to bring a better cloud and application experience for digital-native startups in Australia.

Our current focus areas are AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery and Service Mesh.

We are continually hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.

Check our open-source projects at https://github.com/DNXLabs and follow us on our Twitter or Linkedin.

References:
https://aws.amazon.com/amplify/
https://aws.amazon.com/blogs/mobile/amplify-adds-support-for-multiple-environments-custom-resolvers-larger-data-models-and-iam-roles-including-mfa/
https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-lambda-resolvers.html#create-a-lam-function
https://docs.aws.amazon.com/appsync/latest/devguide/test-debug-resolvers.html
https://docs.amplify.aws/cli/graphql-transformer/resolvers#add-a-custom-resolver-that-targets-a-dynamodb-table-from-model

AWS Amplify: Create Custom Resolvers Programmatically (to fetch data from AWS RDS databases) was originally published in DNX Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.