Iinspired by Atalyk Akash’s discussion, here are several common DP patterns rewritten in Python.

1. Minimum (Maximum) Path to Reach a Target

Explanation:
This pattern involves finding an optimal (minimum or maximum) cost, path, or sum to reach a target state. The idea is to choose the best among all possible previous states and then add the cost (or benefit) of the current state. This approach can be applied to problems such as climbing stairs, grid paths, or coin change.

Python Examples:

Top‐Down (Recursive with Memoization):

def min_path(target, ways, cost, memo=None):
    if memo is None:
        memo = {}
    if target == 0:
        return 0
    if target in memo:
        return memo[target]
    
    best = float('inf')
    for w in ways:
        if target - w >= 0:
            best = min(best, min_path(target - w, ways, cost, memo) + cost[target])
    memo[target] = best
    return best

# Example usage:
target = 5
cost = [0, 1, 2, 3, 4, 5]  # sample cost array
print(min_path(target, ways=[1, 2], cost=cost))

Bottom‐Up (Iterative):

def min_path_bottom_up(target, ways, cost):
    dp = [float('inf')] * (target + 1)
    dp[0] = 0  # base case
    
    for i in range(1, target + 1):
        for w in ways:
            if w <= i:
                dp[i] = min(dp[i], dp[i - w] + cost[i])
    return dp[target]

print(min_path_bottom_up(target, ways=[1, 2], cost=cost))

Practice Problems:

• [746] Min Cost Climbing Stairs (Easy):
  Practice accumulating minimal cost while climbing steps.
• [64] Minimum Path Sum (Medium):
  Find the minimum sum path through a 2D grid.
• [322] Coin Change (Medium):
  Determine the minimum number of coins needed to form a given amount.
• [120] Triangle (Medium):
  Compute the minimum path sum from top to bottom in a triangle structure.
• [1049] Last Stone Weight II (Medium):
  Partition stones into two groups to minimize the difference between their sums.
• [174] Dungeon Game (Hard):
  Calculate the minimum initial health required to navigate a grid with hazards.
• [871] Minimum Number of Refueling Stops (Hard):
  Decide optimal refueling points along a journey to reach your destination.

2. Distinct Ways to Reach a Target

Explanation:
This pattern is used when you need to count the number of unique ways to reach a target state. Instead of taking a minimum or maximum, you sum up the ways to reach the current state from all valid previous states. It’s common in problems like climbing stairs or grid paths.

Python Examples:

Top‐Down (Recursive with Memoization):

def count_ways(target, ways, memo=None):
    if memo is None:
        memo = {}
    if target == 0:
        return 1
    if target in memo:
        return memo[target]
    
    total = 0
    for w in ways:
        if target - w >= 0:
            total += count_ways(target - w, ways, memo)
    memo[target] = total
    return total

# Example usage:
print(count_ways(5, ways=[1, 2]))

Bottom‐Up (Iterative):

def count_ways_bottom_up(target, ways):
    dp = [0] * (target + 1)
    dp[0] = 1  # one way to reach 0
    for i in range(1, target + 1):
        for w in ways:
            if w <= i:
                dp[i] += dp[i - w]
    return dp[target]

print(count_ways_bottom_up(5, ways=[1, 2]))

Practice Problems:

• [70] Climbing Stairs (Easy):
  Count the number of distinct ways to climb stairs taking one or two steps at a time.
• [62] Unique Paths (Medium):
  Count the number of unique paths from the top-left to the bottom-right of a grid.
• [1155] Number of Dice Rolls With Target Sum (Medium):
  Calculate the number of ways to achieve a target sum with dice rolls.
• [494] Target Sum (Medium):
  Decide how to add or subtract numbers to reach a target sum.
• [377] Combination Sum IV (Medium):
  Count combinations that add up to a target using given numbers.
• [1269] Number of Ways to Stay in the Same Place After Some Steps (Hard):
  Solve a challenging problem involving constrained movement.
• [1220] Count Vowels Permutation (Hard):
  Count valid vowel permutations under specified constraints.

3. Merging Intervals

Explanation:
This pattern involves breaking a problem into subintervals and merging the solutions from the left and right segments optimally. It’s often used when the decision at one point depends on both preceding and succeeding elements. Problems like “Burst Balloons” are classic examples where merging intervals helps optimize the solution.

Python Examples:

Top‐Down (Recursive):

def merge_intervals(nums, i, j, memo):
    if i > j:
        return 0
    if (i, j) in memo:
        return memo[(i, j)]
    
    best = 0
    # Try every possible splitting point.
    for k in range(i, j + 1):
        left = merge_intervals(nums, i, k - 1, memo)
        right = merge_intervals(nums, k + 1, j, memo)
        best = max(best, left + nums[k] + right)
    
    memo[(i, j)] = best
    return best

# Example usage:
nums = [3, 1, 5, 8]
print(merge_intervals(nums, 0, len(nums) - 1, {}))

Bottom‐Up (Iterative):

def merge_intervals_bottom_up(nums):
    n = len(nums)
    dp = [[0] * n for _ in range(n)]
    
    # l is the interval length.
    for l in range(1, n):
        for i in range(n - l):
            j = i + l
            for k in range(i, j + 1):
                left = dp[i][k - 1] if k - 1 >= i else 0
                right = dp[k + 1][j] if k + 1 <= j else 0
                dp[i][j] = max(dp[i][j], left + nums[k] + right)
    return dp[0][n - 1]

print(merge_intervals_bottom_up(nums))

Practice Problems:

• [1130] Minimum Cost Tree From Leaf Values (Medium):
  Merge numbers in an array to minimize the total cost.
• [1039] Minimum Score Triangulation of Polygon (Medium):
  Determine the optimal way to triangulate a polygon for minimum cost.
• [546] Remove Boxes (Medium):
  Remove boxes in an optimal order to maximize points.
• [1000] Minimum Cost to Merge Stones (Medium):
  Find the best strategy to merge stone piles with minimal cost.
• [312] Burst Balloons (Hard):
  Burst balloons in an order that maximizes coins earned.
• [375] Guess Number Higher or Lower II (Medium):
  Optimize guessing strategy to minimize the worst-case cost.

4. DP on Strings

Explanation:
Many DP problems involve strings — either one string or two — where you need to compute results such as the longest common subsequence, edit distance, or palindromic substrings. Typically, you set up a 2D DP table where indices correspond to positions in the string(s) and update the table based on character matches or decisions.

Python Examples:

Longest Common Subsequence (LCS):

def longestCommonSubsequence(text1, text2):
    n, m = len(text1), len(text2)
    dp = [[0] * (m + 1) for _ in range(n + 1)]
    
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if text1[i - 1] == text2[j - 1]:
                dp[i][j] = dp[i - 1][j - 1] + 1
            else:
                dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])
    return dp[n][m]

print(longestCommonSubsequence("abcde", "ace"))

Longest Palindromic Subsequence:

def longestPalindromeSubseq(s):
    n = len(s)
    dp = [[0] * n for _ in range(n)]
    
    for i in range(n):
        dp[i][i] = 1
        
    for cl in range(2, n + 1):
        for i in range(n - cl + 1):
            j = i + cl - 1
            if s[i] == s[j]:
                dp[i][j] = 2 if cl == 2 else dp[i + 1][j - 1] + 2
            else:
                dp[i][j] = max(dp[i + 1][j], dp[i][j - 1])
    return dp[0][n - 1]

print(longestPalindromeSubseq("bbbab"))

Practice Problems:

• [1143] Longest Common Subsequence (Medium):
  Find the longest subsequence common to two strings.
• [647] Palindromic Substrings (Medium):
  Count all palindromic substrings within a string.
• [516] Longest Palindromic Subsequence (Medium):
  Determine the longest palindromic subsequence in a string.
• [72] Edit Distance (Hard):
  Compute the minimum number of operations to convert one string into another.
• [115] Distinct Subsequences (Hard):
  Count how many times a subsequence appears in a string.
• [712] Minimum ASCII Delete Sum for Two Strings (Medium):
  Remove characters to make two strings equal with minimum ASCII cost.
• [5] Longest Palindromic Substring (Medium):
  Identify the longest contiguous palindromic substring in a string.

5. Decision Making

Explanation:
This pattern centers on making choices at each step — whether to include or exclude the current element — to optimize the overall result. The decision often involves comparing outcomes between taking the current element (with its associated benefit) or skipping it (preserving the previous optimal state). This is common in problems like house robbing or stock trading.

Python Examples:

House Robber:

def rob(nums):
    if not nums:
        return 0
    n = len(nums)
    if n == 1:
        return nums[0]
        
    dp = [0] * (n + 1)
    dp[0] = 0
    dp[1] = nums[0]
    
    for i in range(2, n + 1):
        dp[i] = max(dp[i - 1], dp[i - 2] + nums[i - 1])
    return dp[n]

print(rob([2, 7, 9, 3, 1]))

Stock Trading with Transaction Fee:

def maxProfit(prices, fee):
    cash, hold = 0, -prices[0]
    for price in prices[1:]:
        cash = max(cash, hold + price - fee)  # Sell the stock if profitable
        hold = max(hold, cash - price)          # Buy new stock if it improves the state
    return cash

print(maxProfit([1, 3, 2, 8, 4, 9], fee=2))

Practice Problems:

• [198] House Robber (Easy):
  Choose which houses to rob to maximize profit without alerting the police.
• [121] Best Time to Buy and Sell Stock (Easy):
  Find the optimal day to buy and sell stocks to maximize profit.
• [714] Best Time to Buy and Sell Stock with Transaction Fee (Medium):
  Incorporate transaction fees into your trading strategy.
• [309] Best Time to Buy and Sell Stock with Cooldown (Medium):
  Factor in a cooldown period between transactions for optimal trading.
• [123] Best Time to Buy and Sell Stock III (Hard):
  Optimize up to two transactions to maximize profit.
• [188] Best Time to Buy and Sell Stock IV (Hard):
  Tackle multiple transactions under constrained conditions.

Final Thoughts:
Each section above outlines the core idea behind a common DP pattern, provides sample Python code to illustrate how to approach these problems, and lists targeted practice problems to help you deepen your understanding. Working through these examples and challenges will build a strong foundation in dynamic programming and prepare you for a variety of interview and competitive programming scenarios.

Happy coding and good luck mastering DP!

Dynamic Programming Patterns was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

Installing Jenkins

1. Open a web browser and navigate to the Jenkins download page: https://jenkins.io/download/
2. Scroll down to the “Long-term support release” section and click on the latest version of Jenkins.
3. On the next page, select the appropriate installer for your operating system.
4. Once the download is complete, run the installer and follow the on-screen instructions.
5. During the installation process, you will be prompted to choose a username and password for the Jenkins administrator account.
6. After the installation, open a web browser and navigate to http://localhost:8080/ (or the custom port you may have chosen during installation).
7. You will be prompted to enter the administrator username and password you created during installation.
8. Once you have successfully logged in, you can begin configuring Jenkins for your project needs.

Install BlueOcean

To install Blue Ocean in Jenkins, you will need to follow these steps:

1. Open a web browser and navigate to your Jenkins instance.
2. Click on the “Manage Jenkins” link in the left-hand menu.
3. Click on the “Manage Plugins” link.
4. Click on the “Available” tab.
5. Search for “Blue Ocean” in the search bar.
6. Check the box next to “Blue Ocean” and any other desired plugins.
7. Click on the “Download now and install after restart” button.
8. Restart Jenkins to complete the installation of Blue Ocean.

Once the installation is complete, you should see a new “Blue Ocean” link in the left-hand menu of your Jenkins instance.

Creating Jenkins job

Installing SonarQube

1. Open a web browser and navigate to the SonarQube download page: https://www.sonarqube.org/downloads/
2. Scroll down to the “LTS” section and click on the latest version of SonarQube.
3. On the next page, select the appropriate installer for your operating system.
4. Once the download is complete, run the installer and follow the on-screen instructions.
5. During the installation process, you will be prompted to choose a username and password for the SonarQube administrator account.
6. After the installation, open a web browser and navigate to http://localhost:9000/ (or the custom port you may have chosen during installation).
7. You will be prompted to enter the administrator username and password you created during installation.
8. Once you have successfully logged in, you can begin configuring SonarQube for your project needs.

Configure SonarQube in Jenkins

To install the SonarQube Scanner plugin in Jenkins, you can follow these steps:

1. Open your Jenkins dashboard in a web browser and navigate to “Manage Jenkins” on the left-hand side.
2. Click on “Manage Plugins” to open the plugin manager.
3. In the “Available” tab, search for “SonarQube Scanner” using the search box.
4. Check the checkbox next to “SonarQube Scanner” in the list of available plugins.
5. Click on the “Install without restart” button to start the installation process.
6. On the next page, make sure to select the option “Download now and install after restart” if you prefer to install any additional required plugins.
7. Once the installation is complete, you will see a confirmation message.
8. Restart Jenkins by navigating to “Manage Jenkins” and clicking on the “Restart Jenkins” link if the installation requires a restart.

Now you can use the SonarQube Scanner plugin in your Jenkins pipeline or projects to analyze code quality with SonarQube.

Create a token for Jenkins in sonarqube

1. Go to ‘My Account’ and under ‘Security’ give a name and type as “Global Analysis Token” set your expiry days and hit Generate.
2. Copy the token and use it in Jenkins.

To configure the Environment of SonarQube with Jenkins, you will need to follow these steps:

1. Get the token from the sonarqube and add it to the system credentials shown above.
2. Configure the SonarQube server in Jenkins by providing the server URL and authentication token.
3. Give sonarserver as the name and click on add under Server authentication token and add the token.
4. In your Jenkins pipeline script, add the withSonarQubeEnv step to specify the SonarQube server environment variable.
5. Add the sonar:sonar command to your mvn command in the Jenkins pipeline script to run the SonarQube analysis.
6. Run the Jenkins pipeline script to build your project and run the SonarQube analysis.

Create a Jenkinsfile in the working repository

Add the following code to the Jenkinsfile

pipeline {
    agent any
    environment{
        SONARSERVER = 'sonarserver'
    }
    stages {
        stage('SonarQube analysis') {
            steps {
                withSonarQubeEnv("${SONARSERVER}") {
                    sh 'mvn clean package sonar:sonar'
                    }
            }
        }
        stage('Build') {
            steps {
                sh 'mvn -B -DskipTests clean package'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/*.xml'
                }
            }
        }
        stage('Transfer to Artifactory') {
            steps {
                script {
                    sh 'mv ~/.jenkins/workspace/fp-java-maven/target/my-app-1.0-SNAPSHOT.jar ~/.jenkins/workspace/fp-java-maven/target/root.war'
                    sh 'jfrog rt u ~/.jenkins/workspace/fp-java-maven/target/root.war fp-java-maven/root.war'
                }
            }
        }
        stage('Deploy to tomcat server') {
            steps {
                script {
                    sh 'mv ~/.jenkins/workspace/fp-java-maven/target/root.war /usr/local/Cellar/tomcat/10.1.9/libexec/webapps/root.war'
                }
            }
        }
    }
}

This is a Jenkins pipeline script written in Groovy. It has three stages:

1. Build: This stage compiles the code and packages it into an executable JAR file.
2. Test: This stage runs the unit tests and generates a report using JUnit.
3. build && SonarQube analysis: This stage builds the code and runs a SonarQube analysis to evaluate code quality and security issues.

PS: It also requires a SonarQube server to be running, which is specified as an environment variable.

• The pipeline block defines the overall structure of the pipeline, with the agent specifying the agent that will execute the pipeline, the environment defining environment variables, and stages defining the individual stages of the pipeline.
• The first stage is SonarQube analysis, which runs a SonarQube analysis on the project. The withSonarQubeEnv step sets up the environment for running the analysis, including specifying the SonarQube server to use. The sh step then runs the mvn clean package sonar:sonar command to build the project and run the analysis.
• The second stage is Build, which simply builds the project without running any analysis. The sh step runs the mvn -B -DskipTests clean package command to build the project.
• The third stage is Test, which runs the project's tests. The sh step runs the mvn test command to execute the tests. The post block specifies a junit step to publish the test results in JUnit format.
• The fourth stage is Transfer to Artifactory, which transfers the built artifact to an Artifactory repository. The sh step moves the built JAR file to a WAR file, and then the jfrog rt u command uploads the WAR file to the Artifactory repository.
• The fifth stage is Deploy to tomcat server, which deploys the built WAR file to a Tomcat server. The sh step moves the WAR file to the Tomcat webapps directory.

Make sure to replace the sonarserver with the name of your SonarQube server.

Creating a JFrog Account

1. Open a web browser and navigate to the JFrog sign-up page: https://jfrog.com/signup/
2. Fill out the form with your personal information, including your name, email address, and password.
3. Choose a subscription plan that best fits your needs.
4. Review and accept the terms and conditions of the JFrog platform.
5. Click the “Create Account” button to complete the sign-up process.
6. After creating your account, you can log in to the JFrog platform and begin using its features for your project needs.

Create a token for Jenkins

Switch to Classic UI

1. Log in to your JFrog instance and navigate to the JFrog platform’s administration section.
2. In the administration section, locate and click on the “Security” or “Users” tab.
3. Find the user account associated with Jenkins or the user account you want to generate the token for.
4. Click on the user account to open its details or edit the user account.
5. Look for an option like “Access Tokens,” “API Keys,” or “Authentication Tokens” (the exact wording may vary depending on the JFrog version you are using).
6. Click on the option to create a new token.
7. Provide a name or description for the token, such as “Jenkins Token” or any other meaningful name.
8. Select the appropriate permissions or scopes for the token, depending on the actions you want Jenkins to perform in JFrog. Ensure the token has the necessary permissions for uploading artifacts, downloading dependencies, or any other actions required by your Jenkins pipeline.
9. Save or generate the token. Note that some systems may display the token only once, so make sure to copy and securely store it.

Configure Jfrog with Jenkins

1. In your Jenkins environment, navigate to the Jenkins dashboard and locate the project or pipeline configuration that requires access to JFrog.
2. Within the Jenkins configuration, find the appropriate location to add the JFrog token. This can be in the Jenkins global configuration or specific job/pipeline configurations, depending on your setup.
3. Add the JFrog token as a credential or environment variable in Jenkins. This will enable Jenkins to authenticate with JFrog using the token when performing relevant actions.

Installing Tomcat

1. Open a web browser and navigate to the Apache Tomcat download page: https://tomcat.apache.org/download-80.cgi
2. Scroll down to the “Binary Distributions” section and click on the latest version of Tomcat.
3. On the next page, select the appropriate installer for your operating system.
4. Once the download is complete, run the installer and follow the on-screen instructions.
5. During the installation process, you will be prompted to choose a username and password for the Tomcat administrator account.
6. After the installation, open a web browser and navigate to http://localhost:8000/root (or the custom port you may have chosen during installation).
7. You will be prompted to enter the administrator username and password you created during installation.
8. Once you have successfully logged in, you can begin deploying your application to Tomcat.

Run the script from BlueOcean

Open the project in BlueOcean and hit Run

You will now see the scripts running

To view the SonarQube report:

1. Log in to your SonarQube server.
2. Click on the project you want to view the report for.
3. In the left-hand menu, click on “Measures”.
4. You will see a list of measures for your project, including code coverage, code complexity, and code duplication.
5. Click on any of the measures to view a more detailed report.
6. You can also view the code quality “gates” for your project by clicking on “Quality Gates” in the left-hand menu.
7. If your project fails any of the quality gates, you will need to address the issues before the code can be considered “production-ready”.

View the Status of the Jenkins build

Uploaded build file in Jfrog

Result

www.linkedin.com/in/alvisf/

Complete Maven, Jenkins, SonarQube, Jfrog & Tomcat set-up was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

START-Define Cloud Computing, its Essential Characteristics, History, and Emerging Trends

In Define Cloud Computing, its Essential Characteristics, History, and Emerging Trends, you should have learned that:

• In 2006, Amazon launched what would become AWS, with IBM, Microsoft, and Google launching services in the following years.
• Cloud computing is a technology that has evolved over time.
• Cloud computing enables convenient access to a shared network of computer resources.
• On-demand self service is a simple interface with no human interaction.
• Multiple virtual machines can run on a single computer, improving resource utilization and company returns.
• Cloud computing changes quickly and trends should be monitored

Business Case Studies for Cloud Computing

In American Airlines Case Study-Transformation, you should have learned that:

• Some companies motivated to adopt the cloud are looking to transform their operations to better serve customers.
• The pay-as-you-go model helps companies save money.
• Switching to the cloud results in faster development and release of new applications.
• Migrating to the cloud can help businesses respond faster to customer needs.

In Port of Rotterdam Case Study-Modernization, you should have learned that:

• Companies who switch to the cloud experience improved efficiency, decreased development time, and decreased expenses.
• Companies who transform using cloud technologies find they are leading change and innovation in their industry.
• Operation efficiency improves by changing to a cloud environment.
• Companies have easier access to data and an ability to more efficiently process large amounts of data in the cloud.

Service and Deployment Models of Cloud Computing

In The Cloud Service Models, you should have learned that:

• Cloud service providers own, manage, and maintain IT assets.
• Organizations save money by accessing and scaling IT resources through a cloud service provider.
• IaaS, PaaS, and SaaS are the most common types of cloud services offered.
• More than one cloud service model can be used at the same time.

In The Cloud Deployment Models, you should have learned that:

• Ongoing operating expenses are more predictable when using a public cloud model.
• Using a combination of cloud environments at the same time can be more cost effective and efficient for businesses.
• Individual security needs can be tailored by using a virtual private cloud.
• A private cloud provides exclusive access to the infrastructure with greater control over resources.

Cloud Architecture Components and Relevance

In IBM Cloud Platform, you should have learned that:

• Many components within the cloud platform work together to provide a consistent, dependable cloud experience for users.
• With the cloud platform, it is possible to provide higher levels of compliance, security, and management.
• The three core services the cloud provides are containers, storage, and servers.
• Data centers around the world deploy cloud platform services.

In IBM Cloud Infrastructure Architecture, you should have learned that:

• Cloud architecture models provide a secure foundation for various workloads.
• Cloud infrastructure provides a secure environment for applications.
• Data is isolated in a cloud infrastructure.
• The cloud infrastructure architecure uses open source technologies.

In Virtual Private Cloud Architecture, you should have learned that:

• Private cloud architecture provides greater control and security along with the benefits of a public cloud.
• Customers can connect securely to on premises or remote infrastructure.
• A network can be extended by creating a dedicated connection between the cloud and on premises resources.
• Responsive cloud applications are possible with a virtual private cloud architecture.

Cloud Compliance and Governance Practices

In IBM Compliance Programs, you should have learned:

• IBM Cloud provides programs and certifications that help organizations meet global guidelines.
• The CIS IBM Cloud Foundations Benchmark helps organizations securely adopt IBM Cloud services.
• The cloud features multiple, overlapping tiers of protection to stop cybersecurity threats.
• IBM Cloud for Government data centers adhere to global, industry, and regional compliance programs.

In IBM Cloud Hosting Options, you should have learned:

• Government agencies can trust that the IBM Cloud environment housing their applications provides flexibility and enhanced security features.
• IBM data centers are available across multiple regions and countries around the world.
• The same set of regulatory services are available in each IBM Cloud availability zone within six regions around the world.
• Those managing multiple clouds have full visibility and control across the cloud and data center infrastructure.

Cloud Catalog Services

In IBM Cloud Catalog and IBM Cloud Docs Overview, you learned that:

• The IBM Cloud Catalog is where services created by IBM and third parties are located.
• You can filter services based on a variety of parameters including provider, pricing plan, and category.
• Services can be added to your IBM Cloud account using their “Create” tab in the catalog.
• The API Docs and general documentation can be accessed using the appropriate links in the IBM Cloud Catalog.
• The left-hand menu in documentation is used to navigate through major topics.
• The right-hand menu in documentation is used to navigate the topics and sub-topics included in the currently open documentation.
• Breadcrumbs at the top of the screen allow you to quickly go back to a previous topic.

Locating Services to Deliver Specific Functions

In IBM Cloud Catalog Services, you learned that:

• You can find additional information about each service in the IBM Cloud Catalog using the “About” tab.
• Code Engine can be used to build container images.
• IBM Cloud Object Storage is a way to store data in units called buckets for easy access.
• Db2 is a relational database with enterprise-grade performance.
• Cloudant is a NoSQL database that uses JSON files to store and index documents.
• Watson Assistant can be used to add chat functionality to an application.
• Watson Discovery is used to gain a deeper understanding of data and implement natural language document search capabilities.
• Watson Studio allows you to customize machine learning models.
• Certificate Manager provides a centralized location to store and manage your security certificates.
• Hyper Protect Crypto Service provides hardware solutions to manage your crypto keys.

Delivering Cloud Services to DevSecOps Teams

In DevOps vs DevSecOps, you learned that:

• DevOps was designed to create an integrated working environment between developers and IT operations.
• DevSecOps added security into the development cycle.
• DevOps was intended to decrease development time by adding automation to the process.
• Instead of releasing a couple updates per year, DevOps is designed to continually release smaller updates as needed.

In Continuous Delivery, you learned that:

• Toolchains are sets of tools that are used to automate the process of developing and deploying code.
• Pipelines are used to build, test, and deploy code with minimal human intervention.
• An integrated web-based environment allows developers to work from anywhere.

Core Groups of Available Database Integration and Analytical Services

In Databases on IBM Cloud, you learned that:

• IBM Cloud Databases for PostgreSQL are customizable, open-source object-relational databases.
• IBM Cloud Databases for EnterpriseDB optimize the built-in features of PostgreSQL while adding compatibility with Oracle.
• IBM Db2 on Cloud is based on the enterprise-class IBM Db2 database engine and provides a fully managed solution.
• IBM Cloudant is a document database that uses JSON.
• IBM Cloud Databases for MongoDB is a JSON-based document store which includes rich query functionality.
• IBM Cloud Databases for DataStax is a NoSQL database built on Apache Cassandra which is best for high-availability and workload flexibility.
• IBM Cloud Databases for Elastisearch are based on JSON document databases and allow full-text search.
• IBM Cloud Databases for Redis are designed for in-memory functionality making them very fast.
• IBM Cloud HyperProtect can be used with PostgreSQL and MongoDB for fully managed, highly secure applications.

In Analytics of IBM Cloud, you learned that:

• There are three types of analytics: descriptive, diagnostic, and prescriptive.
• IBM Cloud Analytics use two open-source technologies: Apache Hadoop and Apache Spark.
• IBM Analytics Engine builds on Apache Hadoop and Apache Spark but separates compute and storage functionality.
• IBM Information Server on Cloud can analyze a wide variety of data and is easily scalable.
• IBM Streaming Analytics allows you to analyze fast moving, real-time data.

In Integration on IBM Cloud, you learned that:

• API Connect is used to create and manage APIs to your applications.
• App Connect is used to connect various applications to each other.
• Event Streams are built on Apache Kafka and are used as high throughput message buses.
• MQ provides enterprise-grade messaging capabilities between applications.

IBM Cloud Documentation

In IBM Cloud Documentation, you should have learned:

• Where to find product-specific guides and tutorials from across all of IBM docs, including site tours, learning more about the console, developer tools, resources for deploying, and API & SDK references.
• IBM’s reference architectures are a source to help architect solutions that enable ways to meet business objectives using leading edge hybrid cloud and AI technologies.
• The IBM developer site contains a collection of learning paths for step-by-step technical training, tutorials, articles, videos, and more.
• Code patterns are a collection of common use cases with ready to use code.

Building IBM Cloud Solutions that Adhere to IBM Security Compliance Guidelines

In IBM Cloud Security Strategy Overview, you learned that:

• There are three types of encryption: data at rest, data in motion, and data in use.
• IBM Cloud Identity and Access Management (IAM) is used to identify users and grant them access to specific parts of an application or network.
• There are two common types of IAM used in IBM Cloud: Attribute-Based Access Control (ABAC) and Role-Based Access Control (RBAC).
• There are three network encryption options: Secure Socket Layer (SSL), Transport Layer Security (TLS), and Hypertext Transfer Protocol Secure (HTTPS).
• IBM Certificate Manager is used to manage SSL and TLS certificates.
• Data in IBM Object Storage is divided up and distributed across multiple data centers.
• Once data in IBM Object Storage is deleted, it is impossible to retrieve it.

Compute Options

In IBM Cloud Bare Metal Offerings, you should have learned:

• Bare Metal servers are servers that are physical and dedicated to a single tenant.
• Some of the benefits of Bare Metal servers are that they are isolated physically, powerful, consistent, and can scale seamlessly.
• The two Bare Metal server options are fast provisioning servers and custom-based servers.

In IBM Cloud Virtual Server for Classic, you should have learned:

• Classic Virtual Servers are easily deployed and can be created on shared or dedicated infrastructure.
• Classic Virtual Servers are customizable, can be provisioned quickly, are scalable, and integrated seamlessly.
• Some of the deployment options are public virtual servers, transient virtual servers, and dedicated virtual servers.
• Some of the key differences between bare metal servers and virtual servers are that bare metal servers can offer better performance and security for certain workloads and are more ideal for heavy workloads. However, virtual servers provision more quickly and offer a more flexible and scalable environment than bare metal.

In IBM Cloud Virtual Private Cloud, you should have learned:

• Virtual Private Cloud (VPC) allows the creation of a private cloud computing environment on a shared public cloud infrastructure. VPC saves on cost and focuses on convenience for clients.
• VPC offers virtual server instances (VSIs) with a wide range of vCPU and memory options.
• Under VPC, block storage can be provisioned and attached to virtual server instances. Block storage either has defined IOPS tiers or you can provision storage with custom IOPS.

In IBM Cloud for VMware Solutions Offerings, you should have learned:

• Some benefits of VMware Solutions are being able to have the full use of VMware vCenter Server (VCS), multiple storage options, private and dedicated environment, and the use of VMware vCenter Server with Hybridity Bundle.
• The key features of VMware Solutions Shared, VMware Solutions Dedicated, VMware Solutions Regulated Workloads, and VMware Solutions Dedicated Security and Compliance Readiness Bundle.

In IBM Cloud Power Systems Virtual Server, you should have learned:

• Power Systems Virtual Servers can be quickly created and deployed through the IBM Cloud console but are colocated with IBM Cloud resources.
• Power Systems Virtual Servers can run AIX, IBM i, or Linux operating systems.
• Some key features are monthly billing rates, customized infrastructure, and images from AIX and IBM i.
• Backup strategies for AIX are Veeam AIX and IBM Spectrum Project and IBM i uses IBM Backup, Recovery, and Media Services.

In IBM Cloud Code Engine, you should have learned:

• Code engine is a serverless platform and offers several benefits including:
• Building apps with your code using IBM’s infrastructure
• Being able to run HTTP-driven apps and run-to-completion batch jobs,
• Having private workloads
• Allowing clients to determine who gets access to the resources

Compute Option Use Cases

In Bare Metal Use Case, you should have learned:

• Bare Metal servers are typically used where performance is a key concern — such as where data needs to be retrieved quickly from local storage or where a specific hardware type is required for high bandwidth, encrypted data processing.
• Bare Metal servers are also great when a dedicated environment is needed due to security, policy, or compliance requirements.

In Virtual Server Use Cases on IBM Cloud, you should have learned:

• There are three virtual compute options, Virtual Server for VPC, Virtual Servers for Classic Infrastructure, and Power Systems Virtual Servers.
• Some use cases for Virtual Server for VPC are hosting and building apps and advancing workload capacities.
• Use cases for Virtual Server for Classic include high-level needed for computing and apps that have multiple levels.
• Use cases for Power Systems Virtual Servers include developing and experimenting IBM power apps and workloads, along with disaster recovery.

In VMware Use Cases on IBM Cloud, you should have learned:

• VMware in IBM Cloud enables:
• Datacenter transformation and the ability to easily increase capacity or migrate to newer hardware more cost-effectively.
• The simple creation of a disaster recovery environment in the cloud.
• A secure environment through data encryption.
• The option of running RedHat OpenShift on VMware to provide cloud-native services.

In Skytap Service on IBM Cloud Offerings, you should have learned:

• Skytap is a cloud service that can easily migrate AIX, IBM i, and Linux and can be quickly provisioned. It is highly available and can create a self-managed application environment.

In Virtual Desktop Applications for IBM Cloud Offering, you should have learned:

• Some virtual desktop applications include Citrix and Dizzion. Citrix allows for easy management of the virtual desktop infrastructure on the IBM Cloud.
• Dizzion allows organizations to build remote work environments. There are two service levels for Dizzion: Dizzion DaaS and Dizzion Managed DaaS.
• Dizzion DaaS is provided by IBM and Dizzion and controls Horizon 7 to handle virtual desktops. Dizzion Managed DaaS is similar to Dizzion DaaS but has additional features such as Desktop Patching, Pool Management, GPO assistance, etc.

IBM Cloud Satellite

In IBM Cloud Satellite, you have learned:

• IBM Cloud Satellite enables organizations to run IBM Cloud Services in the location of their choice: on-premises, a competitor’s cloud, or at the edge.
• IBM Cloud Satellite enables users to run select IBM Cloud Services on-premises, behind their corporate firewall, allowing them to meet more demanding security rules or simply process data closer to its source
• There are four main Satellite components: hosts, link, location, and endpoints.
• A host is a customer-provided computer running Red Hat Enterprise Linux 7.x. A minimum of three hosts are required for a Satellite location.
• A Satellite location is a customer data center, competitors’ cloud, or edge location, at which the Satellite Hosts are deployed and run.
• A Satellite Link is an encrypted TLS tunnel that allows you to securely connect the Satellite installation to other services running in IBM Cloud and to manage and monitor the Satellite instance from the IBM Cloud Console.
• Satellite Link has two types of Endpoint: a Cloud Endpoint and a Location Endpoint. The Cloud Endpoint destination is outside of the Satellite location, while the Location Endpoint provides access to a server, service, or app that runs in your Satellite location from a client that is connected to the IBM Cloud private network.

IBM Cloud Storage Offerings and Use Cases

In Introduction to Storage Options, you should have learned that:

• The three most common types of storage in IBM Cloud are block, file, and object.

In Block Storage, you should have learned that:

• Block storage stores data by breaking it into evenly sized blocks, which is written to disk. The data stored is highly available because each block is stored multiple times across different disks.
• Block storage is used for computing situations where users require fast, efficient, and reliable data storage.
• Block storage decouples data from user environments. This allows that data to be spread across multiple environments enabling the user to retrieve it quickly.
• Block storage can be deployed on a Virtual Private Cloud (VPC) or classic environments.
• There are two available tiers: the IOPS tier, and the Custom IOPS profile tier.

In File Storage, you should have learned that:

• File storage uses a hierarchical structure to organize files, folders, and subfolders.
• There are strategic benefits to choosing file storage: reducing costs, and scaling up capacity.
• It is available in the endurance and performance tier.

In Object Storage, you should have learned that:

• Object storage is a storage option that manages unstructured data into self-contained units called Objects. Object storage is referred to as unstructured as it does not use a conventional ‘folder/subfolder’ structure to store objects. Instead, it uses a flat structure known as a “bucket.”
• IBM Cloud Object Storage stores encrypted and dispersed data across multiple geographic locations.
• There are some advantages to using object storage. It enables customers to handle large amounts of unstructured data, is scalable and cost-effective, uses metadata allowing users to maximize the search feature, and quickly access the object they need.
• IBM Cloud Object Storage service offers different levels of resiliency: cross region, regional, and single data center.
• There are four tiers, including Smart, Standard, Vault, and Cold Vault tier.

In Software Defined Storage (SDS) Offerings, you should have learned that:

• Software-defined storage (SDS) is a storage architecture that separates storage software from its hardware.
• It enables organizations to increase their storage capacity quickly making it flexible and scalable.
• SDS allows the organization to use existing hardware that they currently have, which can be a tremendous cost savings.
• Portworx is a highly available SDS that can be used as persistent storage management for containerized applications.

Backup, Recovery, and Replication Solutions

In IBM Cloud Backup Capabilities, you should have learned that:

• IBM Cloud Backup Portal is a browser-based management utility that enables customers to back up data between servers or data centers on the IBM Cloud network.
• Backup and restore make periodic copies of data and applications to a secondary device.
• RTO is the maximum amount of time the organization can afford to be without access to the data or application. In other words, how quickly the customer needs to recover the data and application.
• RPO is the amount of data an organization can afford to lose and effectively dictates how frequently they need to back up their data to avoid losing more.
• There are four types of backup devices or services a customer can use: tape drive, HDD or SDD, backup server, or cloud backup.
• Commonly used back and restore methods include full image only, incremental, differential, CDP, bare metal backup, and instant recovery.
• Disaster recovery is much different than backup and restore. Disaster recovery is a plan that is created to deal with an outage that impacts applications, data, and IT resources.
• Data replication allows the use of real-time information that captures data that is constantly changing to allow for efficient data growth management.

In Zerto, Veeam, and IBM Spectrum Protect Plus within VMware Solutions in IBM Cloud, you should have learned that:

• There are a wide variety of services used for disaster recovery and protection. Zerto, Veeam, and IBM Spectrum Protect Plus are the most commonly used for these services.
• Zerto is a service that provides replication and disaster recovery capabilities using continuous data replication with journaling versus snapshots.
• Veeam is a service that integrates directly with VMware hypervisors to help an organization achieve high availability to control backup and restore for all the virtual machines attached to their infrastructure from one console.
• IBM Spectrum Protect Plus creates an efficient and scalable solution for clients who need data protection, reuse, and recovery in virtual environments.

In Disaster Recovery options in IBM Power Systems Virtual Server, you should have learned that:

• In the event of a hardware failure, the IBM Power Systems Virtual Server service will restart the virtual servers on a different host system to provide uninterrupted service.
• There are four strategies used for data recovery, including image capture, AIX backup, IBM i backups, and cloud object storage.

Networking Components and their Advantages

In Basic Networking Options, you should have learned:

• Automatic VLANs are provisioned and removed automatically based on whether there are cloud services which need them.
• VRF allows multiple instances of a routing table to exist in a router and to work simultaneously.
• The cloud automatically assigns and manages primary subnets.

In Load Balancer Options, you should have learned:

• Distributing connections using load balancers prevents server overload and enhances uptime.
• Cloud Internet Services provide global load balancer services.

In Direct Link Offerings, you should have learned:

• Direct Link offerings provide connectivity from external sources into a private cloud network.
• Direct Link Connect on Classic provides private access to the cloud infrastructure.
• Global routing can be added to all Direct Link products.

In Virtual Private Cloud Networking, you should have learned:

• A VPC provides cloud security to workloads.
• By default, resources in a VPC are isolated from other services.
• A VPC can contain multiple subnets within each zone.
• A public gateway can be attached to a VPC subnet to allow the subnet to route traffic to and from the internet.

IBM Networking Components

In Direct Link Use Cases, you should have learned that:

• Direct Link creates private connections between on-premises environment and cloud resources.
• Private connects are created without using the public internet.
• Workloads with large and frequent data transfers are supported.

In Public And Private Components Use Cases, you should have learned that:

• VRF connections to resources are more secure because of the use of a private network endpoint.
• The BYOIP option enables users to connect existing networks to cloud infrastructure.
• Workload isolation helps virtual machines and bare metal servers securely deploy with less risk.

In Use Cases for Local and Global Load Balancers, you should have learned that:

• Load balancing helps prevent failure when resources are overloaded.
• Hardware load balancers are less flexible and scalable.
• A global load balancer helps balance workloads across regions.

VPC and Related Networking Concepts

In VPCs, you should have learned that:

• A virtual private cloud provides a private cloud environment on a shared public cloud infrastructure.
• VPC infrastructure is deployed across three zones.
• VPC cloud resources have their own isolated virtual network.

In Transit Gateways, you should have learned that:

• Transit gateways connect VPC resources across multiple regions.
• Routing options stay in the private cloud infrastructure.
• Connecting transit gateways to Direct Link enables connection to an on-premises network and other networks connected to the transit gateway.

Integration use case

In When to Use API Connect, APP Connect, or IBM Messaging Options, you should have learned that:

• Some organizations have adapted to the added difficulty of integration by replacing large code silos with small, independent components called microservices. This strategy offers greater scalability, resilience, and agility, which is why it is called agile integration.
• Agile integration decentralizes services, so applications teams can better control the creation and exposure of their application program interfaces (APIs), messages, and events.
• IBM API Connect lets users create, secure, expose, manage, socialize, and analyze its APIs across clouds.
• IBM APP Connect integrates data and applications from existing systems, and ties together technologies across environments, including legacy and SaaS systems.
• Apache Kafka-based IBM Event Streams enable organizations to create smart applications that react to events as soon as they happen.
• IBM MQ is a scalable messaging platform that uses Java Message Service (JMS) and other technologies to integrate applications and put information such as product availability on the cloud, where consumers can easily retrieve it.

Edge Solutions

In How Edge Functions Work, you should have learned that:

• Edge computing is a distributed computing method that aims to use bandwidth more efficiently, by bringing applications close to where data is created and actions are performed.
• By eliminating the need to send data over a network, edge computing reduces latency and enables real-time processing.
• When implementing an IBM Cloud Internet Service (CIS) Edge function, developers consider the actions they want it to perform. For each action, they define a uniform resource identifier (URIs) called a trigger. IBM CIS Edge intercepts user requests, compares them against the list of triggers, and performs the associated action if it finds a match, called a trigger event.
• Because edge functions run code on trusted CIS Edge servers, users don’t need to use a modern browser.
• If processing needs to occur on client-provided servers physically located at the edge, IBM Cloud Satellite may be a more appropriate solution.

Cloud AI and analytics use case

In IBM’s AI Ladder, you should have learned that:

• IBM’s AI Ladder views the process of gathering, preparing, and using data in terms of a ladder comprised of four steps: collecting data, organizing data, analyzing data to scale business insight, and infusing data to implement AI with trust and transparency.

In IBM Services That Support Analytics and AI, you should have learned that:

• IBM offers many services that use AI and analytics to improve business processes. These include:
• Watson Assistant, which can build dynamic, branded chatbots.
• Watson Studio, which can create custom data models using RStudio and Jupyter Notebooks.
• Watson OpenScale, which can track AI outcomes, monitor compliance, and measure alignment with business goals.

In Use Cases Associated with Analytics and AI, you should have learned that:

• Two applications of AI and analytics include COVID-19 information delivery and explanations of AI methods.

IBM Cloud Managed Database Options

In When to Use SQL and NoSQL Databases, you should have learned that:

• Most databases fall into one of two groups:
• SQ
• NoSQL

In Benefits of Database-as-a-Service (DBaaS), you should have learned that:

• DBaaS, sometimes called “managed database service,” lets users access and use a database over the cloud. The provider handles upgrades, backups, and other maintenance tasks to keep the system running 24/7.
• Benefits include cost savings, scalability, simplicity, rapid development, security, reduces risk, and quality.
• Factors to consider in selecting a DBaaS provider include how specialized the application will be, the architecture, test results, and other provider offerings.

In IBM-Managed Cloud Databases, you should have learned that:

• IBM offers managed cloud databases for various applications, including web and mobile apps, developer tools, confidential data, and business intelligence.

Benefits and Options of IBM Hyper Protect Crypto Services

In IBM Hyper Protect Crypto Services Features, you should have learned that:

• IBM Cloud Hyper Protect Crypto Services (HPCS) is a dedicated key management service that uses a dedicated cryptographic processing Hardware Security Module (HSM) to generate, encrypt, store, and decrypt keys.
• The HSM is built on FIPS 140–2 Level 4 certified hardware: the highest level available for cryptographic security.
• Customers retain exclusive control of their keys with a feature called Keep Your Own Key (KYOK).

In Key Database Services that Integrate with IBM Hyper Protect Crypto Services, you should have learned that:

• Users can integrate HPCS with selected managed database services to bring and manage their own encryption in the cloud.
• Integration involves associating user-managed HPCS root keys to control the randomly generated ones from the database service, and then adding another layer of protection, envelope encryption, to the data.

In Key Management Concepts, you should have learned that:

• By default, cloud providers control system keys. But for even greater protection, users should control their own keys.
• In order to keep all of their keys, users must set up the HSM themselves, which requires a dedicated solution like HPCS.

In Use Cases Associated with IBM Hyper Protect Crypto Services, you should have learned that:

• Customers benefit from HPCS cryptographic capabilities in many ways: image protection, app development, database encryption, and enterprise environments.

Compute Security Options

In Compute Security Options, you should have learned that:

• A Virtual Private Cloud (VPC) is a private, secure location that enables organizations to define and control a virtual network that is essentially isolated from other users in the public cloud environment.
• Logical isolation secures VPCs.

In Security Groups and Access Control Lists (ACLs) Secure Solutions in IBM Cloud VPC, you should have learned that:

• Security Groups and ACLs are the primary way to enhance security to subnets and instances and can be used independently or together.

In Encryption Options Available in IBM Cloud to Protect Compute Focus Solutions, you should have learned that:

• IBM-managed encryption, customer-managed encryption, and end-to-end encryption are the encryption options available in IBM Cloud.

In Methods of Securing IBM Cloud VMware based solutions, you should have learned that:

• IBM Cloud for VMware Solutions enables users to manage resources like they are on-premises through dedicated infrastructure.
• IBM Cloud for VMware Regulated Workloads provides encryption.

In Role-Based Access Control Security uses within IBM Cloud, you should have learned that:

• IBM Cloud Identity and Access Management (IAM) provides access to services in IBM Cloud through the model of least privilege and does this through Resource Groups and Access Groups.

Network Security Options

In Network Security Options, learners should have learned that:

• ACLs and security groups are two types of network access controls that make up the layers of VPC security.
• Network security provides security to information in a network and also controls who can access the network, but there are vulnerable entry points that warrant defense mechanisms for enhanced security.

In Network Encryption Options Available in IBM Cloud to Protect Solutions, learners should have learned that:

• Juniper vSRX is a router, firewall, and security device that exists as a virtual appliance and provides a firewall, VPN gateway, and NAT as an encryption feature.
• There are multiple ways to secure data in transit or data at rest.

Storage Security Options

In Advantages of Using IBM Cloud Storage Services Security Options when Building Solutions, you should have learned that:

• IBM Cloud File Storage and IBM Cloud Block Storage are provisioned with Endurance or Performance options.
• IBM Cloud Object Storage provides the ability to store large volumes of unstructured data.

In RBAC Secures Solutions built on IBM Cloud, you should have learned that:

• IAM, OpenShift & Containers, and VMware provide varying levels of access policies in order to secure solutions built on IBM Cloud.

In Various Storage Services Protect Solutions in IBM Cloud,you should have learned that:

• Block Storage for VPC gives users the ability to provide hypervisor-mounted, high-performance data storage for any VSIs that can be provisioned within a VPC.
• IBM Cloud Object Storage provides storage for large volumes of unstructured data that provides security, availability, and reliability.

In Secure Storage Options when Utilizing IBM Cloud VMware Solutions, you should have learned that:

• When utilizing VMware Solutions Dedicated, data can be stored locally, or using IBM Cloud File Storage or Cloud Object Storage.
• When using VMware Solutions Shared, the workload data is in an IBM-managed cloud infrastructure account.
• VSIs and customer-managed encryption are other options to secure data.

PaaS Security Options

In Network Security, you should have learned that:

• Data Shield is an IBM Cloud service that helps protect data in containerized workloads that run on Kubernetes Service and OpenShift clusters while the data is in use.
• Service endpoints are a connectivity option for securely accessing cloud service endpoints.
• Application exposure services enable users to securely expose applications to external traffic.

In Authenticating Users, you should have learned that:

• SSO provides authentication between multiple web apps.
• App ID is a service provided by IBM Cloud that allows users to create and use SSO for their own applications.
• vCenter Single Sign-On (SSO) is an authentication broker and security token exchange infrastructure. This allows vSphere to communicate with each other via a secure token mechanism.

In Encryption, Secrets, and Certificates, you should have learned that:

• Secrets Manager creates dynamic secrets and manages the lifecycle of the secrets.
• IBM Key Protect securely stores and applies secrets for apps. It provides encryption solutions and allows data to be secured and stored in IBM Cloud through envelope encryption.
• Hyper Protect Crypto Services is a key management system that provides keep your own key (KYOK) capabilities for cloud data encryption. It provides lifecycle management for keys, encryption for IBM Cloud services, access management, auditing, and security certification.
• VMware integrates on-premises vSphere vCenter networks to the IBM Cloud for VMware solutions deployment.

In IBM Cloud Code Engine and Security, you should have learned that:

• IBM Code Engine provides a security solution by isolating customers and their workloads.

In Delivery Pipeline Private and Public Workers, you should have learned that:

• Delivery Pipeline Private Workers and the IBM Cloud Continuous Delivery Development teams work together to use the private worker in their toolchain.

In Other Security Topics, you should have learned that:

• Terraform is a third-party service used by IBM Cloud, and it enables predictable provisioning of the IBM Cloud platform, classic infrastructure, and VPC infrastructure.
• Data Shield is an IBM Cloud service that helps protect data in containerized workloads that run on Kubernetes and OpenShift clusters while the data is in use.

Core Principles and Practices of Building Cloud-Native Applications

In Define Cloud Native, its Benefits, and List Use Cases, you learned that:

• Scaling of cloud native applications is very easy.
• Cloud native applications are made up of microservices.

In The Twelve-Factor App Methodology and How Microservices Benefit the Organization, you learned that:

• Microservices are independently deployable.
• Microservices have specialized functionality.
• Twelve-factor app methodology uses a declarative approach, reducing the time needed to onboard new developers.

In Key Enabling Technologies and Tools for a Cloud Native Solution, you learned that:

• Containers are self-contained.
• Container orchestration systems are used to manage containers.
• Kubernetes is an example of a container orchestration system.
• IBM Cloud Functions and IBM Code Engine are examples of serverless technologies.
• APIs allow applications to communicate with each other.
• REST APIs have additional architectural constraints to standard APIs.
• Messaging and event streaming are used for routine status updates from applications.
• Serverless architecture is designed to be a flexible alternative to a traditional server.

High-Level Benefits of Modernizing Existing Applications to be Cloud-Native

In The Importance of Application Modernization, you learned that:

• Modernization can reduce the cost of managing applications.
• Containerization can be an easy yet effective approach to modernizing applications.
• Security and compliance of an application can be increased by modernization.
• Development speed can be increased by modernizing.

In Application Modernization Approaches, you learned that:

• Existing applications can be modernized by replacing existing functionality with microservices.
• APIs can be used to expose parts of an existing application, making it easier to use microservices for new functionality.

In Whether to Modernize or Rebuild an Existing Application, you learned that:

• The first step in application modernization is to assess how ready the application is to be moved to the cloud.
• Containerization can reduce costs and resources when modernizing an application.
• Rebuilding an application is made easier by using existing microservices.

IBM Container Orchestration

In Containers and How They Differ From Virtual Machines (VMs), you learned that:

• Containers are self-contained units that are designed to hold applications as well as all libraries and dependencies that the application requires.
• Containers do not have an operating system of their own. Instead, they access the operating system of the device they are running on using a hypervisor.

In Containers, Container Orchestration, and Kubernetes, you learned that:

• The process of designing and packaging software in containers is called containerization.
• When an application is containerized, it is packaged with its relevant environment variables, configuration files, libraries, and software dependencies.
• Container orchestration is a way to manage large volumes of containers.
• Kubernetes is a platform that enables customers to manage their containerized workloads. It provides tools that enable developers to deploy, automate, monitor, and scale their apps efficiently.

In An Overview of Red Hat OpenShift on IBM Cloud, you learned that:

• Red Hat OpenShift on IBM Cloud enables customers to deploy apps on Red Hat OpenShift clusters. These clusters run on the Red Hat OpenShift on IBM Cloud container platform software.
• Red Hat OpenShift on IBM Cloud provides customers with a fast, secure, and scalable solution.

In IBM Cloud Kubernetes Service and its Benefits, you learned that:

• IBM Cloud Kubernetes Service is a certified Kubernetes provider that enables customers to create their own Kubernetes cluster of compute hosts to deploy and manage their containerized apps.
• Some benefits of IBM Cloud Kubernetes Service include that customers can choose to deploy clusters with Red Hat OpenShift or Kubernetes installed, it is a single-tenant Kubernetes clusters with compute, network, and storage infrastructure isolation, supports multizone clusters, has image security compliance with Vulnerability Advisor, has highly available masters, continuous monitoring of the cluster health, and secure exposure of apps to the public.

Solution Scalability

In Horizontal and Vertical Scaling for Containers, you learned that:

• Customers can quickly scale up simply by increasing the CPU and RAM.
• Customers can scale out by provisioning additional VMs to spread out the workload.

In Scaling Clusters Using Cluster Autoscaler, you learned that:

• The cluster autoscaler periodically scans the cluster and adjusts the number of worker nodes based on the workload requests.
• If there are insufficient compute resources to schedule the pod on a worker node, that pod is considered pending. If a pending pod is detected, the autoscaler scales up the worker nodes evenly across zones.
• If a pod is not fully utilized — meaning less than 50% of the total compute resources have been requested in the last 10 minutes — the autoscaler considers it underutilized and will scale the worker node down one at a time. This threshold can be customized by the customer.

In Horizontal and Pod Autoscalers, you learned that:

• An HPA enables customers to determine the minimum or the maximum number of pods to run, the CPU, and memory utilization the pods should utilize.
• The HPA automatically scales the workload to match the demand.
• When there is an increased workload, it deploys more pods.
• When there is a decrease in the workload, it communicates to the workload resource to scale back down.

In Vertical Pod Autoscalers, you learned that:

• Vertical pod autoscaling (VPA) helps size pods for optimal CPU and memory utilization.
• Like HPA, VPA uses the cluster nodes efficiently because the pods use exactly what they need.

Hybrid and Multicloud Architectures

In Technology and Strategies in Hybrid Cloud Solutions, you learned that:

• The hybrid cloud combined the public cloud, private cloud, and on-premises infrastructure. It enables customers to have a distributed environment where they can run their traditional or cloud-native workloads as they need to.
• The hybrid cloud enables customers to have improved developer productivity, greater infrastructure efficiency, improved regulatory compliance and security, and overall business acceleration.
• Multicloud is the use of cloud services from more than one cloud vendor.
• A multicloud solution is a solution that is portable across these providers’ cloud infrastructures, usually build on open-source, cloud-native technologies.
• IBM Cloud Satellite provides consistency in services and the ability to deploy anywhere.

In The Role of IBM Cloud Paks in the Hybrid Multicloud Strategy, you learned that:

• Cloud Paks are pre-integrated Artificial Intelligence (AI), powered software that is designed to run in a customer’s cloud environment managed by a single intelligent control plane.
• IBM Cloud Paks are portable, can run anywhere, are certified, and secure.
• There are 12 foundational services available.

https://www.linkedin.com/in/alvisf/

We need to set up the nginx if you have configured it to HTTP previously. Once you are done with can we can move on to the next step.

Steps to get HTTPS

1. Get SSL/TLS Certificated
2. Create a LoadBalancer
3. Route all Trafic via LoadBalancer
4. Route all Trafic via HTTPS

Now In this example in using route 53’s Domain name service. Even if you don’t have one you can still use the tutorial with slight changes which will be mentioned.

Get SSL/TLS Certificated

Now we can get a free certificate from the AWS Certificate Manager

1. Goto Certificate Manager

2. Click on request a certificate

3. Select Public Certificate & click request certificate

4. Enter your domain name with .com or any subdomain if needed. & click Next

5. Select DNS validation for quick certificate generation but of course, you can use Email validation if needed & click Next

6. If you would like to add tags you can but I’m skipping for now & click Review

7. Review all the details if they are correct now click Confirm and request.

8. Now you need to create a record in your hosted zone with this value. But AWS offers you to do the work for you. So you can just click on create a record in Route 53 or manually do it yourself & click Continue

Wait for a few minutes you will get an issued status. Once you get that you can you can move on to the next step.

Create a LoadBalancer

If your application is hosted on an EC2 instance we need to make sure we have trafficked it via a load balancer.

We can only attach the ACM Certificate with a loadbalancer.

1. Create a new Loadbalancer choose https/https

2. add a listener and add https with port 443 & click configure security settings

3. Select Choose a certificate from ACM and choose the certificate you just got & click next

4. choose the protocol to connect to the target group as HTTP with port 80 because you don’t need your loadbalncer to talk to your ec2 instance via a HTTPS, HTTP would be more than sufficient to do the job & click next

5. Add your EC2 to your target group and click next

6. Review all the information & click Create

Now we have successfully created a loadbalancer and HTTPS should work in a few minutes.

clear your browser cache or see if your target group is set via port 80.

Route all Trafic via HTTPS

This must be the shortest step of all. All we are trying to achieve here is to force traffic to go through HTTPS even if they try to go through HTTP.

1. go to your loadbalncer and select your loadbalancer and choose port 80 and hit edit

2. change the route to redirect to HTTPS and click on Update at the top.

🥳 That’s it, you have successfully changed your HTTP app to HTTPS

Alvis F - Software Engineer - EY | LinkedIn

Get HTTPS 🔐! How to Get SSL\TLS Certificate in AWS for EC2 Hosted Application. was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

Moving data from your application database into another database with negligible effect on the functionality of your application is the key inspiration for using a shift data capture architecture pattern.

Uses

1. tracking data changed to feed into an elastic search index.
2. moving data changes from OLTP to OLAP in real-time
3. creating audit logs, etc

Project overview

We are going to use Mysql for our database

‍

Components

Prereq Installations

In order to follow along you will need the tools specified below

1. docker (preferred) to run postgres, debezium and kafka
2. pgcli to connect to our postgres instance

1. Postgres

This will serve as our application database. To understand how CDC (with debezium) works we need to understand what happens when a transaction occurs. When a transaction occurs, the transaction is logged in a place called Write Ahead Log (WAL) in the disk and then the data change or update or delete is processed. The transactions are generally held in cache and flushed to disk in bulk to keep latency low. In case of a database crash we may loose the cache but the database can recover using the logs in WAL in disk. Using WAL, only logs are written to the disk which is less expensive than writing all the data changes to the disk, this is a tradeoff the developers of postgres had to make to keep latency of transactions low and have the ability to recover in case of crash(using WAL).

You can think of the WAL as an append only log that contains all the operations in a sequential manner with the timestamp denoting when the transaction was logged. The WAL files are periodically removed or archived so that the size of the database is kept small.

We are going to use docker to run a postgres instance

docker run -d --name postgres -p 5432:5432 -e POSTGRES_USER=start_data_engineer \
-e POSTGRES_PASSWORD=password debezium/postgres:12

The above docker command starts a postgres docker container named postgres. The user name is set as start_data_engineer and password is password. If you notice you will see we have used the debezium/postgres:12 image, the reason for using the debezium’s docker of postgres was to enable the settings that postgres requires to operate with debezium. Let’s take a look at those settings

docker exec -ti postgres /bin/bash

Use the above command to execute the /bin/bash command on the postgres container in interactive mode -it. You are now inside your docker container. Type in

cat /var/lib/postgresql/data/postgresql.conf

to view the configuration settings for postgres.

postgresql.conf

‍

The Replication section is where we set the configuration for the database to write to the WAL. The

1. wal_level has options of minimal: minimal information required to restart from a database crash, archive: enables the database engine to be able to do WAL archiving, hot_standby: enables the database engine to create a read only replica of our server, logical: is what we want for our purposes, it adds all the information necessary to make this (WAL) data available for other systems to consume.
2. max_wal_senders the WAL senders are process that run on the database to send WAL to receivers (other replica or systems).This config denotes the maximum number of WAL sender processes allowed.

Let’s create the data in postgres. We use pgcli to interact with our postgres instance

pgcli -h localhost -p 5432 -U start_data_engineer
#password is password

CREATE SCHEMA bank;
SET search_path TO bank,public;
CREATE TABLE bank.holding (
    holding_id int,
    user_id int,
    holding_stock varchar(8),
    holding_quantity int,
    datetime_created timestamp,
    datetime_updated timestamp,
    primary key(holding_id)
);
ALTER TABLE bank.holding replica identity FULL;
insert into bank.holding values (1000, 1, 'VFIAX', 10, now(), now());
\q

The above is standard sql, with the addition of replica identity. This field has the option of being set as one of DEFAULT, NOTHING, FULL and INDEX which determines the amount of detailed information written to the WAL. We choose FULL to get all the before and after data for CRUD change events in our WAL, the INDEX option is the same as full but it also includes changes made to indexes in WAL which we do not require for our project’s objective. We also insert a row into the holding table.

2. Kafka

Kafka is a message queue system with an at least once guarantee. A quick overview of some key kafka concepts

1. Kafka is a distributed message queue system. The distributed cluster management is provided by zookeeper.
2. The broker handles consumer write, producer request and metadata config. One server within a kafka cluster is one kafka broker, there can be multiple brokers within a single kafka cluster
3. A Topic is a specific queue into which the producers can push data into and consumers can read from.
4. Partitions are way to distribute the content of a topic over the cluster.
5. You can think of offset (specific to a partition) as the pointer pointing to which message you are in while reading messages from that topic-partition.

In our project the kafka broker will be used to store the data changes being made in the postgres database as messages. We will set up a consumer in the later sections to read data from the broker.

Let’s start zookeeper and a kafka broker

docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.1
docker run -d --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.1

Again we are using debezium images for ease. You can see for the zookeeper we keeps ports 21821, 2888, 3888 open, these are required for zookeeper operations. And similarly we keep 9092 open for kafka, we can communicate with kafka through this port.

3. Debezium

We use a kafka tool called Connect to run debezium. As the name suggests connect provides a framework to connect input data sources to kafka and connect kafka to output sinks. It runs as a separate service.

Debezium is responsible for reading the data from the source data system (in our example postgres) and pushing it into a kafka topic (automatically named after the table) in a suitable format.

Let’s start a kafka connect container

docker run -d --name connect -p 8083:8083 --link kafka:kafka \
--link postgres:postgres -e BOOTSTRAP_SERVERS=kafka:9092 \
-e GROUP_ID=sde_group -e CONFIG_STORAGE_TOPIC=sde_storage_topic \
-e OFFSET_STORAGE_TOPIC=sde_offset_topic debezium/connect:1.1

Notice we are specifying the kafka host and endpoint with BOOTSTRAP_SERVERS env variable. The GROUP_ID here represents a group this connect service belongs to. We can use curl to check for registered connect services Note: wait for a few seconds (atleast 10 sec) before running the curl command below

curl -H "Accept:application/json" localhost:8083/connectors/
[]

We can register a debezium connect service using a curl command to the connect service on port 8083

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \
localhost:8083/connectors/ -d '{"name": "sde-connector", "config": {"connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres", "database.port": "5432", "database.user": "start_data_engineer", "database.password": "password", "database.dbname" : "start_data_engineer", "database.server.name": "bankserver1", "table.whitelist": "bank.holding"}}'

Let’s take a look at the configuration part of the api call above

{
  "name": "sde-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "start_data_engineer",
    "database.password": "password",
    "database.dbname": "start_data_engineer",
    "database.server.name": "bankserver1",
    "table.whitelist": "bank.holding"
  }
}

1. The database.* configs are connection parameters for our postgres database
2. database.server.name is a name we assign for our database
3. table.whitelist is a field that informs the debezium connector to only read data changes from that table. Similarly you can whitelist or blacklist tables or schemas. By default debezium reads from all tables in a schema.
4. connector.class is the connector used to connect to our postgres database
5. name name we assign to our connector

Let’s check for presence of connector

curl -H "Accept:application/json" localhost:8083/connectors/
["sde-connector"]%

We can see the sde-connector is registered.

4. Consumer

Now that we have our connector pushing in message into our kafka broker, we can consume the messages using a consumer. Let’s take a look at only the first message in the kafka topic bankserver1.bank.holding using the command below

docker run -it --rm --name consumer --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.1 watch-topic -a bankserver1.bank.holding --max-messages 1 | grep '^{' | jq

In the above we start a consumer container to watch the topic bankserver1.bank.holding which follows the format {database.server.name}.{schema}.{table_name} and we have set the maximum number of messages to be read by this consumer to be 1 . The grep is to filter out non JSON lines, as the docker container will print out some configs. jq(optional download) is to format the json. The output should be similar to the structure shown below

{
  "schema": {
    "type": "struct",
    "fields": [
      {
        "type": "struct",
        "fields": [
          {
            "type": "int32",
            "optional": false,
            "field": "holding_id"
          },
          {
            "type": "int32",
            "optional": true,
            "field": "user_id"
          },
          {
            "type": "string",
            "optional": true,
            "field": "holding_stock"
          },
          {
            "type": "int32",
            "optional": true,
            "field": "holding_quantity"
          },
          {
            "type": "int64",
            "optional": true,
            "name": "io.debezium.time.MicroTimestamp",
            "version": 1,
            "field": "datetime_created"
          },
          {
            "type": "int64",
            "optional": true,
            "name": "io.debezium.time.MicroTimestamp",
            "version": 1,
            "field": "datetime_updated"
          }
        ],
        "optional": true,
        "name": "bankserver1.bank.holding.Value",
        "field": "before"
      },
      {
        "type": "struct",
        "fields": [
          {
            "type": "int32",
            "optional": false,
            "field": "holding_id"
          },
          {
            "type": "int32",
            "optional": true,
            "field": "user_id"
          },
          {
            "type": "string",
            "optional": true,
            "field": "holding_stock"
          },
          {
            "type": "int32",
            "optional": true,
            "field": "holding_quantity"
          },
          {
            "type": "int64",
            "optional": true,
            "name": "io.debezium.time.MicroTimestamp",
            "version": 1,
            "field": "datetime_created"
          },
          {
            "type": "int64",
            "optional": true,
            "name": "io.debezium.time.MicroTimestamp",
            "version": 1,
            "field": "datetime_updated"
          }
        ],
        "optional": true,
        "name": "bankserver1.bank.holding.Value",
        "field": "after"
      },
      {
        "type": "struct",
        "fields": [
          {
            "type": "string",
            "optional": false,
            "field": "version"
          },
          {
            "type": "string",
            "optional": false,
            "field": "connector"
          },
          {
            "type": "string",
            "optional": false,
            "field": "name"
          },
          {
            "type": "int64",
            "optional": false,
            "field": "ts_ms"
          },
          {
            "type": "string",
            "optional": true,
            "name": "io.debezium.data.Enum",
            "version": 1,
            "parameters": {
              "allowed": "true,last,false"
            },
            "default": "false",
            "field": "snapshot"
          },
          {
            "type": "string",
            "optional": false,
            "field": "db"
          },
          {
            "type": "string",
            "optional": false,
            "field": "schema"
          },
          {
            "type": "string",
            "optional": false,
            "field": "table"
          },
          {
            "type": "int64",
            "optional": true,
            "field": "txId"
          },
          {
            "type": "int64",
            "optional": true,
            "field": "lsn"
          },
          {
            "type": "int64",
            "optional": true,
            "field": "xmin"
          }
        ],
        "optional": false,
        "name": "io.debezium.connector.postgresql.Source",
        "field": "source"
      },
      {
        "type": "string",
        "optional": false,
        "field": "op"
      },
      {
        "type": "int64",
        "optional": true,
        "field": "ts_ms"
      },
      {
        "type": "struct",
        "fields": [
          {
            "type": "string",
            "optional": false,
            "field": "id"
          },
          {
            "type": "int64",
            "optional": false,
            "field": "total_order"
          },
          {
            "type": "int64",
            "optional": false,
            "field": "data_collection_order"
          }
        ],
        "optional": true,
        "field": "transaction"
      }
    ],
    "optional": false,
    "name": "bankserver1.bank.holding.Envelope"
  },
  "payload": {
    "before": null,
    "after": {
      "holding_id": 1000,
      "user_id": 1,
      "holding_stock": "VFIAX",
      "holding_quantity": 10,
      "datetime_created": 1589121006547486,
      "datetime_updated": 1589121006547486
    },
    "source": {
      "version": "1.1.1.Final",
      "connector": "postgresql",
      "name": "bankserver1",
      "ts_ms": 1589121006548,
      "snapshot": "false",
      "db": "start_data_engineer",
      "schema": "bank",
      "table": "holding",
      "txId": 492,
      "lsn": 24563232,
      "xmin": null
    },
    "op": "c",
    "ts_ms": 1589121006813,
    "transaction": null
  }
}

Basically there 2 main sections

1. schema: The schema defines the schema of the payload for before, after, source, op, ts_ms and transaction sections. This data can be used by the consumer client program to parse the input.
2. payload: This contains the actual data, there are before and after sections showing the columns of holding, before and after the data change. The data change is represented by the op which is one of c create (insert), u for update, d for delete, and r for read (at snapshot). The source section shows the WAL configurations with the tx_id denoting the transaction id and lsn log sequence number which are used to store the byte offset of the log file per WAL record.

We care about the before and after sections. We will use python to format this into the pivot structure we defined at the start.

#!/usr/bin/env python3 -u
# Note: the -u denotes unbuffered (i.e output straing to stdout without buffering data and then writing to stdout)

import json
import os
import sys
from datetime import datetime

FIELDS_TO_PARSE = ['holding_stock', 'holding_quantity']

def parse_create(payload_after, op_type):
    current_ts = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
    out_tuples = []
    for field_to_parse in FIELDS_TO_PARSE:
        out_tuples.append(
            (
                payload_after.get('holding_id'),
                payload_after.get('user_id'),
                field_to_parse,
                None,
                payload_after.get(field_to_parse),
                payload_after.get('datetime_created'),
                None,
                None,
                current_ts,
                op_type
            )
        )

return out_tuples

def parse_delete(payload_before, ts_ms, op_type):
    current_ts = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
    out_tuples = []
    for field_to_parse in FIELDS_TO_PARSE:
        out_tuples.append(
            (
                payload_before.get('holding_id'),
                payload_before.get('user_id'),
                field_to_parse,
                payload_before.get(field_to_parse),
                None,
                None,
                ts_ms,
                current_ts,
                op_type
            )
        )

return out_tuples

def parse_update(payload, op_type):
    current_ts = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
    out_tuples = []
    for field_to_parse in FIELDS_TO_PARSE:
        out_tuples.append(
            (
                payload.get('after', {}).get('holding_id'),
                payload.get('after', {}).get('user_id'),
                field_to_parse,
                payload.get('before', {}).get(field_to_parse),
                payload.get('after', {}).get(field_to_parse),
                None,
                payload.get('ts_ms'),
                None,
                current_ts,
                op_type
            )
        )

return out_tuples

def parse_payload(input_raw_json):
    input_json = json.loads(input_raw_json)
    op_type = input_json.get('payload', {}).get('op')
    if op_type == 'c':
        return parse_create(
            input_json.get('payload', {}).get('after', {}),
            op_type
        )
    elif op_type == 'd':
        return parse_delete(
            input_json.get('payload', {}).get('before', {}),
            input_json.get('payload', {}).get('ts_ms', None),
            op_type
        )
    elif op_type == 'u':
        return parse_update(
            input_json.get('payload', {}),
            op_type
        )
    # no need to log read events
    return []

for line in sys.stdin:
    # 1. reads line from unix pipe, assume only valid json come through
    # 2. parse the payload into a format we can use
    # 3. prints out the formatted data as a string to stdout
    # 4. the string is of format
    #    holding_id, user_id, change_field, old_value, new_value, datetime_created, datetime_updated, datetime_deleted, datetime_inserted
    data = parse_payload(line)
    for log in data:
        log_str = ','.join([str(elt) for elt in log])
        print(log_str, flush=True)

Note: Use tmux for easy multi terminal viewing.

docker run -it --rm --name consumer --link zookeeper:zookeeper \
--link kafka:kafka debezium/kafka:1.1 watch-topic \
-a bankserver1.bank.holding | grep --line-buffered '^{' \
| <your-file-path>/stream.py > <your-output-path>/holding_pivot.txt

The above command starts a consumer container, grep to allow only line that start with { denoting a json, the –line-buffered option tells grep to output one line at a time without buffering(the buffer size depends on your os) and then use the python script to transform the message into a format we want and write the output to a file called holding_pivot.txt.

In another terminal do the following

tail -f <your-output-path>/holding_pivot.txt

The above command opens the holding_pivot.txt file and follows along, so you can see if there are any new lines added to the file. In another terminal open a connection to your postgres instance using

pgcli -h localhost -p 5432 -U start_data_engineer
# the password is password

Try the following sql scripts and make sure the output you see in the holding_pivot.txt is accurate

-- C
insert into bank.holding values (1001, 2, 'SP500', 1, now(), now());
insert into bank.holding values (1002, 3, 'SP500', 1, now(), now());

-- U
update bank.holding set holding_quantity = 100 where holding_id=1000;

-- d
delete from bank.holding where user_id = 3;
delete from bank.holding where user_id = 2;

-- c
insert into bank.holding values (1003, 3, 'VTSAX', 100, now(), now());

-- u
update bank.holding set holding_quantity = 10 where holding_id=1003;

Your output should be

1000,1,holding_stock,None,VFIAX,1589121006547486,None,None,2020-05-10-10-31-23,c
1000,1,holding_quantity,None,10,1589121006547486,None,None,2020-05-10-10-31-23,c
1001,2,holding_stock,None,SP500,1589121109691459,None,None,2020-05-10-10-31-50,c
1001,2,holding_quantity,None,1,1589121109691459,None,None,2020-05-10-10-31-50,c
1002,3,holding_stock,None,SP500,1589121109695248,None,None,2020-05-10-10-31-50,c
1002,3,holding_quantity,None,1,1589121109695248,None,None,2020-05-10-10-31-50,c
1000,1,holding_stock,VFIAX,VFIAX,None,1589121112732,None,2020-05-10-10-31-53,u
1000,1,holding_quantity,10,100,None,1589121112732,None,2020-05-10-10-31-53,u
1002,3,holding_stock,SP500,None,None,1589121116301,2020-05-10-10-31-56,d
1002,3,holding_quantity,1,None,None,1589121116301,2020-05-10-10-31-56,d
1001,2,holding_stock,SP500,None,None,1589121116303,2020-05-10-10-31-56,d
1001,2,holding_quantity,1,None,None,1589121116303,2020-05-10-10-31-56,d
1003,3,holding_stock,None,VTSAX,1589121119075024,None,None,2020-05-10-10-31-59,c
1003,3,holding_quantity,None,100,1589121119075024,None,None,2020-05-10-10-31-59,c
1003,3,holding_stock,VTSAX,VTSAX,None,1589121121916,None,2020-05-10-10-32-02,u
1003,3,holding_quantity,100,10,None,1589121121916,None,2020-05-10-10-32-02,u

Data Flow

This represents how we use CDC pattern to capture changes from our application database, transform it into a different format without having an impact on the application performance or memory.

Design Data Flow

‍

Docker commands

# starting a pg instance
docker run -d --name postgres -p 5432:5432 -e POSTGRES_USER=start_data_engineer -e POSTGRES_PASSWORD=password debezium/postgres:12

# bash into postgres instance
docker exec -ti postgres /bin/bash

# zookeeper and kafka broker
docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.1
docker run -d --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.1

# Kafka connect
docker run -d --name connect -p 8083:8083 --link kafka:kafka --link postgres:postgres -e BOOTSTRAP_SERVERS=kafka:9092 -e GROUP_ID=sde_group -e CONFIG_STORAGE_TOPIC=sde_storage_topic -e OFFSET_STORAGE_TOPIC=sde_offset_topic debezium/connect:1.1

# register debezium connector
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{"name": "sde-connector", "config": {"connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres", "database.port": "5432", "database.user": "start_data_engineer", "database.password": "password", "database.dbname" : "start_data_engineer", "database.server.name": "bankserver1", "table.whitelist": "bank.holding"}}'

# sample consumer
docker run -it --rm --name consumer --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.1 watch-topic -a bankserver1.bank.holding --max-messages 1 | grep '^{'

# stream consumer, parse it and write to file
docker run -it --rm --name consumer --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.1 watch-topic -a bankserver1.bank.holding | grep --line-buffered '^{' | <your-file-path>/stream.py > <your-output-path>/holding_pivot.txt

Stop Docker containers

You can stop and remove your docker containers using the following commands

docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

docker stop <your-container-name> # to stop and remove specific container
docker rm <your-container-name>

Things to be aware off

1. We have seen data changed but not data schema changes, depending on your use case this can get extremely tricky.
2. You need postgres version 9.4 or higher.
3. WAL configurations need to be set carefully to prevent overloading the database.
4. As with most distributed systems, there are multiple points of failure, for e.g the database could crash, kafka cluster could crash, your consumer may crash, etc. Always plan for failures and have recovery mechanisms to handle these issues.
5. Both kafka and debezium offers atleast once event delivery guarantee. This is very good, but your consumer or user have to be designed to deal with duplicates. A good approach for this would be to use upsert based data ingestion on the consumer side.

Conclusion

Hope this post gives you an idea of what change data capture is, when you use it, how to implement it using debezium and common pitfalls to be aware of. Let me know if you have any questions in the comment section below.

Change Data Capture with Debezium Kafka and MySQL was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

This explains how to configure Flask, Celery, RabbitMQ, and Redis, together with Docker to build a web service that dynamically uploads the content and loads this content when it is ready to be displayed. We’ll focus mainly on Celery and the services that surround it. Docker is a bit more straightforward.

A Video Explanation

Project Structure

The finished project structure will be as follows:

├── Dockerfile
├── docker-compose.yml
├── README.md
├── app
│ ├── app.py
│ ├── tasks.py
│ └── templates
│ ├── download.html
│ └── index.html
├── scripts
│ ├── run_celery.sh
│ └── run_web.sh
└── requirements.txt

Creating the Flask application 🌶

First, we create a folder for our app. For this example, our folder is called `app`. Within this folder, create an `app.py` file and an empty folder named `templates` where our HTML templates will be stored.

For our app, we first include some basic Flask libraries and create an instance of the app:

from io import BytesIO
from flask import Flask, request
from flask import render_template, make_response
APP = Flask(__name__)

We define three routes for Flask to implement: a landing page, a secondary page that embeds and image, and a route for the image itself. Our image route crops an image dynamically. For this example, it crops an image using `pillow`, and some delays are also included so that the time taken to create the image is more apparent.

@APP.route(‘/’)
def index():
   return render_template(‘index.html’)
@APP.route(‘/image_page’)
def image_page():
   job = tasks.get_data_from_strava.delay()
   return render_template(‘home.html’)
@APP.route('/result.png')
def result():
   '''
   Pull our generated .png binary from redis and return it
   '''
   jobid = request.values.get('jobid')
   if jobid:
      job = tasks.get_job(jobid)
      png_output = job.get()
      png_output="../"+png_output
      return png_output
   else:
      return 404

Next, we need to open our `templates` folder and create the following two templates:

index.html

<div id=”imgpl”><img src=”result.png?{{JOBID}}” /></div>

If we add the following code then run the script, we can load up our webpage and test the image generation.

if __name__ == ‘__main__’:
   APP.run(host=’0.0.0.0')

We see that our page load takes a while to complete because the request to `result.png` doesn’t return until the image generation has completed.

Expanding our web app to use Celery 🥬

In our `app` directory, create the `tasks.py` file that will contain our Celery tasks. We add the necessary Celery includes:

from celery import Celery, current_task
from celery.result import AsyncResult

Assuming that our RabbitMQ service is on a host that we can reference by `rabbit` and our Redis service is on a host referred to by `Redis` we can create an instance of Celery using the following:

REDIS_URL = ‘redis://redis:6379/0’
BROKER_URL = ‘amqp://admin:mypass@rabbit//’CELERY = Celery(‘tasks’,backend=REDIS_URL,broker=BROKER_URL)

We then need to change the default serializer for results. Celery with versions 4.0 and above use JSON as a serializer, which doesn’t support serialization of binary data. We can either switch back to the old default serializer (pickle) or use the newer MessagePack which supports binary data and is very efficient.

Since we’re changing the serializer, we also need to tell Celery to accept the results from a non-default serializer (as well as still accepting those from JSON).

CELERY.conf.accept_content = [‘json’, ‘msgpack’]
CELERY.conf.result_serializer = ‘msgpack’

First, we’ll implement a function that returns jobs given an ID. This allows our app and the Celery tasks to talk to each other:

def get_job(job_id):
 return AsyncResult(job_id, app=CELERY)

Next, we define the asynchronous function and move the image generation code from `app.py` and add the function decorator that allows the method to be queued for execution:

@CELERY.task()
def image_demension(img):
   time.sleep(2)
   im = Image.open(img)
   width, height = im.size
   left = 4
   top = height / 5
   right = 154
   bottom = 3 * height / 5
   # Cropped image of above dimension \
   im1 = im.crop((left, top, right, bottom))
   newsize = (300, 300)
   im1 = im1.resize(newsize)
   width, height = im1.size
   location=os.path.join(‘static/worker-img’,’cropped_img.’+im.format.lower())
    im1.save(os.path.join(‘static/worker-img’,’cropped_img.’+im.format.lower()))
   print(width,height)
   print(“pass”)
   return location

Instead of building a response, we return the binary image which will be stored on Redis. We also update the task at various points with a progress indicator that can be queried from the Flask app.

We add a new route to `app.py` that checks the progress and returns the state as a JSON object so that we can write an ajax function that our client can query before loading the final image when it’s ready.

@APP.route(‘/progress’)
def progress():
   jobid = request.values.get(‘jobid’)
   if jobid:
   job = tasks.get_job(jobid)
   if job.state == ‘PROGRESS’:
   return json.dumps(dict(
      state=job.state,
      progress=job.result[‘current’],
   ))
   elif job.state == ‘SUCCESS’:
   return json.dumps(dict( 
      state=job.state,
      progress=1.0,
   ))
return ‘{}’

Extend our `templates/download.html` with the following Javascript code:

<script src=”//code.jquery.com/jquery-2.1.1.min.js”></script>
<script>
function poll() {
   $.ajax(“{{url_for(‘.progress’, jobid=JOBID)}}”, {
      dataType: “json”
     , success: function(resp) {
     if(resp.progress >= 0.99) {
         $(“#wrapper”).html(‘’);
         $.get(“result.png?jobid={{JOBID}}”, function(data, status){
            end_file=data;  
            $(“#imgpl”).html(‘<img src=’+end_file+’>’);
            console.log(“success”)
          });
      return;
     }
     else {
        setTimeout(poll, 500.0);  
     }
   } 
  });
}
$(function() {
   var JOBID = “{{ JOBID }}”;
   poll();
});
</script>

The `poll` function repeatedly requires the `/progress` route of our web app. When it reports that the image has been generated, it replaces the HTML code within the placeholder with the URL of the image, which is then loaded dynamically from our modified `/result.png` route:

@APP.route(‘/result.png’)
def result():
‘’’
Pull our generated .png and return it
‘’’
   jobid = request.values.get(‘jobid’)
   if jobid:
   job = tasks.get_job(jobid)
   png_output = job.get()
   png_output=”../”+png_output
   return png_output
   else:
   return 404

At this stage, we have a working web app with asynchronous image generation.

Using Docker to package our application 🐳

Our app requires 4 separate containers for each of our services:

- Flask

- Celery

- RabbitMQ 🐇

- Redis

Docker provides prebuilt containers for [RabbitMQ](https://hub.docker.com/_/rabbitmq/) and [Redis](https://hub.docker.com/_/redis/). These both work well, and we’ll use them as is.

For Flask and Celery, we’ll build two identical containers from a simple `Dockerfile`.

Dockerfile

# Pull the latest version of the Python container.
FROM python:latest
# Add the requirements.txt file to the image.
ADD requirements.txt /app/requirements.txt
# Set the working directory to /app/.
WORKDIR /app/
# Install Python dependencies.
RUN pip install -r requirements.txt
# Create an unprivileged user for running our Python code.
RUN adduser — disabled-password — gecos ‘’ app

We pull all of this together with a Docker compose file, `docker-sdfcompose.yml`. While early versions composing one needed to expose ports for each service, we can link the services together using the `links` keyword. The `depends` keyword ensures that all of our services start in the correct order.

docker-compose.yaml

version: '3'
services:
  redis:
    image: redis:latest
    hostname: redis
  rabbit:
    hostname: rabbit
    image: rabbitmq:latest
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
  web:
    build:
      context: .
      dockerfile: Dockerfile
    hostname: web
    command: ./scripts/run_web.sh
    volumes:
      - .:/app
    ports:
      - "5000:5000"
    links:
      - rabbit
      - redis
  worker:
    build:
      context: .
      dockerfile: Dockerfile
    command: ./scripts/run_celery.sh
    volumes:
      - .:/app
    links:
      - rabbit
      - redis
    depends_on:
      - rabbit

To create and run the container, use:

docker-compose builddocker-compose up

One of the major benefits of Docker is that we can run multiple instances of a container if required. To run multiple instances of our Celery consumers, do:

docker-compose scale worker=N

where N is the desired number of backend worker nodes. Visit http://localhost:5000 to view our complete application. 🥳

Github 🧟

alvisf/Dockerized-Flask-Celery-RabbitMQ-Redis

Conclusion 🤝

This is a simple POC on how to use everything together and get the async function up and running.

Dockerized 🐳 Flask Celery RabbitMQ Redis Application was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this story, you will learn how to set up SSH between Jenkins and Github. Using this approach, you do not need to provide your credentials to configure the git repo in your Jenkins job, and you will achieve the password-less connection

Recently other methods have been deprecated for security reasons, and this is the new standard.

Steps

1. Generate the SSH keys
2. Copy the public key in Github
3. Configure in Jenkins Credentials
4. Configure a sample job in Jenkins using SSH connection

Configure Jenkins user

Generate the ssh keys and ensure the ssh-agent is running

ssh-keygen
sudo eval $(sudo ssh-agent -s)
ssh-add ~/.ssh/id_rsa

An SSH key is generated successfully and allows you to achieve the password-less connection between Jenkins and Github

Copy the public key using the below command (or where ever your ‘.ssh’ folder is and copy the key)

cat /home/ubuntu/.ssh/id_rsa.pub

Github Configuration

Go to GitHub repository -> setting -> add deploy keys

Configure Jenkins Credentials

Now we will configure the private key of the jenkins user in the Jenkins configuration

Add SSH Key inside Jenkins

• Login to Jenkins
• Now go to Manage Jenkins from the left panel inside the Jenkins console and then click Manage Credentials:

After this, select ‘Add Credentials’. This will open a new form for us. In the dropdown, select ‘SSH username with private key’ and then give a name for it.

Copy the private key from the Jenkins server.

Now you can clone any git repo in this Jenkins instance. You do not need to provide the credentials while configuring the job in Jenkins.

Configure Jenkins Job

• Create a freestyle job in Jenkins
• Configure SSH URL of your git repo

Conclusion:

We have successfully set up SSH between Jenkins and Github for achieving a passwordless connection. Using this approach, you do not need to provide your credentials to configure the git repo in your Jenkins job.

Setup SSH between Jenkins and Github was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.