Introduction This article discusses common patterns in Infrastructure as Code(IaC) to summarize and highlight the design principle underlying good architecture. Terraform is a well-known IaC tool from Hashicorp. Terragrunt is a thin wrapper of terraform that provides extra tools for keeping your configurations DRY, working with multiple Terraform modules, and managing…

Introduction This article studies basic principles of migrating terraform tfstate. It can happen due to big infrastructure code refactoring. The essence of migration is to keep alignment between terraform script and its corresponding tfstate (i.e. in aspects of resource addressing, configurtion etc) between tfstate and reality in cloud between source tfstate…

Introduction 工欲善其事， 必先利其器 There is an old chinese saying, “To do a good job, an artisan needs the best tools”. My toolbox to boost development productivity is Visual Studio Code + Docker. Using Visual Studio Code + remote container facilitates the provision of full-featured IDE with docker containers and installation…

Introduction This article describes a systematic way of applying terraform at scale. At scale refers to: high technical complexity: deploy infrastructure to any number of accounts and cloud providers high organizational complexity: enable multiple teams of developers to work collaboratively The requirement calls for a modularized and hierarchical architecture as well…

Forward Recently inspired by talks from Dmitry Petrov about Machine learning model and dataset versioning practices and Data versioning in machine learning projects. In the majority of the current ML systems, there is a lack of efficient and systematic ways to deliver the value of data through Data Science into the…

Gitlab CI/CD pipelines are configured using a yaml file called .gitlab-ci.yml within each project. yaml is a feature-rich data serialization language, if combining it with gitlab CI/CD features we can keep CI/CD pipelines DRY — an important criteria of deployment pipeline as code. Introduction Gitlab CI/CD pipelines are composed of jobs. …

Machine Learning operations (let’s call it mlOps under the current buzzword pattern xxOps) are quite different from traditional software development operations (devOps). One of the reasons is that ML experiments demand large dataset and model artifact besides code (small plain file). This post presents a solution to version control machine…

Twenty years domination of Microsoft Access , nowadays more and more companys choose to migrate their services to cloud for lower cost and higher elasticity. This blog tells a story of migrating a MS Access Miniapp to aws serverless stack. This MiniApp has three parts:

This article aims to demonstrate the capability and agility of AWS to develop and host both industry-standard machine learning products and research-level algorithms. The concept is validated with an implementation of CRISP Data Science Workflow on AWS cloud using “object detection algorithm” example (https://github.com/wutianchen/object-detection-algo). Separation of Concerns in Data Science CRISP (Cross-industry standard process for data…