Airflow is a scalable, dynamic, extensible, and elegant platform that allows you to author workflows as Directed Acyclic Graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies.
In this article, I provide a detailed minimal-ready configuration for Kubernetes. It is scalable, secured, and automated with CI/CD.
Before starting, I strongly recommend that you read this article:
It’s about a true way of how to use Airflow within the Kubernetes.
Kubernetes setup
Kubernetes should be ready to work with the Airflow.
First, I use Helm to deploy public charts into my Kubernetes…
What if I say, that you can just draw your cloud infrastructure in the web interface and get a terraform code? Don’t you believe me? Anton Babenko already did that in the modules.tf project, just build your infra diagram on cloudcraft and get your terraform code. Sounds pretty nice, huh?
But what, if you already have some infrastructure and want to migrate it to the code? Or another case: you always created Datadog monitors manually, and now, you want to manage them as a code. There is a tool that could help you with that. …
In this article, I’ll share our experience with postmortems at Preply.
Here’s an example of one of our latest incidents with DNS on production in the form of a postmortem. The article could be helpful for those who want to know more about postmortems or want to prevent DNS issues in the future.
My name is Amet Umerov and I’m a DevOps Engineer at Preply.com. Let’s get started!
A little bit about postmortem and processes at Preply
A postmortem describes a production outage or paging event including a timeline, description of user impact, root cause, action items, and lessons learned.
In our company, we are trying to use ‘Everything as a Code’ paradigm which is about having reproducible infrastructure, monitoring, jobs, etc. But in this article, I’ll show you how we use it for Jenkins. Yes, I mean the fully reproducible configuration for Jenkins, with infrastructure, plugins, credentials, jobs and many other things in the code. Besides, in this article you will find answers to the following questions:
- Has our Jenkins become more stable?
- Could we frequently change Jenkins’s and job configuration?
- Is updating Jenkins or plugins not a pain for us anymore?
- Have we managed all our changes in…
If you run your Kubernetes workloads, probably you use self-managed clusters managed by kops (supports k8s from version 1.4) or Amazon-managed Elastic Kubernetes Service (EKS) which generally available in the US since Jun 5, 2018.
In this article, I’ll describe the differences which I found while using them.
Pricing
In both cases, you use EC2 instances for running workloads (workers, load balancers, VPCs, etc), so AWS doesn’t charge you additionally for nodes. The only difference in pricing between kops and EKS — masters.
The master node in EKS calls Control Plane, it’s a fixed price of $0.2/hour ($144/month).
With kops, you…
I’ll describe basic things about how I generate and store my passwords and how to easily transfer credentials inside the company on a simple example.
How do I generate and store my personal or corporate passwords
I used to have one strong password many years ago for all my services (email, social network, and few sites), I didn’t have a persistent internet in my life, so I didn’t think about it, because my password was strong.
In May 2019, GitHub announced the release of the Package Registry service. After that, in August, the support for CI/CD in Actions was announced.
In this article, I’ll tell you about these services and will show how can we use it on the example of a small pet project on GitHub.
What’s that?
GitHub Actions is a platform that allows you to manage the software life cycle, for GitHub-based source code. In fact, this is a new competitor for TravisCI, CircleCI, and many other free CI/CD platforms.
GitHub Package Registry is the central repository of artifacts. …
In this article, I’ll share our experiences migrating the Preply platform to Kubernetes, how and why did we did it, the difficulties we’ve faced, and the benefits we’ve seen since the migration.
My name is Amet Umerov and I’m a DevOps Engineer at Preply.com. Let’s get started!
Kubernetes for Kubernetes? No, for business requirements!
There’s a lot of hype around Kubernetes. While many people say it will solve all your problems, there’s much discussion about why you shouldn’t use Kubernetes: some say you should avoid it because it’s not a silver bullet.
In this article, I’ll shortly describe how to get an SSL certificate with HTTP01 validation and a wildcard certificate with DNS01 validation on AWS example.
So we already have some ingress and HELM for our k8s cluster, and we want to get some certs for domain dummy.example.com.
Let’s install cert-manager using HELM:
helm install --namespace kube-system -n cert-manager stable/cert-managerIf you prefer to use the latest chart version for cert-manager you can follow the instructions here.
For issuing some certificates we need to have at least one Issuer or ClusterIssuer. …
In this article, I’ll explain how to send logs to Amazon and why.
What do we have?
We have a production Kubernetes cluster managed by kops in AWS.
Also, we want to collect logs from this cluster, especially from Nginx Ingress to Elasticsearch. It sounds pretty easy, so let’s start.
Simply, when you have NAT gateway IPs you can easily add them to whitelist and get connection to Elasticsearch. But what if you don’t have these NAT gateways?
So we have a simple plan for sending logs to Elasticsearch.
On every k8s node, Fluentd pod collects logs and pushes it into the Elasticsearch domain.
Which Elasticsearch should I use?
…
Amet Umerov
DevOps Engineer — https://www.linkedin.com/in/amet13/
