Now let’s have some fun with code! In this part I’m using Python3 to automate the creation of our saved searches, visualizations and dashboards.

tl;dr: here’s the code to make this happen:

I’m intentionally leaving in debugging code here ( warts and all! ) in case someone needs to learn with me. It’s important to me that people see that this was a work in progress.

Here’s the workflow for what goes on here:

  • Create the search saved_objects. Saved objects are things used by Kibana in dashboards and search results.
  • Create a different type of saved_object called a visualization. …

This is a two-part series on how we created an automated, centralized logging system at Renovo using BanzaiCloud’s FluentD operator. Our layout for domains and deployments ends up being quite a bit more complex than the average installation.

We currently have 5 Renovo domains ( like prod, demo, etc… ) 1 developer domain, and a collection of customer domains that get increasingly more complex the closer they get to the 5g / WaveLength deployments.

Here I will lay out our journey to centralized logging in two chapters:

  • Flows: this will include how we setup ES + FluentD on k8s and how some of the complicated parts work. Plus some logging data samples and configuration items that are not easy to figure out. …

Image for post
Image for post

Let’s have some fun! First, the user story: As a developer who needs to test developing software, I need to create a way for developers to automatically build and deploy a new version of their service (usually a Docker file image) into an integration environment.

We’re going to do this by using:

  • GitLab CI/CD pipelines which execute…
  • A AWS::StepFunction state engine, which calls
  • AWS::Lambda functions which are
  • Written in Python3.8, which uses
  • Jinja2 for templating values files which are then sent to
  • Rancher which run helm commands based on the instructions sent in as input

When that’s all done, our developer creates a MergeRequest which fires off the pipeline, which deploys the new image into dev. …

Yet another adventure in automation land

Image for post
Image for post

In this article we’re going to be using these tools:

  • AWS — EC2 + ASG and some other things
  • AWS EKS for our k8s provider
  • CloudFormation for managing secrets
  • Terraform for managing clusters
  • Python as our “glue” language, plus some jinja2 and boto stuff
  • Rancher as a sort of middleware layer to help us coordinate everything

User Story

Let’s say we have a customer called “SeaView.” SeaView is a company which produces vehicles of all sorts from trucks, to trains and even consumer vehicles like mini-vans. SeaView has an IoT device in each vehicle that sends telemetry to the cloud:

  • Blob data from several cameras ( in the case of trains, potentially dozens of cameras…

Belting out edge devices with automation for fun and profit

I wanted to do something fun to really show off the potential of k3s and Rancher. In most cases these days, most people, most of the time run a single k8s cluster for production. Most of these clusters will have some kind of capacity trigger to help the cluster scale up and down based on some kind of metric. Sometimes it’s the CPU usage, sometimes it’s the number of requests hitting the front end load balancer.

In the case I’m presenting here, I’m not really not dealing with scale so much as remote nodes running on the edge, which can present its own set of challenges. Let’s say I’m taking advantage of the new hotness like AWS Wavelength or AWS Outpost. We have the ability to shove compute nodes closer to the edge, then run workloads on those compute nodes. Chick-fil-A did something similar to this by pushing k8 clusters down to their POS rigs in each location. …

Image for post
Image for post

These days writing software has become a complicated endeavor. When I first started in the operations field I used vim for almost everything I did, including php stuff ( way back in the day ). Now I use RubyMine and PyCharm to get things done.

We have wonderful tools to help us make writing software better and easier. For example, autocomplete can help save a ton of time by offloading the list of classes on any object to the editor and out of your head.

Image for post
Image for post

I’m currently on a working remote experiment. I’m writing this article from Sioux Falls, SD. I don’t go home very often, so when I’m walking around on my morning rucks playing Ingress and Pokemon Go I’m often catching myself admiring all of the new neighborhoods that have sprung up around the area.

Apparently, despite the cold, Sioux Falls is growing at a fairly good pace. I’m always thinking about automation and tooling, so looking at all the new houses was no different. I started wondering how we went from building small, inefficient houses to building large blocks of houses that are capable of sustaining through harsh winters. What tools had to evolve to make this more possible? …

Configuration management and validation with the push model.

One thing I love about the Devops world is the general availability of such a wide range of tooling. This article isn’t for the novice user of either tool, I’ll leave the introductions for either tool to better minds than myself.

This document is more about describing and important workflow in the Infrastructure as Code space. Specifically our “top of the pyramid” testing.

Image for post
Image for post

This project was aimed at hitting the E2E slice of the pyramid. We’re aiming at hitting two goals with this project:

  1. Verify that the Ansible run has done everything that we think it should have done.
  2. Provide us with a tool that we can use as a “is this thing running the way it should…

Taking a little break from the technical stuff to talk about something that is equally as important. In this article I’m going to talk about a specific soft skill that comes up in the world of IT, or really, any situation involving humans.

I’ve relayed information to a group of people, but once again they are asking about the information as if they’ve completely forgotten everything about it. Despite my diligence in creating wiki pages and Jira tickets, I am once again confronted with repeating myself on a topic.

This article is an attempt to uncover the reasons why this happens and maybe inspire some ideas in how to handle this. I’ve found this to be a frustrating situation at times because I feel like my information should be important to other people. I know I’ve worked with other people who have been frustrated with me for not remembering things as well, so I try to keep that in mind. Ultimately we’re all doing this to each other. …

Continuing on my story line of documenting with code, I have found a neat way to visualize docker-compose files.

Using this docker-compose file…

I generate this image

Image for post
Image for post

The red dashed line indicates a dependency mapping. In this case, the grafana and chronograf services are depending on the influxdb service.

This is the code base.

But why?

Isn’t docker-compose mainly designed for developing applications? Shouldn’t we be using something like kube in production?

Absolutely. However, there is a use case where this can be handy. The use case is basically around IoT devices. …



Building fun things at EDOS

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store