ZoneMinder is a popular open-source video surveillance system. AWS Rekognition is an easy to use image recognition service from Amazon. Together they are a perfect match.

ZoneMinder is a remarkably versatile system. It’s compatible with pretty much any IP camera that you can buy, you can run it on your own server and therefore you don’t have to trust your security video feeds to some random vendor with dubious security standards. Being open-source there are no monthly “subscription fees” nor any expensive optional “power packs”. It’s all free.

ZoneMinder of course supports many advanced features like motion detection or image…


Data Engineering Testing Series

Welcome to Part 4 of the Data Engineering Testing Series. So far, this series has mostly been concerned with how do we ensure that our data transformations are correct. The solution we came up with is one that is borrowed from software engineering, and relies on data generation with code, a concept quite foreign to a lot of professionals I’ve come across.

The reason I suspect it is so uncommon is because the idea of having to mimic changes in data with code can be ludicrous. First I have to understand what the changes are, then I got to change…


Raising notifications in Snowflake

The use of Snowflake has evolved into much more than just a data warehouse, rather, the data cloud, a service offering for data solutions. Data pipelines are being built every day using the likes of Tasks and Procedures, but when you think about it, whenever your data pipeline fails, it may be critical to know when it fails and how it fails, as opposed to manually monitoring it.

As a result, one of the most requested capabilities is to be able to raise a notification whenever the data pipeline fails.

Unfortunately, there isn’t any straightforward off-the-shelf feature that Snowflake provides…


DATA ENGINEERING

Notes on Migrating an existing Data Warehouse from Aurora PostgreSQL to Redshift

Background

Once upon a time, I joined a beautiful, chaotic startup where I was handed over the keys to unlock the potential of data spread out in many places, but mainly on a couple of MySQL instances hosted on EC2 with a multi-source, multi-channel replication with a mix of Tungsten and native MySQL replication. All data was landing on a single MySQL instance. It was getting data from several databases (multi-source) for a web application, a mobile application, sales and marketing campaign websites, product experiments, event aggregators, and many others. …


Suppose you were to analyze the top 10 common jobs encountered when onboarding businesses to the cloud; it’s more than likely that moving data A to B will be one of the most frequently encountered. The datasets of multiple structures and sources are hiding everywhere in the dark, waiting to be extracted to the warehouse and put to work. The good thing is that we have plenty of options — from native cloud services to 3rd party commercial solutions.

But it’s still not simple. At Servian, we need to deal with the data ingestion of many different 3rd party systems…


Introduction

In this article, I discuss the disparity between what outcomes businesses expect from AI/ML and how businesses expect to leverage AI/ML, and how the technology and data analytics team see the advantages and use of AI/ML.

The key takeaways come down to:

  • Communications and understanding
  • The value businesses are after
  • There are some good ways to help solve these challenges

When discussing artificial intelligence I think the business has 3 major categories of issues stopping them from getting the value and outcomes from AI/ML:

  1. A lack of understanding between what the business wants and needs and what the technology…


Setting the Context

We are all too familiar that in today’s business setting, traditional machine learning and artificial intelligence techniques such as deep learning using neural networks are applied to solve prevalent use-cases in prediction, classification and anomaly detection problems. In this article, I am going to take an in-depth assessment view on framing machine learning pipelines as a panacea towards striving for operational efficiencies in business processes. An overwhelming hype has amplified the noise for a majority of organisations rushing to place this transformational technology on their roadmap for competitive advantage in the fear of missing out (FOMO syndrome) on this tech…


unit testing in dbt

and how to go about it

In my previous article How to set up a unit testing framework for your dbt projects, I addressed unit testing for relatively simple models inside of dbt. dbt has the concept of materialisations, where you can choose how your model physicalizes the data once it is run. According to dbt,

When using the table materialization, your model is rebuilt as a table on each run, via a create table as statement.

This is perhaps more familiar as a truncate-and-load strategy. However, constant rebuilds can be unwieldy and slow when dealing with large amounts of event-based data.

Incremental models are tables…


or How to not get swept away in the cool new tech

What’s more difficult than learning a new technology without context? Not much, I’d say; and since I’m writing this article I’d say my opinion on the things I say is something of relevance. Furthermore, as a consultant, the things I say are intimately linked to the things I do. Not only do I need to be able to walk the walk when I talk the talk; I need to be able to talk the talk to sell the walk for which I’ve toiled so hard to be capable of walking. …


Software Development Automation

Speeding up developers, reducing downtime, and lowering project costs with CI/CD

Continuous integration & continuous delivery (CICD) is one of the most important aspects of a modern software development project. Without the confidence that a mature, modern CICD suite provides, developers are left blindly hoping that their code holds up as they march towards production.

Hope is not a strategy.

Old car production line

In a recent project, my team was tasked with a brand new, brown-field cloud uplift project - a fresh start for an existing system. We designed a rough, high-level design, chose our development language and tooling, and planned for the first couple of iterations of the system. …

Servian

The Cloud & Data Professionals

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store