From packages installation, batch saving to sub-pipelining and logging. May some of these tips make your life easier with kedro.

Image for post
Image for post
Photo by tian kuan on Unsplash

What is kedro and why should we use it?
#1. Design structure with modular pipelines
#2. Use Jupyter notebooks to build up the flow
#3. Use loggings for separate pipelines
#4. Batch saving results
#5. Packages version control
#6. Common functions storage location

This article would assume that you know how to spin off a kedro project on your own, or at least went through the “Spaceship” example on its official page. …


New tools and methodologies show up every day, so how can your analytics team stay vigilant? A thought on the 4 pillars of the industrial revolutions: standardization, modularization, automation, and scalability

Image for post
Image for post
Image by Author

Throughout history, we have gone through three industrial revolutions, and many believe that we are currently experiencing the 4th. From the 1st revolution with coal and steam; the 2nd with mass production, the 3rd with computers to the latest 4th with artificial intelligence and internet-of-things, there are many lessons that humanity learned with sweats and tears, some times even with blood [1]. As “Industry 4.0” …


How to build your own simple demographics atlas using OpenStreetMap & Open Government Data with kedro pipeline

Image for post
Image for post
Building classification pipeline in a nutshell. Image by Author

Part 1 — Data gathering and preparation

Part 2 — Building footprints classification

This series attempts to build an end-to-end data product using kedro pipeline: from gathering data, preprocess to reporting and visualize the results. Although this project focuses on Germany, the methods can be applied to create a simple demographics atlas for other locations in the world.


How to build your own simple demographics atlas using OSM (OpenStreetMap) & OGD (Open Government Data) with kedro pipeline.

Image for post
Image for post
The project result in a nutshell. Image by Author.

Part 1 — Data gathering and preparation

Part 2 — Building footprints classification

This series attempts to build an end-to-end data product using kedro pipeline: from gathering data, preprocess to reporting and visualize the results. Although this project focuses on Germany, the methods can be applied to create a simple demographics atlas for other locations in the world.

In this article, objectives and methodologies are discussed, then the first 2 out of 4 modular pipelines (data_acquisition & data_preparation) will be explained with hands-on codes.

GitHub link to the project (work-in-progress)

Table of contents

1 Introduction 2 Project Overview 3 Kedro initialization (prerequisites) —…


How should organizations build up their automation & analytics task forces while avoiding “siloization” traps?

Image for post
Image for post
Photo by Lenin Estrada on Unsplash

In this article, the 1st part is my attempt to clarify the misunderstandings between RPA & AI. Then, I introduce a technology adoption map for organizations that are embarking on this “smart automation” journey. Part 3 is my proposed collaboration framework for the two departments RPA & Data Science. Finally, part 4 ends the article by providing collaborative use cases where both parties can be utilized to their true potentials.

Table of Contents

1. From RPA to Data Science (?) 2. RPA & Data Science adoption map 3…


When can we resume our normal lives (Germany-scenario)?

Image for post
Image for post
Image by kalhh from Pixabay

TL;DR

I simulated 5 scenarios using RKI’s benchmark (reach 10 million active cases by mid-June 2020 if there is no social distancing implemented). Scroll to part 5 if you want to just see the results.

All the codes are published in my GitHub here

Table of Contents

0. Disclaimer 1. Introduction 2. Overview 3. Data Input — 3.1 MIDAS research networks — 3.2 JHU and RKI repositories — 3.3 Social demographics data 4. Models — 4.1 SIR — 4.2 SEIR — 4.3 SEIR + DH 5. Simulation — 5.1 No social distancing — 5.2 10% social distancing — 5.3 25%…

Trung Nguyen

Data Scientist @ MSD | Curiousity doesn’t always kill the cat

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store