Hi all, today in “How to make happy a BI user” chapter, we will make it possible to download ‘big data’ without endangering our own server.

To be able to do this we are going to use our favorite database, PostgreSQL, and nodejs with the help of two packages, pg and pg-copy-streams.

The main problem of downloading or processing all this data is the size of them. So to avoid having to load all the table in memory at the same time, as it could be pretty big, we use nodejs streams.

Node.js Streams have tons of benefits, for example they have a low memory footprint, they are consumed and processed in buffered chunks and they do not block the thread among others. …


In this post I’m going to introduce what PostGIS is, how can use it in PostgreSQL, and how to represent spatial data in a map using QGIS.
In future posts I will explain how to manipulate spatial data to be able to do different kind of geometric analyses.

PostGIS

PostGIS is an extension for PostgreSQL. It adds support for geographic object like points, line-strings, polygons and more.

It also includes a large quantity of spatial functions, such as ‘get the area of a polygon’, ‘calculate the length of a line’ or ‘get the distance between two different objects’ , etc. …


In computer science in general and in data science in particular dealing with continuous values is sometimes tricky. There are plenty of ways to understand a given dataset and many more of extracting the value of it. However, in this entry, we will focus on an application of using discretization and clustering with spatial data. More precisely, we will use the DBSCAN algorithm and the MGRS representation with a small GPS dataset.

But, wait!! DBSCAN and what else ?

Well, let me introduce you some concepts first (skip until the following header otherwise). Discretization and clustering are well-known procedures exported from the data mining field to many others. A formal definition of discretization might be “the reduction of value numbers in a continuous variable by grouping them into intervals” a further explanation may be found at [Web]. Regarding clustering, a formal definition might be “detection of registry groups that share similar properties” more details can be read from [Web]. …

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store