Clickhouse has a way to quickly estimate number of duplicates in a table as well as remove duplicates so only unique rows remain. Popular solution here is to GROUP BY columns and insert resulting data into new table. But this can be a challenge if tables are large and disks…

Using windows in Mysql is a way to query data based on a set of rows instead of a single row. This helps in comparative analysis, when we do not only calculate values based on groups but also based on preceding or following rows in a group. Popular examples here…

Any analysis starts with exploratory stage — when we try to understand the big picture of the data itself. We usually do things like geting medians or finding distributions during this step, which Python Numpy/Pandas allows us to easily accomplish. But if we talk about large data volumes (Terabytes+) we…

Any data usually (always?) contain errors. In order to do accurate analysis and build efficient ML models, data needs to be cleansed prior to digging into it. Python Pandas has powerfull tools for data cleansing and further preparation for modeling. Let’s see how that’s done… Test dataset We’re going to use Netflix…

Most data environments usually have 2 main groups of data objects. First — event based objects, usually organized into timeseries tables. Second — instances (with properies/features), usually organized into ID-based tables (and stored in row-based storages). This is what called star (or normalized) schema. This stucture is storage efficient, but…

Clickhouse is efficient enough so most analytical queries will execute fast in many cases without extra optimization activities. But, if you have frequent queries that require a lot of resources for execution, it’s a good idea to help Clickhouse find more efficient ways to execute those. First of all, make…

There’s plenty of cases when we can’t define data structure in advance due to its dynamic nature. Data objects can have different set of features from time to time or depending on other circumstances. …

Matrices stay at the very basis of all math used for ML. Let’s understand why it is so and how matrices can be used to solve systems of linear equations from perspective of 2 different methods. Systems of linear equations in ML Standard task that ML methods solves can be generally illustrated as:

Mysql EXPLAIN statement lets us understand efficiency of our queries and grasp ideas on possible ways to optimize it. Just creating indexes is not sufficient to improve performace, using EXPLAIN is mandatory to understand if indexes are used correctly and efficiently. Long story short, EXPLAIN shows how Mysql executes given…