Forecast several time series at once with prophet and pandas UDF without looping. — Before starting anything to work with pandas-udf the prerequisite are spark ≥ 2.4 pyarrow ≤ 0.14.1 (above this version there’s some issue) then we need to set up an environment variable for pyarrow to 1. (see import code) sudo pip3 install pyarrow=0.14.1 Then we can proceed to libraries import. from…

Collaborative filtering, KNN, Deep learning, transfer learning, Tfidf…ect explore all of these In this article we will review several recommendation algorithms, evaluate through KPI and compare them in real time. We will see in order : a popularity based recommender a content based recommender (Through KNN, TFIDF, Transfert Learning) a…

A full guide from preprocessing to modeling — In this article, we will build a step-by-step demand forecasting project with Pyspark. Here, the list of tasks: Import data Filter data Features engineering (features creation) Imputing data Features engineering (features transformation) Applying a gradient boosted tree regressor Optimise the model with Kfold and GridSearch Method Oneshot I) Import data

Build a CNN classifier and turn it to a recommendation engine. — I have some manga data, I even made an article so that you can collect this data set (with some modifications) see: https://towardsdatascience.com/scrape-multiple-pages-with-scrapy-ea8edfa4318

Calcul, aggregate, transform any data — In this article, we will see how to calculate new variables via joins, window functions, UDFs and vector manipulations. As a reminder, here is the table we use :

A brief guide to import data with Spark — With this article, I will start a series of short tutorials on Pyspark, from data pre-processing to modeling. The first will deal with the import and export of any type of data, CSV , text file, Avro, Json …etc. I work on a virtual machine on google cloud platform data…

A journey through the world of pre-trained models applied to image recommendation — In this article, we will develop a recommendation system based on unstructured data that are images. In order to have a fast, operational model without the laborious work of fine-tuning a Convolutional neural network algorithm, we will use Transfer Learning. Transfer learning is a research problem in machine learning that…

A guide to integrating variable creation into a spark pipeline — In order to have a cleaner and more industrializable code, it may be useful to create a pipeline object that handles feature engineering. suppose we have this type of dataframe: df.show() +----------+-----+ | date|sales| +----------+-----+ |2018-12-22| 17| |2017-01-08| 22| |2015-08-25| 48| |2015-03-12| 150| +----------+-----+