In this tutorial, I explain how to develop a Spark application that allows creating a machine learning model for a real-world problem, using real-world data: Predicting the arrival delay of commercial flights.

To find the best model, several machine learning techniques were used and provided with several inputs combinations. The machine learning methods explored are Linear Regression, Random Forest Trees and Gradient-Boosted Trees. All the code will be written in Scala.

The full code for this application and how to run it can be found in the GitHub repo: https://github.com/pedroduartecosta/Spark-PredictFlightDelay

Problem

We want to be able to predict, based on historical…

In this tutorial, we will set up a Spark Machine Learning project with Scala, Spark MLlib and sbt.

sbt is an open-source build tool for Scala and Java projects, similar to Java’s Maven and Ant.

sbt requires the Java Development Kit 8 (JDK 8), so if you don’t have it installed follow this link to install.

Installing sbt:

On Ubuntu:

$ echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823$ sudo apt-get update$ sudo apt-get install sbt

For other Linux distributions: https://www.scala-sbt.org/download.html

On Mac using homebrew:

$ brew install sbt

Creating the project

To quickly…

Serverless is the latest stage of On-Demand Economy applied to software development.

A bit of history on how we got here before explaining what serverless is. In the early days of cloud computing, companies had to set up their infrastructure for their services and applications, including mounting the racks with the servers and configuring all the underlying network and security. Emerging from the necessity of an abstract layer and quicker response from conception to production (TTM — time to market), Infrastructure as a Service was born.

IaaS (Infrastructure as a Service), allowed companies to request the required underlaying infrastructure from…

There were three reasons that lead me to search for a summer internship:

• First of all, I wanted to put in practice what I’ve learned at the university so far, put my skills to the test.
• Secondly, learn new things, new technologies, find out what is actually used in the market to foster the real services that we use every day.
• At last, acquire professional experience, working in a professional environment using development methodologies such as Agile.

So what to do?

Quite easy actually, I just sent e-mails to a few companies, in Porto, asking if they were interested in me working for a…

Because you should. Ok, there’s more to it, but basically they shortly (usually) address issues in a detailed manner that allows you to comprehend the logic behind some subject and that realization is worth the time. Or, they simply give you an overview of a specific topic. Either way, you’ll gain much from it, you just have to choose those that are of your interest.

The list below is intended to not address papers/articles that are too complex or in depth. They are rather a good start!

Please, click here to make a suggestion or comment!

My list

• (Link) Bitcoin: A Peer-to-Peer…

Here lies a list of books I’ve read and some that I intend to read in a near future. Please feel free to contact me with suggestions. I will try, in time, to give a small opinion on each one.

There are some books that I’ve read such as Inferno by Dan Brown and other by some Portuguese authors that I don’t feel like having on this list for now.

Please, click here to make a suggestion or comment!

Books I’ve read

• Harry Potter (All of them)
• The Art of War by Sun Tzu
• The Prince by Niccolò Machiavelli
• How to Win Friends…

In this post, I am going to present you my views on the On-Demand Economy, Economies of Scale and Customer-Centric Economy, by giving real examples of how giants like Amazon, Netflix and Uber got where they are today. You don’t require any technical background to go through this post.

On-Demand Economy

The On-Demand Economy is defined as the economic activity created by technology companies that fulfil consumer demand via the immediate provisioning of goods and services.

— Business Insider

This trend started with software companies like Amazon, who allowed any customer, either small startups or large companies and federal governments like the…

In this tutorial we’re going to build a NodeJS website with the following features: — User authentication — Back office — Blog (optional)

The first to parts of the tutorial are meant to quickly set you up and running. The third part is a more in depth explanation of some points I find important, so feel free to read them later as you need.

We’ll be using KeystoneJS , which is a promising framework, based on ExpressJS, that allows for a quick setup of the previous features.

Prerequisites: — Node.js 0.10+

Initial set-up

Install the generator You’ll be using the KeystoneJS generator…