Yesterday May 21, here in southern Ontario, there was a severe thunderstorm warning issued by Environment Canada via cell phone asking taking cover immediately. Sadly 7 people are reported dead as of this afternoon. Didn’t find severe weather dataset in Environment Canada after a quick search, in stead, we’ve gotten…

A/B testing is to compare a measurement difference between two groups to find if they are statistical significant. For example, 1 Yield of two barley fields — one uses fertilizer A and the other uses fertilizer B; 2 Time of two groups of runners — one wears pants and the…

Air quality dataset built in R is a daily air quality measurements in New York from May to September. Quality values include: Ozone Solar.R Wind Temp We’re plotting Ozone against the other three measures to understand the basic relations among them. It looks like as follows: This sample webapp can…

We have built some bigram and trigram data model/data frame in this post. So we can further predict next word based on the input. This simple webapp is hosted by Shiny framework. Front end: library(shiny) # Define UI for application shinyUI(fluidPage( # Application title titlePanel("Words Prediction!")…

This is actually a very small project back to a while ago…Dataset can be found here which includes blogs, news and twitter as text files. It’s about 548MB. Using RStudio, we’ll download dataset and generate two word clouds. 1 Download and extract data setwd("~/wordcloud") file.url <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip" if(!file.exists("./Coursera-SwiftKey.zip")) { download.file(file.url, destfile="./Coursera-SwiftKey.zip", method="curl") } if(!file.exists("./final")) {…

This error is seen when tornado tries to bind IP/port when Jupyter Enterprise Gateway starts. jupyter-enterprisegateway --ip=hostname.domain.com It’s complaining about something wrong with the IP address instead of the port. What is the “requested address” JEG wants to bind? We’re using strace to look at details. As we know strace…

Jupyter Notebook server and enterprise gateway server use ZeroMQ (ZMQ) to communicate with backend kernels. (See another post for a diagram of “ports and communication” related to notebook kernel.) When starting a kernel a connection file is generated and used by Notebook server to start kernel process. A sample connection…

When running into following error using Spark sql it seems Spark function “createOrReplaceTempView” is not able to associate the “temp view or table” to the default database with or without Hive enabled. yelp_business_df_from_parquet_partitioned_by_postal_code.createOrReplaceTempView("yelp_business_parquet_table")

In a prior post, we used Spark and looked at partitioning and shuffling behaviors in brief. Now we’re using Dask to do the similar thing. Again, this is not about data set feature engineering or visualization stuff, we’re trying to look at what’s going on from platform level. A very…