With the start of Telescope project there are several areas to work with. My first task was to understand the Scope of the project to figure out the right spots to work with. Last class professor Humphrey discussed his ideas on Telescope. From the discussion I understood the following architecture (the diagram must be discussed to be validated by prof. Humphrey):
After understanding the basic idea of the Telescope structure, I went on studying the environment where the system will be hosted. Telescope will run in a Kubernetes cluster, in my research I followed the instructions from DigitalOcean (https://www.digitalocean.com/community/tutorials/how-to-create-a-kubernetes-cluster-using-kubeadm-on-ubuntu-16-04) and worked in a DO droplet with Ubuntu 16.04 to create a master node which, by the way, was not an easy task. I wrote a bash script to automatize the work and ran into several errors before making something workable.
From the available documents from CDOT, I believe Telescope will be just one new node in a pre-existing CDOT cluster, therefore right now there is not much to be done on this area. To create a new cluster in a pre-existing cluster it is necessary to rewrite the .yml scripts to add the new worker as in the example bellow and update the rest of the scripts to accept the new minion.
1) Read a list of URLs in the format provided by CDOT in (https://wiki.cdot.senecacollege.ca/wiki/Planet_CDOT_Feed_List).
2) From the list, saves the URLs of the RSS feeds in an array;
3) Loops the list of RSS feeds and downloads the blog posts.
The code can be found on my GitHub repository: https://github.com/Cadesh/JS_Attic/blob/master/RSSFeeder/rssFeeder.js
Obviously, I ran into some problems that must be solved:
1) The output did not get only the latest news but everything since the first post, therefore it is necessary to learn how to get only the newest updates to be able to create the module 1 — Listener defined in the Telescope Overview diagram above.
2) The format that comes in is different for each blog, I tested with WordPress, Medium and others. The module 2 — Parser must filter these different formats to a standard JSON format.
3) Each post comes with lots of trash that must be cleaned. As defined in the Telescope overview above, this is the work of module 3 — Process.
This week I started to define the scope of the Telescope project and imagine how to encapsulate the different tasks into micro-services. I also began to understand the technologies we will use to get to a successful outcome.