The CDOT Telescope Project

André Rosa
Nov 3, 2019 · 3 min read

TELESCOPE OVERVIEW

With the start of Telescope project there are several areas to work with. My first task was to understand the Scope of the project to figure out the right spots to work with. Last class professor Humphrey discussed his ideas on Telescope. From the discussion I understood the following architecture (the diagram must be discussed to be validated by prof. Humphrey):

A proposed overview of CDOT-Telescope

KUBERNETES

After understanding the basic idea of the Telescope structure, I went on studying the environment where the system will be hosted. Telescope will run in a Kubernetes cluster, in my research I followed the instructions from DigitalOcean (https://www.digitalocean.com/community/tutorials/how-to-create-a-kubernetes-cluster-using-kubeadm-on-ubuntu-16-04) and worked in a DO droplet with Ubuntu 16.04 to create a master node which, by the way, was not an easy task. I wrote a bash script to automatize the work and ran into several errors before making something workable.

From the available documents from CDOT, I believe Telescope will be just one new node in a pre-existing CDOT cluster, therefore right now there is not much to be done on this area. To create a new cluster in a pre-existing cluster it is necessary to rewrite the .yml scripts to add the new worker as in the example bellow and update the rest of the scripts to accept the new minion.

Tests to install Kubernetes

RSS FEEDER

The second problem I wanted to understand was how to access an RSS feed. I went after some examples on the internet and developed a short JavaScript code that:

1) Read a list of URLs in the format provided by CDOT in (https://wiki.cdot.senecacollege.ca/wiki/Planet_CDOT_Feed_List).

2) From the list, saves the URLs of the RSS feeds in an array;

3) Loops the list of RSS feeds and downloads the blog posts.

Coding to get a RSS feed

The code can be found on my GitHub repository: https://github.com/Cadesh/JS_Attic/blob/master/RSSFeeder/rssFeeder.js

Obviously, I ran into some problems that must be solved:

1) The output did not get only the latest news but everything since the first post, therefore it is necessary to learn how to get only the newest updates to be able to create the module 1 — Listener defined in the Telescope Overview diagram above.

2) The format that comes in is different for each blog, I tested with WordPress, Medium and others. The module 2 — Parser must filter these different formats to a standard JSON format.

3) Each post comes with lots of trash that must be cleaned. As defined in the Telescope overview above, this is the work of module 3 — Process.

CONCLUSION

This week I started to define the scope of the Telescope project and imagine how to encapsulate the different tasks into micro-services. I also began to understand the technologies we will use to get to a successful outcome.

André Rosa

Written by

Computer Programmer

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade