rsp — realtime search platform

For the past 8 months or so, I have been interested in realtime search: e.g tracking tweets in realtime(in a very close proximity of time that events occur), and acting on them. During the last US presidential election (2016), one of my personal side projects one evening was a tool to examine content from election related tweets and act on phrases and media e.g videos, in realtime.

Delivering information in realtime is a complex feat of engineering that I applaud the respective providers e.g Twitter, Google, Facebook for, because the process involves uploading, processing, filtering, indexing, delivering content in order to your social graph, and so much more work than what meets the eye. It means that as soon as a tweet is posted from your device, it is searchable and examinable by parties that have access to your feed. The power to make content available instantly opens up opportunities to build reactive services that adapt to input from users e.g sentiment analysis to determine reactions to events, generate articles about trending topics, inform a company when their brand or product is trending, find the most popular videos and share them immediately, make memes, make T-shirts of trending catchphrases and incidents before your competition, make parodies, buy or sale stocks of a company, target ads etc.

Being a kinesthetic learner (one who learns by doing), I figured that the best way to get into this field is by building some apps that act on information in realtime. Also, having just started a backend/server-side engineering startup orijtech https://orijtech.com/, one of my goals is to provide out-of-the-box software solutions to help companies that want to build their products, but would like to budget allocate their resources, labour, expertise, or the prorities. Think of it as a partnership to provide you with the custom pieces to complete your jigsaw puzzle, without you having to break the bank in order to hire elite engineers, or be stuck in the mud on your idea for a company that could be built entirely with just a front-end, or one off pieces for which you don’t want your engineering team taking off weeks or if not months of work.

In late 2016, during a public holiday extended weekend, while in my apartment in San Francisco, I was bored, everyone was out of town, I had just gotten back from a nice long run, everything was shut down, I had rewatched 2 James Bond movies, but the day was still young, so I decided I to work on a Facebook Messenger bot. The bot is one that I can use to search for content on iTunes and YouTube, and it spits back to me results instantly. That led me down the rabbit hole of working on searchyt https://m.me/searchyt, which I then decided to take another step further in February 2017, and built an equivalent web app, medisa https://medisa.orijtech.com/. For medisa, I built a functioning prototype with a properly working backend (my area of focus) on a Friday night in my school’s library, but the front-end was quite ugly (I understand my limits with UI design), and so to fix the front-end, I contracted the services of a talented friend and former co-worker from 2012. After about 2 iterations on it, I started to see this tiny app fitting into my goal of working on realtime search.

medisa demo

Today I have open sourced the code for medisa at https://github.com/orijtech/medisa and it directly uses the also just open sourced rsp Javascript client rspjs https://github.com/orijtech/rspjs. medisa uses a simple Go backend to setup the UI, by serving a template of the display page as well as setting up the custom initial keywords and also to serving the static files, Javascript, CSS and HTML on the front-end with some jQuery, it is quite vanilla and the entire app is runnable locally for you to test it yourself.

rspjs is a Javascript library for delivering results of instant searches on the front-end, simply by passing in the ID of the search input element and creating a handler to receive the results.

rsp-js usage

or in particular within medisa https://github.com/orijtech/medisa/blob/master/static/main.js#L2-L17

For example this is the search input element for medisa:

The term “realtime search”, while at first it might seem misleading in this article, because I don’t yet have tangibly hard realtime search results, my vision is to build a platform that ingests realtime data from different providers. I currently have instant search, however, I shall add services in a piecemeal fashion, making adapters for the market’s demand. In critique of my current direction, each one of those adapters could be built by your engineering team, and that’s a fair assesment. However, after reading specifications, writing, testing for more than one service and stitching together the frontend and backend you’ll start to see how laborious it is, why not use a simple and unified API that sends you events that you can act upon? Instant search is a baby step, and even in the current state, it delivers a service that many companies currently lack, while bootstrapping and getting for me more requirements for the eventual and full realtime search. A direct application of the instant search is adding gif search, Twitter keyword tracking to your website, without you having to deploy a custom backend; you get instant results for your searches or streams of data for content that you are watching delivered directly to your device not just your backend. It should just be a matter of importing an API client, like I did in medisa’s code. I plan on building clients in many different languages. rsp has and will have simple API clients to help you quickly build your product. rsp should also be useable on mobile, desktop and whatever platform. Realtime Search is one of my long term visions and I am very happy to start these baby steps towards building a realtime search platform.

In conclusion, the goal of rsp is to build a service that ingests content from different providers e.g YouTube, iTunes, Twitter etc and then disperses the results to parties interested, by virtue of a simple API, in realtime. In order to start building the realtime search platform (rsp), I have starting working on a backend for rsp(currently closed source) and I have open sourced rspjs as well medisa that shows the power of rsp. Its goal will be to have different providers e.g Twitter, Facebook, Google Cloud Storage, YouTube, Spotify, gif search, Dropbox, AWS for which you’ll be able to easily select from (already written adapters) or write your own, and then deploy a custom adapter to track events, and then describe the channel that you’d like to receive the results on.

For now, I’ll go catch some sleep, I have been up for way too many hours, having been fired up by the idea of rsp, writing rspjs, then rewriting, refactoring and open sourcing the code for medisa. I hope you enjoyed this piece and please feel free to get in touch if you would like to discuss software solutions or corporate partnerships, or even just to say hi.

Thank you for your time, and for reading this far!

Emmanuel T Odeke