At Geospark Analytics we built a SaaS platform for Global Risk called Hyperion. Our Hyperion Platform was built using an API First Strategy. As we build our web based user interface (Vue if you are interested) and our Native iOS and Android mobile applications our API evolves.

Our API is microservices based and primarily runs on the AWS Lambda runtime engine (mostly node and python) and some Google Cloud Functions. Early in our development API Gateway, which is the AWS way of exposing lambda functions to the web, was very nascent and not easy to use. Tooling such as SAM

At GeoSpark Analytics we work with a lot of data. Daily we are processing millions of news articles, running sentiment analysis on social media posts, and translating RSS feeds along with various other types of specialized data then running our analytics and machine learning models on the enriched data. Recently we have been doing a lot of this orchestration using AWS Step Functions and more specifically the State Machines. State Machines are an awesome little tool that allows you to execute AWS Lambda’s (and now using Step Function Activities you can use just about any AWS service), wait for results…

UPDATE (20181129) - re:Invent 2018 Step Functions now support Glue Jobs directly, no need to build a lambda function to start a Glue Job.

Recently AWS made major changes to their ETL (Extract, Transform, Load) offerings, many were introduced at re:Invent 2017. After re:Invent I started using them at GeoSpark Analytics to build up our S3 based data lake. One such change is migrating Amazon Athena schemas to AWS Glue schemas. Athena is an AWS serverless database offering that can be used to query data stored in S3 using SQL syntax. Glue can be used to crawl existing data hosted in S3 and suggest Athena schemas that can then be further refined. Any developer that has spent time working with data knows that it…

Chad Dalton

Chief Technology Officer at GeoSpark.io.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store