Engine humming, New milestone names

It’s been a little while, but don’t worry, we’ve been busy at work building a lot of great features and capabilities. But for the fun stuff, we decided to give a little bit character to our milestone names as “Sprint 7” was a bit boring. So, we’ve decided to name our milestones after local (KC) beers, and in the future, we’ll expand it to other alcoholic beverages; the possibilities will be limitless. So without further ado, here’s an update on our previous milestone “Boulevard Wheat”

Our general objective was to start integrating functionality from our previous version into our current system (which primarily focused on IoT Controls). Thus, we wanted to expand it to Analysis of data. Some key milestones were

• Migrated to InfluxDB for our time-series data
• Implemented InfluxDB “driver” in Elixir
• Integrated our real-time monitoring capabilities for Power, Demand and Consumption Heatmap
• Explored different graphing tools, choose React HighCharts over Grafana
• Implemented Schedule capabilities for our thermostats

Tough Choices

Our first big hurdle for the last milestone was how do we handle our growing thermostat data, and the vast amounts of new data we’ll need to support from our real-time monitors. We decided that our rudimentary Postgres model did not fit, and decided to move it to a time-series database. So why time series? There were some key deciding factors

• Built for large time-based data
• Integrated query capabilities to do calculations across large time spans (i.e. max, moving average, sum over a year)
• Distributed data and query model

Now we decided to move to a TSDB, we now had to pick which one. There are several good choices, and we looked at this comparison chart pitting Prometheus against Graphite, InfluxDB and OpenTSDB. Despite that being a list that Prometheus created, it was very objective and helped us decide on the right fit for us. What we learned is, the different databases are architected differently and have unique aspects, so its very important to understand your own use case and decide. We settled on InfluxDB for these key reasons

• Rich metadata support for each event
• Distributed scaling
• Downsampling / Retention Policy

Now, the interesting thing is, part of how we got to pick a time-series database was actually looking at graphing options for our data. We found Grafana and itself only supported a handful of TSDB, one of which was InfluxDB. So, the next part was ensuring that once the data was in InfluxDB, the graphing would meet our needs. So the final decision was to pick a graphing tool. Grafana was the “incumbent” in this case, but it turned out we needed something a bit more flexible. We considered this

• Customization of the interface
• Security considerations
• Energy related graphing (ie electric demand)

This is how we settled on Highcharts (for ReactJS), and have not looked back.

Tank7

With Blvd Wheat successfully completed we moved on to our next milestone “Tank 7”. The general concept is to implement our “Analyze” system, which will aggregate various data points and display insightful information to our clients. We’ve already finished Sprint 1, and we’re on to Sprint 2, and here’s what we’ve finished so far

• Implemented “Web box” dataloader for our Solar Inverters.
• Begin migration of Solar data into our new TSDB
• Hardening our Digi code for more resiliency
• Prototype of Analyze (Utilities Page)
• InVision and Mockups on new Analyze pages to show utility, solar and real-time data together
• Build concept of Permissions and what to display to the customer

Realities of Production

It’s exciting that we’ve finally got customers, and the product is really started to come together. However, we’ve been hit with the realities of developing in a production environment. We’re past the free wheeling days in early April and May where we literally could do anything but now have to take into account how our changes will affect the existing production state and our customers usage. This is when the real fun begins. Anyone can build a prototype to demo and show it to a handful of people, but it takes a great team to maintain that pace of innovation while making sure customers are happy. We’ll leave the details to another blog, but suffice to say, it’s required us to up our game.

Final Thoughts

Its been a crazy month or so, and we’re excited with our progress, and the virtuous cycle of development we’ve created. Challenges continue to pop up, but we’re taking it in stride and adjusting accordingly. We also want to make sure these entries are both informative and impactful, so we’ll be doing these type of “milestone” based updates every month, and then have specific blog entries that address a specific issue we ran into. Just like our code and our processes, this blog is not immune to continuous improvement, and we hope to make it more useful and engaging. In the meantime, happy hacking!