SlicingDice Roadmap — Q2–2017

As we are finally approaching the end of the private-beta phase and (anxiously) preparing for the publicly launch, we believe it’s important to share with our existing and future customers as well as the community what we have being thinking about the SlicingDice’s future.

The first important thing to remember is that SlicingDice still has some restrictions and they are our top priority until completely solved.

We list below the most important features and improvements we expect to have ready or in an advanced stage by the end of Q2 2017 (end of June).

The Q2 2017 roadmap

We divide our roadmap in two main categories:

• Customer facing features, that is everything we add or enhance in SlicingDice that our customers can directly benefit from, such as SQL support.
• Business prosperity features, that are all the other features, business and technical decisions that we make that will impact our customers, but they will not be able to directly see it. Data compression, for example.

Customer facing features

• SQL support — We honestly don’t like SQL. We believe there must be a better way to make questions for a database than SQL, but we also do understand that SQL is the lingua franca in the data world and the easiest way to allow our customers to use their preferred data visualization solutions on top of SlicingDice, so we will implement it.
• Data encryption — Although all the stored data is already encoded in S1Search, making it very hard to be hacked and exposed, we understand that some companies requires full encryption of their data, so we are going to implement it on SlicingDice.
• Automatic data change alert — We will extend the saved queries endpoint to support automatic alerts and notifications based on changes in your data, something similar to the ElasticSearch Watcher.
• Database synchronization — We want to create plugins and synchronization tools for the most used databases like MySQL, PostgreSQL and Oracle, allowing our customers to automatically push and sync their data to SlicingDice, without the need of a custom solution.
• Geo data type — SlicingDice will support insertion and querying for geographical location data, like lat/lon pairs.
• Ordering support — Our team will work on adding support for ordering conditions such as “ORDER BY” on all possible query types.
• Data deletion — We will extend SlicingDice to allow a column or its values to be deleted for a desired Entity ID.
• Data insertion, update and deletion by query — We want to allow data to be inserted, updated or deleted using a query to select the desired subset of entities that will be affected.
• Support new query types — We really want to implement cool queries, such as percentiles, median and others.

Business prosperity features

• Compression, compression and compression—In this quarter we want to double down on data compression on S1Search as this strongly affects the query speed and our ability to lower costs, consequently reducing our prices.

Side note: Our main goal with data compression is that some day data processing and storage costs will be so marginal that we will be able to make the standard SlicingDice offering totally free, looking for other ways to profit. We don’t think we are light-years from this and it’s probably a great topic for a future post.

• Raw device access — S1Search currently uses a mix of techniques to store and retrieve all the data that it needs to be processed during a query. We already have prototypes working to completely bypass all the Linux file system and its cache, writing and reading directly from SSD segments, as other really performant databases do.
  
  Although this is a long and complicated discussion, it’s clear this can bring a huge boost, as it doesn’t make so much sense to read 4 Kb segments from file system if most posting lists loaded on a query are smaller than, say, 500 bytes.
• Improve our query planning and parallelism in order to make our queries even faster.
• Implement probabilistic data structures, such as Bloom Filters, in more places that would allow query computations with less disk accesses, consequently improving querying and insertion performance.
• Adjust our data structure for Machine Learning and review everything we need to change on S1Search in order to run Deep Learning operations within the same server where data is stored. We plan on creating a Machine Learning endpoint on SlicingDice this year.

The roadmap and feature prioritization challenge

As this is our first public post about SlicingDice’s roadmap, we believe it’s important to share what we think and how we handle it internally.

For us roadmap and feature prioritization is a really challenging and tricky thing, because as written by Ben Thompson on his blog post:

One of the ironies of The Innovator’s Dilemma being Silicon Valley’s favorite business book is that Clayton Christensen was writing for a completely different audience than your stereotypical techie. With his theory of disruption Christensen offered a compelling explanation for how it was big company managers, equipped with the best education and aided by the best consultants armed with the latest in best practices, fell prey again and again to upstart competitors with seemingly inferior products; technologists took it as a manual.

What remains so compelling about Christensen’s work is not its analysis of disk drives or steel mills but rather the focus on incentives, and how managers failed precisely because they did what they were supposed to do: meet the needs of their best customers in a way that ensured the viability and continued growth of their business — at least in the short term. The problem came when new technologies that were inferior for current customers’ needs made it possible to serve completely new customers at lower prices; rational managers at incumbent companies dismissed those technologies, allowing new entrants to serve those markets, but given that inferior technologies improve far more quickly than user needs grow, those new entrants eventually threaten incumbents with cheaper technology that is just as good, if not better.

With that in mind, of course it’s still really important for us to receive feedbacks from existing customers about what they want to see on SlicingDice, but at this moment we are more likely to try something completely new, like reinventing SQL with NLP, than simply implementing all SQL statements on the SlicingDice API, just because most of our customers requested it. Probably we will end up doing both.

Of course we try to balance internally these two ways of thinking, but there is a strong internal voice always remembering us that if we were conformed with existing databases we would never be writing this blog post today.

So please, if you have a really crazy idea about how to insert or query data that seems ridiculous and that no one is currently doing, but you would like to see it on SlicingDice, tell us, because we will probably give it a try. We love approaching to problems differently and this is the behavior that actually made us build S1Search and consequently SlicingDice in the first place.

As the saying goes: they did not know it was impossible, so they did it.

Feedbacks and advices

We would love to hear your opinion about this roadmap and also to know if you think there are other things that should come first.

Please, if you have anything to say or to protest, let us know.