It’s The Perfect Fit
In a previous post we discussed how data modeling always starts with the questions you want to ask of your data. We tried to keep that post clear of anything that could start a technology war, but in this one we’re going to take a very strong stance.
We Love Elasticsearch
The thing is, we have a confession to make. Everyone has their favorite tools and we are no different.
Elasticsearch is our first love and it is a perfect fit for ClearQuery.
Don’t worry — we will spend the rest of this post justifying and defending that position.
On Denormalization & User Experiences
ClearQuery was designed from the beginning with a core goal of answering questions. It does this by providing a more conversational experience to end users trying to interact with their data. We believe that you should not have to learn a new query language just to ask questions. Conversing with your data should be as natural as speaking.
With that said — it becomes very important to the user experience to ensure that answers are provided as quickly as possible. The more instantaneous the better. It feels more natural than writing a query and waiting a while before you can follow up with yet another query. It just works.
In a traditional relational structure, the data required to answer the questions you want you ask of your data could live in separate tables. This means, at query time you could have to wait while the query performs various joins across tables. Sure in most cases this is a few seconds, but those few seconds have an adverse effect on the user experience. It becomes even worse as your user base starts to grow.
Denormalization shifts when and where the work occurs. Rather than at query time, this wait occurs at indexing time (although with Elasticsearch that’s pretty fast too). As you follow up with questions, the machine does not have to keep doing the complicated parts of joining things because the data is available in every record.
In the case of Elasticsearch, this results in low double (sometimes single) digit millisecond response times — it may as well be instant.
The denormalization of data also allows for very flexible schemas which also aids in the modeling of data for new types of questions that arise as you begin to explore your data. As your schemas change over time to be able to ask new questions.
Added a new numeric field? Now you have the ability to ask a large number of metric related questions around that particular data point and how it relates to the rest of your data. New date field? You can now ask questions about how your data changes over time. Geo coordinates? Ask about where important events are occurring.
The Query DSL
While Elasticsearch’s query DSL may not be the easiest thing to write, it is extremely intuitive. The query DSL, like everything else in Elasticsearch is JSON based, which means it is super verbose. The plus side is that almost anyone can read an Elasticsearch query and understand most of what’s going on.
There are some nuances to the DSL, but this structure is key to programmatically piecing together a query when translating from a natural language question into the query that gets run against the index. ClearQuery takes advantage of this structure and the optimal ways to query data so that your knowledge discovery experience is fantastic.
Not Just Search
When people hear the names ClearQuery and Elasticsearch, there is a natural inclination to just think about search. And while both solutions have major search components to them, neither of them is just about search. At their core, they are both designed to help you converse with your data and tell stories with it.
Search capabilities allow you to filter data down to the information that is relevant to the questions you want to ask, but it is the other features that allow you to get more in depth when your data is modeled appropriately.
Aggregations allow you to ask questions like “What was the average number of sales in the North Eastern Region last month?” or “How many visitors did our website receive broken down by country?”. These are the questions that help you derive actionable business intelligence that you can leverage.
Percolation gives you the ability to schedule alerts or automate actions as soon as new data hits your system. “Alert us whenever a website visitor tries to access a restricted section.”
“More Like This” queries are contextually aware and allow you to ask questions like “What customers have similar profiles to this one?”.
And those are just the free things that come out of the box. When you take advantage of X-Pack’s Machine Learning and Graph features you can ask “What are my forecasted sales for the next two quarters?” and “Who is the biggest influencer of our brand on Twitter?”
An Interface To Magic
If you are not convinced by now that Elasticsearch was the perfect fit for ClearQuery, then you haven’t been paying attention. Elasticsearch already gives you the ability to ask these questions, ClearQuery just makes it much easier.
“Any sufficiently advanced technology is indistinguishable from magic”
-Arthur C. Clarke
As far as business leaders are concerned, using ClearQuery is like waving a magic wand to activate the Elasticsearch magic that provides answers to their questions.
And that is why we love Elasticsearch.