Querying mechanism for python-hydra-agent

A GSoC 2018 story

Hello there! I am Sandeep Chauhan, GSoC student under Python Hydra. So, Today we will discuss about the querying mechanism for our Python client `python-hydra-agent` which is basically based on the hydra-client’s graph as demonstrated here.

So, First of all, we are using Redis as graph database for hydra-client. We store data fetched from Hydra server (hydrus). Once we have the data represented as described in the previous post we can query it from the graph using the OpenCypher querying language. For some complex queries (that will be discussed in a later blog), we are using one of the powerful structure that Redis makes possible to handle: Faceted indexing.

There are some different types of querying is possible as shown below:

    "querying format"
    "Get all endpoints:- show endpoints"
    "Get all class_endpoints:- show classEndpoints"
    "Get all collection_endpoints:- show collectionEndpoints"
    "Get all members of collection_endpoint:-",
        "show <collection_endpoint> members"
    "Get all properties of objects:-",
        "show objects<endpoint_type> properties"
    "Get all properties of any member:-",
        "show object<id_of_member> properties"
    "Get all classes properties:-show class<class_endpoint> properties"
    "Get data with compare properties:-",
        "show <key> <value> and/or <key1> <value1>"
    "Get data by using both opeartions(and,or)",
          " you should use brackets like:-",
          "show model xyz and (name Drone1 or name Drone2)",
          "or, show <key> <value> and (<key> <value> or <key> <value>)"

For first three kinds of query the client simply fetches data from the graph with the help of OpenCypher. For other kinds, the agent first checks that it has data for the requested endpoint or not. If it has no data for that endpoint then it will fetch the data from the server and store it in Redis memory. Example: let’s try to query `DroneCollection` members; for this query, the agent first will check and then it will take the required action. Code for the above method is shown below:

class CollectionmembersQuery:
    ...
    ...
    def data_from_server(self, endpoint):
        ...
        self.collection.load_from_server(endpoint,
                                         self.api_doc,
                                         self.url,
                                         self.connection)
        ...
    def get_members(self, query):
        ...
        get_data = self.connection.execute_command( 
                                     'GRAPH.QUERY',
            ,                        'apidoc'  
                                     """MATCH(p:collection)
                                     WHERE(p.type='{}')
                                     RETURN p.members""".format(
                                     endpoint)) 
        ...
    ...                          

In the above code, `data_from_server()` is used for fetching data from hydrus server while `get_members()` is used for getting directly data from Redis.

Last two are the complex type of queries, for these type of queries, the agent is using a special type of indexing in Redis memory, called Faceted indexing. Other kinds of queries are all simple and using `Opencypher`. Here is how agent leverages faceted indexing. Code for these types of queries is working like:

class CompareProperties:
    ...
    def faceted_key(self, key, value):
    ...
    
    def and_or_query(self, query_list):
    ...
    ...
    
    def object_property_comparison_list(self, query):
    ...
    if query.count(" ") > 1:
        ...
    else:
        ...

In the above code, we have defined a class to handle complex queries. The`faceted_key()` function is used for indexing the properties in a way and `object_property_comparison_list()` is the main function, which can filter using expressions like: “model equals xyz” and “name equals Drone1 and model equals xyz”. Above both are the complex queries but as you can see the second is more complex than the first. Some of them can be nested queries ( they use parenthesis); `object_property_comparison_list` call the function `and_or_query()` which handle these queries types.

So, here we have discussed about the querying mechanism of hydra-client. For more about querying mechanism, you can take look at:

1. Documentation of hydra-client with Redis.
2. Source code for querying mechanism.
3. More Blogs: early querying process, extend querying process.

Other important stuffs related to querying mechanism:

1. Documentation of Redis graph in hydra-client
2. Redisgraph documentation
3. Commands for Redisgraph
4. OpenCypher query language

Thanks! Have a nice day.