Advanced Filters In JavaScript Apps — Part 2

So I made a case for contextual queries that can also be combined to intelligently filter data in javascript apps.

All of this was born from needing to build this same feature in an Angular app I was working on. So I spent some time last week, #werking, and the result experiment is here.

What Is This About?

Having lots of data, say about developers, structured thus

[ {
    "name": "Dara Dunn",
    "dob": "Oct 12, 1988",
    "city": "North Vancouver",
    "country": "Chile",
    "company": "Dolor Donec Fringilla Associates",
    "tags": "GDE, Game"
}, ... ];

and able to type, say, “1998, Q4, #GDE” to only surface entries for GDEs (Google Developer Experts) born in the fourth quarter of 1988.

How I Approached A Solution

1. Custom Query Handlers — The app needs to understand “Q4” and “#GDE” as valid queries, it then needs to map such a query to a particular function (handler) that knows exactly how to translate “Q4” into returning a developer only if he was born in Q4.
2. Choose a delimiter and handle multiple queries — The app needs to know that the user entered more than one query. Without much thought, I elected to use a comma and thus have multiple queries in the form of “Q4, #GDE”.
3. Only apply the last query, but on the result of any previous one — At any point in time, we are only applying one query to filter the data and this will be the last query the user entered. We are also applying the queries in a “recursive” manner of some sorts, such that when you type “Q4” and later append “, #GDE”, we would have filtered for “Q4” so we proceed by filtering the resultant data for “#GDE”.
4. Have a generic handler — This handles regular filtering like how the default Angular filtering does it for fields in your data model.

The Query Handlers

An array of objects, each with a matcher and a handler. The matcher is a RegExp that has to match your query for the handler function to be “designated” to filter data. So when the query is Q4 as any other one for that matter, we iterate over the query handlers and if the matcher RegExp of any one “passes”, then we have ourselves a handler for the query. When there is no match, we can always fall back the generic hander in (4) above.

.controller("AdvancedFilters", 
             function($scope, $filter,    dataFactory){
        
    var DF = dataFactory;
    // used to fetch data,
    // say, across the network
    DF.loadDevelopers(function(devs){
        $scope.developers  = devs;
        $scope.allDevsLen = devs.length;
        $scope.matchedDevsLen = $scope.allDevsLen;
    });
    $scope.trimDevelopers = function(){
        var q = $scope.search;
        $scope.matchedDevsLen = $scope.allDevsLen;
        if(q){
            q = q.toLowerCase();
            var queryParts = q.split(/,\s*/);
            var query = queryParts[ queryParts.length-1 ];
            // proceed if query is not a blank string
            if(/^\s*$/.test(query) == false && query.length >= 2){
                var queryHandler = null;
                filterHandlers.forEach(function(fH){
                    if( fH.matcher.test( query )  == true){
                        queryHandler = fH.handler;
                    }
                });
                if(queryHandler == null){
                    console.warn("No custom handler for " 
                        + query + ". Attempting generic handler");
                    if(genericHandler.matcher.test(query) == true){
                        queryHandler = genericHandler.handler;
                    }else{
                        // no handler was found for the query
                        // query is therefore invalid,
                        // un-supported or yet-to-be-supported.
                        // You might want to provide a visual
                        // que to the user
                        console.error("No handler for " + query);
                            return;
                    }
                }
                // if last query returned no results
                if($scope.developers.length == 0){ 
                    $scope.developers = DF.withDevelopers();
                }
                $scope.developers = $scope.developers
                    .filter(function(dev){
                        return queryHandler(query, dev);
                     });
                $scope.matchedDevsLen = $scope.developers.length;
            }
        }else{
            $scope.developers = DF.withDevelopers();
        }
    };
    $scope.getDobMonthForDev = function(dev){
        var monthIndex = ( new Date(dev.dob) ).getMonth();
        return DF.months[ monthIndex ];
    };
    var filterHandlers = [{
        // matches 2016
        // handler returns true if the developer
        // was born in the year specified by query
        matcher: new RegExp("^\\d{4}$", "i"),
        handler: function(query, dev){
            var year = ( new Date(dev.dob) ).getFullYear();
            return parseInt(year) == parseInt(query);
        }
    }, {
        // matches Q1, Q3, Q3, and Q4.
        // which represents the quarters in a year
        // handler returns true if the developer was
        // born in the quarter specifield by query
        matcher: new RegExp("^q[1-4]$", "i"),
        handler: function(query, dev){
            var month =  $scope.getDobMonthForDev(dev);
            return DF.quarterly[ query ]
                && DF.quarterly[ query ].indexOf(month) != -1;
        }
    }, {
        // matches H1or H2, which
        // represents the 2 halves in a year.
        // handler returns true if the developer was
        // born in the half of the year specifield by query
        matcher: new RegExp("^h[1-2]$", "i"),
        handler: function(query, dev){
            query = query.toLowerCase();
            var month =  $scope.getDobMonthForDev(dev);
            return DF.yearInTwo[ query ]
                && DF.yearInTwo[ query ].indexOf(month) != -1;
        }
    }, {
        // matches #Firebase
        // handler returns true if the developer's tags
        // contains the tag specified by query
        matcher: new RegExp("^#[a-zA-Z0-9._]+$", "i"),
        handler: function(query, dev){
            query = query.substring(query.indexOf("#")+1)
                      .toLowerCase();
            var tags = dev.tags.toLowerCase();
            return tags.indexOf(query) != -1;
        }
    }];
    var genericHandler = {
        // matches any alpha numeric character
        // placed last in the collection of handlers
        // and used to match any remaining generic
        // property from the data set
        // handler returns true if the developer's
        // name, company, city, or country matches query
        matcher: new RegExp("^[a-zA-Z0-9._ ]+$", "i"),
        handler: function(query, dev){
            return dev.name.toLowerCase().indexOf(query) != -1
                || dev.city.toLowerCase().indexOf(query) != -1
                || dev.country.toLowerCase().indexOf(query) != -1
                || dev.company.toLowerCase().indexOf(query) != -1;
        }
    }
});

What The Code Does

A keyup event on the “search” field calls the trimDevelopers() function in the AdvancedFilters controller. We convert the query string to lowercase and split it at every occurrence of a comma, into an array, but retrieve the last or only item in the array.

If query is not a blank string (for “Q4, #GDE”, this can happen after typing the comma and space, but before typing #GDE) we iterate over our collection of defined query-handlers, testing the matcher RegExp for each one to see if it matches. If a match is found (“^q[1–4]$” matches Q1, Q2, Q3, and Q4) we assign queryHandler the appropriate handler function else we fallback a generic handler whose RegExp pattern is like a “catch-all”.

Given an object representing a developer, our Q1-Q4 handler works by first retrieving the birth month for the developer and then returning true if the month is within the quarter been queried for.

With the “chosen” handler function in place (assigned to queryHandler), we simply filter the data by returning the outcome (true or false) of calls to queryHandler for each developer object and having the final trimmed data assigned back to $scope.developers, which is what the rows on our table is bound to.

Go here to try out the experiment and hit me up on twitter via @chaluwa

Possible Improvements To This Implementation

Could be to be explicit on how to combine multiple queries. The comma in “Q4, #GDE” is a tad ambigious. Right now it means Q4 AND #GDE, so how do we filter for developers tagged with #Firebase but not #PWA ?

Maybe we can use “+” to indicate AND, “|” to indicate OR, and “!” to indicate negation, such that “#Firebase +#PWA !Q1” will mean “gimme devs tagged with Firebase and PWA who were not born in Q1”, and “#Firebase |#GDE !Q1” will mean “gimme devs tagged with Firebase or GDE who were not born in Q1”.

We can also extract any app coupling and expose all configurable parts (like the comma right now) so that we can make a design pattern and foster reuse.

Further Crazy Improvements . These are loud thoughts …

1. Making the code intelligent enough to automatically build an autocomplete / suggestion system for users, all from the matcher patterns. With this, a user typing “Q1, “ will get hints on what next to type, or typing “Q” creates hints of Q1, Q2, Q3, and Q4.

I will be trying my hands on the above “possible” improvements and share the outcome in part 3 of this series, so #WatchThisSpace. For now, let me know what you think of this approach and if it creates marginal or substantial value to data centric web app users