The app I’m working on in my spare time has a need to consume large lists of data, with a key requirement of making it easy to filter the list based on properties of items in the list. Additionally, the list should be searchable to further narrow the results.
There are several places I was going to need this functionality, so it needed to be as generic as possible. I’m making heavy use of the
flutter_bloc package for state management, so it should ideally be able to slot right into the existing app architecture. Needing to be generic anyway, it seemed like a perfect opportunity to create a Flutter package that could be helpful to everyone!
To recap, our end goal is a package that meets the following criteria
- Follows the
- Accepts a data source
- Exposes state derived from specified properties of the data source items that can be used to render UI that manages the available filter options
- Is able to filter that data source by user-activated options
- Is able to narrow that data source by searching provided properties
- Exposes state that can be used to render a list
- Is as generic as possible
It shouldn’t be difficult to get started with this package. To that end, I knew I only wanted one main entry-point that needed to be integrated. Any remaining functionality should be exposed underneath that widget but automatically connected and ready to use.
As someone interfacing with the package, you should be able to render your UI however (and wherever) you’d like. We’ll be integrating with
flutter_bloc which already has an excellent means to accomplish that via the
BlocBuilder widget… which means our primary means of communicating state out should be a bloc.
It stands to reason, then, that everything could be hooked together as shown in the following diagram:
It’s the only widget directly rendered by the application, and is supplied with:
- The widget you want to render that will have access to the package-injected blocs in its build context
- Properties (corresponding to the items that will be provided by the source bloc) to use while filtering/searching
- A source bloc providing the base data
A bloc that takes the source bloc and filter/search properties and:
- Uses them to generate groups of values that match all of the incoming data
- Exposes state (as a stream) to the list state bloc
- Is available to the child build context to render an appropriate filtering UI and to allow that UI to toggle filtering conditions as active or inactive
A bloc that takes the source bloc and state from the filter bloc to:
- Filter the incoming data based on any filter conditions that have been selected as active by the app
- Is available to the child build context to render an appropriate list UI
One would imagine the logical place to start would be the entry widget… after all, that’s going to be the integration point with the package. Yup, that would have been a great idea! At the time, though, I still wasn’t sure exactly how I wanted to tie everything together. Yeah, I know the diagram up there looks really flashy, but it comes from attempting to piece this timeline together once everything was completed. (The next guide I take on I plan to actually write it as I’m going through the process, which should be a great change of pace).
Perhaps the next best place to start would be the list state? Sounds really good, especially since the whole point here is to render a list of filtered data to our UI. However, my brain was really just ready to start tackling the filtering portion of the problem, so that’s where I ended up starting.
Having never created a standalone package before, I decided a prudent first step would be to start development in my current project until such time that I was able to prove out most of the functionality. As I was working on the filtering bloc (in-depth dive below), I didn’t like that I planned to be dealing with both the filtering state and the search state in the same place. Dealing with both the list of potential filter conditions and active filter conditions was already shaping up to be more than enough for one bloc. Additionally, it was acting as a straight passthrough of the target search properties from the entry widget, which was also not ideal.
Before revisiting the plan, I decided it was time to migrate all of the code (basically the working filter conditions bloc and data classes) into a separate repository. However, there was a huge hiccup when I started that process! You can read all about that adventure here.
The new plan
In hindsight, the new plan wasn’t drastically different from the old plan. It simply spread out the concerns a little more.
The main entry point to using the package. The required
child has access to all of the below blocs to use while constructing UI. At a minimum, you must also supply a list of keys to be passed along to the filter conditions bloc.
There’s really nothing much to this widget, it just sets up the other blocs and provides the
child to be rendered.
Filter conditions bloc
Now that the filter conditions bloc is no longer responsible for any of the search state, it’s a bit easier to reason through.
On initialization, we need to subscribe to the
sourceBloc in order to regenerate the
availableConditions any time our source information changes. We also want to ensure no
activeConditions are left dangling.
sourceBloc isn’t in its loaded state (as determined by the supplied type from the parent widget), we want to skip parsing the data for now.
For every item in the source state, we need to keep track of the corresponding value for every
filterProperty. In an effort to reduce the number of iterations through the source items, we also need to go ahead and store all potential incoming condition keys so we can narrow down any
activeConditions that have been removed from the source state.
generateConditionKey, why didn’t we use the same storage format as we did for the
availableConditions? That was initially the plan! However, the use-case for
availableConditions — rendering the available values for every filter property key — lends itself nicely to nested iteration. In the case of
activeConditions, we always want direct and easy access to that list. A concatenation of the filter property key and its value serves as a unique enough identifier. At the moment we’re filtering out everything but
String values, but adding number/boolean support would be rather easy.
Last but not least we want to ensure a stable (and unique) ordering of
availableConditions such that the filtering UI doesn’t shift around as the source state is updated.
Adding and removing conditions
These two functions only differ by two lines, so we’ll discuss them together.
If we haven’t entered into an initialized state yet, we’ll have no means to accurately set a condition as active and don’t want to modify the state.
If the set of
activeConditions already has a matching entry (or in the case of
RemoveCondition if there isn’t already a matching entry), there’s no need to modify the state.
When using the
Bloc pattern state mutation is a no-no (see ‘The mutation problem’ below), so we need to create a new
Set when manipulating
Search query bloc
There’s really nothing at all to this bloc.
searchQuery sets it back to an empty string. Setting a
searchQuery will store the provided value lowercase in order to make filtering source items more reliable.
Item list bloc
ItemListBloc is the bread and butter of the package. It takes the source items,
activeConditions from the
searchQuery from the
searchProperties passed down from the base widget and distills them into a complete list of items that can be rendered however you’d like.
As we rely on data from three other blocs, we need to set up our listeners. We could do work inside these callbacks (as we did in the
FilterConditionsBloc), but it made more sense to keep everything simple and in the event system.
Don’t forget to cancel the listeners when the bloc is closed! I initially tried to pipe all three of these into a
Future.wait, but that doesn’t play nicely with the null aware syntax as a promise wasn’t provided in the case where the subscription didn’t exist. The only other option is a lot of conditional logic or providing empty promises as default values. I made the decision to not optimize that now, as closing the
ItemListBloc should be a rare occurrence.
I generally try to keep logic out of
mapEventToState, however, this bloc needs to respond in the same way whether new source items come in, new
activeConditions come in, or a new
searchQuery comes in. The entire list needs to be regenerated every time.
If conditions aren’t already initialized, or if the source bloc isn’t in its loaded state (it shouldn’t really be possible for the filter conditions to be initialized and at the same time have the source bloc not loaded… but it doesn’t hurt to protect against that case) we have a special state we want to emit for that in order to differentiate from a standard empty state.
We then send the source items through filtering and searching (see below) and dispatch either an event to let the caller know there are no results (an empty state), or an event with the filtered and searched results.
Filtering source items
As with the rest of the implementations, no one part is terribly complex… that’s by design. Filtering the source items is very straight forward.
If there are no
activeConditions we can short-circuit some of the logic and immediately return the source items.
We then proceed to check the source items against all of the
activeConditions. We can also short-circuit some of this logic using
any… we don’t care if the item matches every single active condition, it just needs to match one.
Searching source items
Similar to filtering the source items, we can short-circuit some of the logic and immediately return the source items if there is no
We then proceed to check the source items against the
searchQuery for every provided
searchPropery. We can also short-circuit some of this logic using
any… only one search property needs a positive match.
This is, for sure, a very basic search algorithm (if you can even call it that). It gets the job done, but that’s about it. More information below, but a proposed update is to provide a pluggable search callback such that you could implement whatever search works for you (I’m mainly thinking about fuzzy searching).
I’m not going to cover every single line of test code here, I don’t think it would be valuable. Instead, I’m going to focus on a few test cases that were noteworthy.
Wherever possible I’ve moved to the
bloc_test package for testing. It saves hassle, provides pretty much all of the helpers you would need, and encourages isolated test state (even if that does make things a little more verbose in the end). Very highly recommended.
The mutation problem
As most good tests should, this particular case helped validate assumptions I had surrounding underlying state management with the
Bloc pattern. The test errors quickly highlighted where I was going wrong and where the fix should be implemented.
More specifically, I had hoped to be able to get away with a little state mutation when updating the active filter conditions. In the current single
Set iteration that’s very easily accomplished. In the previous implementation with nested data (that led to the above-linked test case throwing errors), that required far more boilerplate to accomplish.
As the data was heavily nested we would first have to check if an array of values existed for the provided property, and if that array didn’t already have an entry for the provided value.
If no value existed, we need to create a new map reference to hold the existing entries. Then we need to update the array of the provided property (again, by creating a new array reference and modifying it appropriately).
The stream problem
As mentioned above, the
bloc_test package really helps when testing a
bloc. In almost all situations the
whenListen helper is plenty powerful, however, it doesn’t give any flexibility as to when the provided items are piped into the stream. If you want to test interactions between an external bloc (or stream) and the internal behavior of the bloc under test, you’ll need to look for another solution.
The simple setup below allows you to add items into the source stream at whatever point in time you actually need to test, instead of all at once on the first listen.
Where to go from here
There are absolutely things I don’t like about the finished implementation and things I would have done differently given what I know now.
I feel the implementation would be better served by not having to supply a source bloc, supplying a repository to the widget instead. Since the package is supplying all state necessary to render UI for the list, having a source bloc manage that as well is rather redundant. It also allows for the internal state checking in the package to be much simpler.
I love the flexibility that fuzzy searching brings to the table. However, I don’t feel it should be the default for everyone, nor should it increase package size needlessly if it will never be used. To that end, there should be some form of pluggable search provider or callback the caller can use to replace the current (naive) search implementation.
Non-string filter conditions
Booleans will need special handling as their value isn’t display-ready. Numbers will need special handling as we may want to treat their values as part of a range instead of distinct values.
The only option for filtering at the moment is additive (meaning that an item that matches any one of the active conditions will make the cut). More advanced filter options would be a benefit.
I would love your feedback on the already proposed improvements, as well as thoughts on other ways to improve the package!