Half way through GSoC 2016…
Hi Everyone! For those of you who followed the link I provided and landed up here, this is my second blog. For others, you probably do not know me already. I am Pranay (21 | Male | India). I am pursuing my Bachelors & Masters (Dual Degree) in Computer Science from Indian Institute of Technology Kharagpur. This is my first year in the prestigious Google Summer of Code (GSoC) programme and I am lucky to contribute to CiviCRM during GSoC 2016. This blog is in continuation of the series of blog posts regarding my progress so far in the programme. If you are interested, here is the link to the pilot blog.
Apart from 29+ commits and 800+ LOC, a lot of things changed since the last blog a month ago. First things first, I changed the project title as @JKingsnorth suggested last time. RepoBuilder could be misinterpreted as something related to building repositories rather than reports, so I renamed it to CiviREBUX — REport BUilding eXtension for CiviCRM. I updated the description on the Google Summer of Code website here as well.
Continuing from where I left in the last blog, I started off by building the layout using gridster.js and AngularJS. Spent a week and implemented a canvas which supports functionality for adding/removing, dragging, resizing and placing widgets anywhere inside it. gridster.js also handles collisions that occur between widgets while we are placing them on the canvas efficiently, so in all the overall layout of the canvas was up and running.
These widgets would later be replaced with content blocks of the report like header, footer, media, data etc., which could be dragged and placed anywhere on the canvas.
Later that week during our weekly Skype check-ins, Brian and Jaap, my mentors for this project, mentioned that I should give more attention towards making data more functional and accessible for transformations like filtering, pivoting, sorting etc. rather than making the report layout more user-friendly. We then reached a consensus to have a look at Pivottable.js and how it can be integrated with our project idea. Pivottable.js is a JavaScript library with drag-n-drop functionality built on top of jQuery/jQueryUI and originally written in CoffeeScript by Nicolas Kruchten. The idea of using Pivottable.js was shared with us by @jamienovick through his valuable comment on my previous blog post. And from where I am standing now, I cannot thank him enough for that. Thanks Jamie!
Jamie also shared the link to a repository which contains an extension which he and others were working on then. This extension uses Pivottable.js for creating a pivot table, similar to the ones in most popular spreadsheet softwares, for CiviCRM activity data. The mentors and I collectively decided to take hints from this extension and use Pivottable.js for going ahead with building the report building extension for CiviCRM contribution & membership data.
I will briefly digress and talk a little about Pivottable.js and why it is one of the coolest libraries out there according to me. It would be great if you could take a little pause and have a look at the Pivottable.js by following the link here. I am pretty sure most of you would find it really cool. In our use-case, PivotTable.js enables data exploration and analysis by turning the contribution and membership data, obtained using CiviCRM v3 APIs, into a summary table and then adds an impressive drag-n-drop UI on top of it, which allows users to manipulate this summary table (aka pivot table) by adding/removing attributes (read fields), filtering, sorting and pivoting records based on each of the attributes. The summary table can be rendered as various kinds of charts and representations as well. In all, Pivottable.js fits our requirements exactly.
Coming back to the project, I started off by looking at the extension repository that Jamie mentioned and understanding the overall workflow. Took a couple of days to understand what’s what and draw a line between this extension and the one which I was going to build. I installed the extension by placing the repository files in my extensions dir as well. Thanks to the proper documentation, it was easy to understand how to proceed building CiviREBUX then.
Civix is yet another great tool which came really handy to get the initial skeleton of the extension up and running. Thanks to @totten for building this awesome piece of work. I used civix to get the basic layout of the module and a functional page inside it. Please find the link to the GitHub repository housing the extension here. I will quickly sum up the description of the files in the repository:
CRM/Civirebux/Data.php : Contains CiviCRM Contribution.get API calls and processes the obtained data
CRM/Civirebux/Upgrader.php : Contains install(), uninstall(), enable() etc. methods and code for adding `CiviREBUX` under `Reports` menu;
CRM/Civirebux/Page/ContribReport.php : Contains run() and a custom function for filtering unnecessary attributes obtained from the Contribution.get API calls
CRM/Civirebux/Upgrader/Base.php : Civix generated `Upgrader_Base` class file
css/ : Contains css for the extension page as well as the pivot table generated using pivottable.js
js/pivottable : Contains .js files required for rendering charts (minified and normal c3 and d3.js) as well as jquery-ui and pivottable.js (minified)
templates/CRM/Civirebux/Page/ContribReport.tpl : Contains smarty template file which displays the pivot table and the extension page
Other files are the classic civix-generated files and have the usual meanings.
Enough of the chitchatting, let me quickly get into what the present day extension looks like. After installing the extension by following the instructions in the README file attached with the repository, you will find CiviREBUX under Reports menu in the CiviCRM Dashboard, like below:
Clicking the link lands you directly to the extension page, which currently deals with only CiviCRM Contribution data, as the title of the page reads.
On the top you will see a minified version of an info banner which contains information about the extension and how to use it. Clicking on Expand link expands the banner like this:
Click elsewhere on the page to automatically minimize the banner back into its minified form. If you read the instructions carefully, you are all set to go!
The summary table (aka pivot table) for CiviCRM Contribution data, built using Pivottable.js, looks like this:
In the summary table, you can perform all sorts of data transformation and analysis, which also includes rendering the data into graphical charts built using c3.js and TSV-format. Here is a video of how you can analyze and report your CiviCRM Contribution data by exploiting CiviREBUX —
This concludes the updates for this week. Next, I will focus on integrating support for CiviCRM Membership data as well within CiviREBUX. Stay tuned for more updates. And needless to say, do give your valuable comments and insights — I really need them, just like the last time. Thanks again!
P.S: 10 commits later today, I have successfully added support for Membership data as well. Users can now select one of “Contribution” or “Membership” from a dropdown, which when submitted dynamically updates the pivot table and attributes. Here is how it looks now —
By default, Contribution data is selected and the pivot table corresponds to the Contribution data. The user can now select “Membership” from the drop-down and then submit the form by clicking the “Go” button next to the dropdown. A new pivot table corresponding to Membership data appears in the space like this —
It comes equipped with similar functionalities — filtering, pivoting, rendering etc., just like the pivot table for Contribution data. For sorting according to a particular attribute, move that attribute up/left to the top/left of the list, depending on whether the attribute is in row/column. In the above screen clipping, the data is sorted based upon the topmost attribute in the vertical list of row attributes — ID.
This was much faster than I had guessed. I will next look at ways to export the pivot table data into CSV/PDF formats. Will keep this blog updated. Thanks for reading!
A week later today, I have some interesting progress to share. I have added support for exporting pivot table data into CSV/TSV formats and downloading the same — for both contribution and membership data. Here is a first look of how it looks like:
For all those keen observers out there — see the two buttons on top of the pivot table, Save and Load. They are the next features I am going to talk about. I am currently adding support for saving/loading the configuration of the pivot table — row attributes, column attributes, renderer, aggregator, value, filters et cetera. The user can click the Save button and save the configuration for loading later again.
I began with saving the configuration of the pivot table into browser localStorage because it was faster and easier to set up. A couple of days later, Brian and Jaap suggested that I should probably go for saving the configuration into a CiviCRM table and fetch it while loading.
Its been a couple of days since then and I am currently implementing an AJAX action which is triggered when the Save button is clicked. There were some initial confusion regarding the URL that will be added as a parameter to the AJAX call. After speaking to Brian yesternight, I got to know about the way CiviCRM handles URLs and page callbacks — using xmlMenu file and CRM.url() method. As of now, the AJAX call is going through and a success is returned. Not to forget, I also added a BAO class for handling the save action, which takes a configuration (passed through the data parameter in the AJAX call) and inserts it into a CiviCRM table, for fetching later.
The latest code housing support for saving/loading lives in a new feature branch of my repository: saveConfiguration. Once, this feature branch works well, I will merge it with the master branch…hopefully, within a couple of days.
Thanks for reading! Will keep the blog posted with more updates!!
