It’s Been a Ride 🎢
A Story about Contributing SAP HANA Support to QGIS
Tl;dr QGIS 3.18 comes with support for SAP HANA. Contributing took a little while. Get a free trial instance of SAP HANA Cloud and download the QGIS nightly build.
About the author: Mathias works as a development manager for SAP and is responsible for SAP HANA’s Spatial engine. This article may be biased and reflects personal views, but be assured that it’s based on a true story 😊
In the field of GIS (Geographic Information Systems) the open-source software packages are rather feature rich and mature, so that for many companies these are a valid enhancement or even alternative to proprietary software packages. Thus, at our customer’s site we often find hybrid landscapes using proprietary GIS software, open-source GIS software and SAP systems for covering their non-GIS business processes.
SAP is committed to enabling The Intelligent Enterprise. For us, as a development team within the Business Technology Platform, this means that one of our goals is to provide a platform, which is capable of transparently incorporating our customer’s business data to fuel their business processes. Moreover, as the development team behind SAP HANA’s Spatial Engine, we care about the location-side of things and want to enable our customers to seamlessly integrate their GIS and ERP to harvest the value and the sometimes-untouched potential of their business data.
From a GIS perspective SAP HANA is already well-integrated with Esri software, the open-source middleware GeoServer as well as DBeaver as a tooling for the spatial database developer. Any GIS professional will have already spotted one or two gaps on that list — one of them for sure being the open-source GIS client QGIS.
While spatial data in SAP HANA could already be displayed in QGIS by leveraging the WMS/WFS services provided by GeoServer, we wanted to investigate the possibility of having a more native integration to leverage QGIS extensions like the TimeManager. Back in the days — somewhen around 2018/2019 — we took the decision to contribute native SAP HANA support to the QGIS project.
Said and Done. Well… uhm… almost. Let’s have a look at how our concise and clear path towards contributing to the QGIS project turned out.
May 2019
On May 21st, 2019 and after being clear about what we would like to achieve, Max from our team forked the QGIS repository and started to technically plan the contribution. We started developing and testing the plugin internally before we placed our first (we didn’t know yet, that it wouldn’t be the last) pull request.
July 2019
That was on July 16th, 2019. We were done with the implementation and just had to wait for the pull request to get merged. We thought. Then we got some feedback from the community:
“[…] we might need a bit more context on what this is and why it needs to be a core provider?”
“HANA database, that’s new to me :)”
“[…] a [HANA] provider would not be of use to a vast majority of its user”
“Historically when we’ve done this is becomes a huge maintainance burden on the project”
“I’d like to point out process of QGIS Enhancement Proposal […] before going deeply in implementation like in Pull requests”
There were some valid questions, some advice and a couple of comments that hurt our developer egos (Come on! If there is no HANA support, how could it possibly have a large HANA user base?). There were also kind comments and, of course, there was the pointer to the QGIS Enhancement Proposal (QEP) process, which we have — regrettably — not considered before.
After having a couple of discussions with the community and clarifying some of the questions, including license concerns, we sorted ourselves and made the official enhancement proposal according to the process.
That was July 24th, 2019. Sticking to the process helped (Hey, we are SAP. We should’ve known before). The discussion got considerably more objective. Well, almost.
A: „Adding a new provider isn’t just a free gift to the project. Rather it’s more akin to giving someone an unexpected puppy for Christmas. Sure, it’s all fun at first, but then there’s always someone who inevitably has to feed it, clean up after it, pay the vet bills,…“
B: „It seems unfair to me that QGIS has core providers for commercial databases, but does not allow to contribute support for our database in the same way it has been done for other commercial databases. Even worse, it feels like we are being punished for a competitor neglecting the maintenance of its core provider, while the competitor was actually rewarded by the community taking over the maintenance of that provider.“
A: „Unfair? […] Maybe in the same way that a first child might get different standards for parenting and have different rules from a second child, simply because the parents have learnt more about parenting due to their previous mistakes experiences!“
When looking at the history of communication the above dialogue really nails down the core suspicions between both parties — the QGIS community and our SAP team.
The QGIS community has learnt from past experiences and was concerned about the additional maintenance burden, while we felt misunderstood and unfairly treated.
August 2019
The discussion went on for a couple of weeks. On August 15th, 2019 our original pull request got closed due to inactivity. In the beginning, it felt that discussing with an open community (rather than with a single representative) is like fighting windmills. Once you agreed on one point, another person stepped out of the dark and brought a totally new aspect, so that you could never be sure, what the state of the discussion actually is.
September 2019
Finally, on September 21st, 2019 we concluded on the major pillars of our contribution (e.g. testing platform, CI integration and maintenance).
For us this was a major break-through! We finally had a reliable agreement we could build upon. It was clear, that we had some additional work in our backlog — but it all seemed manageable and clear.
December 2019
Last day before Christmas break on December 20th, 2019, I had the wishful thinking, that we could issue our second pull request and could possibly get into the next version of QGIS. I even had a celebration blocker in my calendar — but of course, this remained a wishful thinking. We have learnt that we would require more community expertise to efficiently flesh things out.
January 2020
There was a QGIS Contributor Meeting (also known as Hackfest) planned to take place in Den Bosch in March 2020. That would be the best opportunity to get in touch with the community and discuss technical topics! (…as a 2021-reader you will already anticipate that something ‘unprecedented’ came in the way). We gathered a travel group of 3 team members and — since you do not go to a house party without having a present — SAP also sponsored the event.
In the meantime, we adapted our 1-year-old codebase to some major API changes, which affected our contribution.
March 2020
On March 10th, 2020 the Contributor Meeting finally got cancelled due to COVID-19.
As it was clear, that in near future there was no opportunity to liaise with the community, we finally issued our second pull request on March 11th, 2020. Almost certain that almost everything has been done and we are almost good to go. Due to my devotion for the number Pi, I remember that a realistic feature release was QGIS 3.14 at that time. (I have learnt that I seem to be an optimist rather than a realist)
April 2020
We received a couple of review comments, which we considered until April 22nd, 2020. Afterwards, Stefan started our internal peer review process — as it is part of our engineering standards.
July 2020
We finished the full peer review process of the meanwhile quite large codebase. On July 3rd, 2020 I took another stab in declaring production-readiness. However, there was still this one nasty issue: Due to memory consumption, we did not manage to run our HANA Express docker image on the Travis CI of the QGIS project next to the images of the other databases. This ultimately led to the fact, that we could not run our test suite on the CI.
September 2020
Meanwhile Max gained expert knowledge of Travis, which is why today he is unwillingly considered to be the CI expert in our team. Finally, we gave up to run the database on the CI.
On September 17th, 2020 we could run our tests using an SAP HANA Cloud instance on the QGIS Travis CI. Thus, we took advantage of the fact that SAP HANA Cloud has been released in the meantime.
December 2020
We have been undergoing further community reviews until December, which we have thankfully considered for our contribution. On December 18th, 2020 — last day before Christmas break — I had a nostalgic look back and thought of the foolish feeling last year, that we could get our contribution merged in 2019. This time, I felt confident to see the merge happening in 2020 still.
January 2021
Of course, based on prior experience my confidence was clouded. On January 6th, 2021 Matthias — one of the main contributors of the QGIS project — announced that he plans to merge our contribution by the end of the week if there are no further concerns.
After not sleeping for 4 full days (actually, I slept well… but this would not add to the dramaturgy), our almost 2-year-old pull request finally got merged into the QGIS master branch on January 10th, 2021.
Still, our tests ran for almost 10 minutes as the CI infrastructure was in the US and our HANA Cloud instance in Europe. So, we spun up an instance on the US east coast and came out with an acceptable latency.
Today
Here I am writing this article and waiting for the last changes to the QGIS Mac Packager to be merged. Once this is done, I will install the nightly build and publish this article not one single second before I see the SAP HANA data source. As mentioned before, I have learnt that I tend to be too optimistic. So I’d rather play it safe this time.
The heck with it! I’ve seen it working with the nightly build on my Windows VM — what could possibly go wrong?
After all, it seems that we have delivered our unexpected puppy present. We were a bit late for Christmas, but we will take it out and clean up after it. And of course, it’ll be fun! No one ever doubted that!
Are we all friends now? Yeah, kind of. There is still a bit of suspiciousness. But I personally do understand the rational behind way better than I did 2 years ago.
We, as a team, are looking forward to becoming a good citizen of the QGIS project. After all, we are developers and spatial enthusiasts. The bumpy ride we had, just got us more engaged with the project.
If in doubt, this article should be interpreted as being self-critical and self-deprecated. Once we left the current pandemic behind, we are very much looking forward joining an upcoming QGIS Contributor Meeting!
Follow me on Twitter and watch out for further communication and tutorials on how to use SAP HANA Cloud with QGIS 3.18. If you cannot wait, just start using the SAP HANA Cloud Trial and the QGIS nightly build (both free of charge) today!
