kdb+: haq-paq
https://github.com/Version1/haq-paq
Author: Christopher Murphy
At Version 1, we are committed to customer success and, as a result of our Capital Markets division's rapid growth, we wanted to build a kdb+ library that would act as a springboard to kick start our kdb+ projects.
We know that this relatively niche technology is used widely in the Capital Markets sector, but kdb+ solutions are not readily shared outside of individual institutions due to the competitiveness of the market. Luckily for you, one of Version 1’s core values is “No Ego”, and so we’ve decided to help our fellow kdb+ devs in making this library open-source!
Introducing: haq-paq
From our daily team conversations, we noticed that over our collective experiences there were common libraries of code that other languages have that are missing for kdb+, therefore meaning that we had to write — from scratch — a new library for each new project we were spinning up. A couple of examples might be basic logging functions that show memory stats alongside timestamps and messages, or a basic maths function like “round”, which may be easy to write in q, but would be even easier to have already loaded in. Off the back of that conversation, we decided to create an open-source project to contain all these various functionalities.
Of course, at Version 1 we deal with the best technology for the specific client project, so we decided to emulate functionality that is available in other languages’ libraries (Python — linspace, prime number calculation, mode of a list, etc.) and include some of this in haq-paq.
Now that we knew we wanted to build out this utilities library, we needed to figure out exactly what utilities it should contain. So, we spun up a Miro board to facilitate our team’s brain dump. Once we had everyone’s initial ideas, we grouped the suggestions into themes and reviewed them again to see what else should be added.
Once that was all worked out and agreed, haq-paq’s functionalities would include (more details can be found in the README):
All the Bells and Whistles
Once we had our goals laid out, we were ready to set up a repo and get to work. We built the library out collaboratively and had continuous code reviews, which led us to three further talking points: unit tests, code standardization and documentation.
Unit tests are an effective way of ensuring all our code works correctly now and after future updates to the haq-paq library. The main benefit of unit tests is that going forward we can be confident that no additional changes will have broken the code and that existing functionality works as expected. They save a lot of time for developers who would otherwise be required to create tests themselves locally each time to check that the functionality works as expected after some changes. Our unit tests for haq-paq are stored privately but rest assured — they will be used as we continue to expand haq-paq!
Code standardization, as you can imagine, sparked a big conversation ranging from naming conventions to comments and style of writing the functions. The main benefit of code standardization is easily readable code and consistency throughout the project. Some things we managed to agree on were:
· No unnecessary terseness
· Standard variable and parameter names
· Function names: lowerCamelCase with some verb to start the name
· Global tables: UpperCamelCase
· Column names for tables: lowerCamelCase with a debate on acronyms
The easiest thing to agree on was that we all wanted to avoid the unnecessary and dreaded terseness that often comes with kdb+ code (even though I have been known to dabble it in from time to time). The issue of standard variable names may seem unimportant, but it’s to avoid things like having input parameter names for tables ranging through t, tbl, tab, etc. depending on that developer’s preference. When one starts on a project for the first time as a young developer these can seem like minor issues but, the more I worked in kdb+, the more I saw the value in a standardised and consistent coding style. Although it’s a popular choice with q developers, we wanted to avoid the mini-game of writing code in the minimum number of characters as in the long run it kills supportability, but it can be fun during the advent of code each year!
Documentation is an integral part of any successful project — it’s useful for:
· Developers using the code for the first time
· Developers coming back to the code after writing it some time ago
· Anyone unfamiliar with kdb+ to get a high-level overview of what is available
The benefit of documentation is that the code is easily understood and whoever uses it knows what it does without needing to drill down into the specifics of each function. For haq-paq, we wanted to use a tool that would generate the documentation from the code comments. After reviewing different options, we found that q-doc allowed us to do exactly what we wanted. We wanted our documentation to be a part of the repo, so that anyone looking to know more about haq-paq could just click a link and see all the latest functionality broken down on one webpage. To do this, we had to make some enhancements to q-doc. We used Python’s BeautifulSoup (bs4) and Selenium to scrape the q-doc page and save it to include it as a one-click resource in the repo for our users while also being able to navigate it easily.
In conclusion, our hope for this library is that it will be advantageous to both newbie and experienced kdb+ developers. Ideally, this could make up the initial core of a utility library for clients that are starting to use kdb+ more and more. If you’ve stuck with us so far then haq-paq may be the tool for you — give it a try.
About the Author:
Christopher Murphy is a Senior Software Engineer here at Version 1.
