From CSV to Buxfer: an unexpected journey — Goxfer

Part 4— Goxfer: a story about pushing a transactions dataset online

Wilk
Wilk
Nov 22, 2017 · 7 min read

Preamble

In Collector — Part 3, I’ve coded the collector program, putting cleaned data inside a MongoDB instance.
Now, it’s time to use the Buxfer’s APIs to put the whole dataset online!

Journey

In this article, I cover the fourth part of this journey:

  1. Part 2: Cleaner
  2. Part 3: Collector
  3. Part 4 (this part): Goxfer
  4. Part 5: Conclusions

Setup

This new program will be coded with GoLang: this means I need to build a custom docker image and to install any dependencies via glide, one of the best package manager for Go.

$ docker-compose build setup-golang
$ docker-compose run --rm setup-golang bash
# from inside the setup-golang container
$ glide create
$ glide get github.com/parnurzeal/gorequest
$ glide get gopkg.in/mgo.v2
  1. glide.lock: it’s a lock file to ensure deterministic restore procedure
$ docker-compose run --rm setup-golang

Goxfer: hands on!

So, what does Goxfer have to do?
Let’s define the program procedure:

  1. initialise Buxfer’s session
  2. retrieve Buxfer’s accounts list
  3. fetch transactions from the DB and build transactions’ bulks
  4. push transactions’ bulks online via Buxfer’s APIs

Database

The very first thing I need to code is the database connection.
mgo is a MongoDB driver and it just need the database host to establish the connection, plus the database name and the name of the collection it will query for the transactions.
I need to define those informations inside the docker-compose.yml as environment variables for the goxfer service, already drafted in Part 1:

Buxfer’s session

Buxfer requires an authentication token for each request performed; so, before proceeding, I’ve to establish a new session.
I’ll use gorequest that is an HTTP client with a practical and easy-to-use interface.
The login API requires a username and a password, so I’ve to put new environment variables inside the docker-compose.yml file:

Accounts list

Before proceeding with the transactions, I need to get the accounts list because later I will need the id of each account.
So, for now, I just have to map the expense and the income accounts with their ids:

Transactions’ bulks

Now, what I want to do is to get all the transactions from MongoDB and then
to push them on Buxfer.
But, there’s a problem: Buxfer’s APIs are limited. In fact, it is possible to push just one transaction per time: https://www.buxfer.com/help/api#add_transaction
Well, I don’t want to wait for the end of each request to perform a new one, neither to perform 1000 calls in parallel.
So, I think I’ll go for pushing a bulk of 20 transactions per time.
To do that, I need two things:

  1. (ab)use GoLang’s goroutines to perform HTTP calls in parallel

Pushing to Buxfer

Ok, this is the tough part.
As I said, I want to push bulks of 20 transactions per time. This means I’ve to launch 20 goroutines in parallel and then wait until they have finished. All of them. Then, I can start with a new bulk and so on.

  1. Wait: stop the main thread until the workers counter is equal to zero
  2. Done: decrement the workers counter
  • then for each row (aka bulk) I’ve to increment the WaitGroup counter by adding the length of the bulk
  • then I need to loop through the current bulk and for each transaction instantiate a new goroutine to push the data on Buxfer
  • for each goroutine that has finished, I need to decrement the WaitGroup counter via the Done method
  • eventually, I’ve to stop until the entire bulk has finished, using the Wait method of the WaitGroup, so the loop can continue with the next row of the matrix
  • amount: transaction’s amount
  • accountId: calculated matching the Buxfer accounts list, fetched previously
  • tags: transaction’s tags converted into a comma-separated string
  • date: date formatted with YYYY-MM-DD format
  • token: session token
  • type: transaction’s type, calculated the same way of accountId

Run it, dude!

The final step: launch it 🚀!

$ docker-compose run --rm goxfer

End of part 4

Wow, this last part was tough 😵
It requires a lot of work and a lot of testing. In fact, initially the API for adding transactions on Buxfer was different (and bugged) and it got changed (and fixed) at the end of October. I had to change the source code, performing new tests with PostMan and curl before running the program safety.


Sources

Source code is available here: https://github.com/wilk/from-csv-to-buxfer

Update

I’ve improved the unpushed transactions logs and I found that one of the raw transactions was corrupted: its date was 29/11/1898 so that’s why Buxfer refused to accept it.
Anyway, Cleaner, Collector and Goxfer did the job and they did it really well!

HackerNoon.com

how hackers start their afternoons.

Wilk

Written by

Wilk

Passionate developer 💻, amazed astronomer 🔭, casual writer ✍️, full-time father 😵

HackerNoon.com

how hackers start their afternoons.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade