How we hacked the Braintree API to store an unlimited number of files
Its 2015, and data is pretty important, so important that some companies are spending thousands, if not millions of $$$ on building and maintaining the infrastructure needed to securely store these golden 1's and 0's. Manoj Nathwani, Mike Hoy and I all believed this to be absolutely insane and we to fix it (by passing along the hassle and costs to someone else).
Ok ok, so even though it is pretty infeasible to use for real, we actually did end up building something pretty cool, so i’ll try to be less sarcastic from here.
So how did we do it? Basically we (ab)used the fact that Braintree lets you store custom fields in their customer objects. These custom fields are a set of key-value pairs that are suppose to be used for storing small pieces of user relevant user data like a user id or session token, so each is limited to only 255 characters. This means that if you only want to store things that are 255 characters long you’ll have no problem just diving in and creating customer objects via the server side SDKs, you don’t even need to send any other values along (like firstName or phone) because customer objects have no required fields.
But we wanted to store more than just tiny strings of text, we wanted to be able to store files, like pictures of cbetta. So we came up with two solutions. Both involved splitting the data up into 255 character chunks, and then either creating a new customer for each chunk of data with some link, or creating a whole bunch of custom fields in each customer.
Both had their pros and cons, splitting the chunks up between multiple customers would mean that there wouldn’t be a maximum file size, but it would also take a long time, and many requests to both create and fetch each customer individually. On the other hand storing everything in one customer over multiple custom fields would mean that we would only have to make one request, but each of the custom fields needs to be created manually in their control panel, and there is no way to automate their creation (It’s almost as if BrainTree didn’t want people to be storing masses amounts of data in their systems).
We decided to go with splitting the data over the custom fields, because the slowness of fetching hundreds of customer objects would of made this already infeasible solution, even less feasible. So, because we didn’t want to spend the next 12 hours leading up the demo, manually creating new custom fields from their control panel form. I wrote a selenium bot to do the work for us, it wasn’t the fastest thing in the world but 10 running simultaneously were able to create 1000 in about 2 hours (while at the same time paralysing my CPU). 1000 custom fields obviously isn’t enough to save anything that big (only 255,000 characters), but it was enough to let us live demo a decent sized image, and a small mp3 file.
So now that problem was solved and we knew what we were doing, we were finally able buckle down and write some actual code (well… all 50 lines of it). We ended up creating a simple “REST” api written in Python using flask, with two endpoints. The first endpoint was a POST to the root, where it just accepts a multipart file which we used to upload, and the other endpoint was a GET with to a URI that was just a /:customer_id (The BrainTree generated customer id) which is used to download the file again. I’ll go into a bit more detail of what actually happened at each of these points now.
Firstly, the application processed the uploaded file and converted it into a byte array, next it base58 encoded the byte array into a long string of characters. We chose base58 because it results in a purely alphanumeric string, rather than base64 which can use unusual characters like backspaces and new lines. We didn’t want to run the risk of BrainTree saving any of these characters incorrectly so we just avoided it. The final step was chopping the long encoded string into 255 character chunks and splitting them up between the 1000 custom fields, and it was done. We had uploaded a file to BrainTree.
Just to give you an idea, the photo of cbetta shown earlier took up 133 of these 255 character fields, at a 400px x 400px filesize. And our demo mp3 file took up about 756 fields, and it was only 5 seconds long!
Downloading the file was just the process of uploading reversed. We requested the customer matching the customer_id provided in the URL from BrainTree. We then sorted the custom fields back into the correct order, joined them all together again, decoded the joint string from base58 back into the byte array and finally sent the raw file data back to the browser with the correct response type and… *gasps for air* hay presto, BTAAFSS (BrainTree as a file storage service).
*drops macbook*
If you have an questions feel free to tweet at me @viralpickaxe :3
