My Experience as a Software Developer Intern at a Trading Firm
This past summer, I interned as a software developer at a top-tier Chicago-based proprietary trading firm. Going in, I expected that a large portion of my internship would be dedicated towards learning about the trading/finance side before writing any code. To my surprise, there were quite a few software developer positions at the firm that required minimal, learn-on-the-go knowledge about finance. I got to work with the Compliance Team, which was split into two teams: one consisting of software developers, another of compliance officers. As a software developer intern, I got to work on various smaller web and data visualization projects that ramped me up for the bigger project that I ended up developing for the majority of my summer.
Although I can’t delve into much detail on my project due to confidentiality, the general gist of it was a job automating server that monitors the statuses of the data analysis studies that have to be run daily, to extract review units for the compliance officers to view. These review units provide information about the trading activity throughout the firm to ensure their compliance with all industry regulations. So, it was pretty important that these study jobs were run and completed to a degree of high accuracy every day.
The problem with the server the developer team had in place was that it didn’t provide full clarity into the statuses of each individual job unit — it only provided information on whether the overall study had produced a review unit or not. There was no way to track whether a single unit of the study job had failed vs multiple units, whether it failed due to lack of data availability, or whether a job was still pending or enqueued. So, the task I was given was to implement a new server that tracked and automated each unit of the jobs run daily.
To do this, I also had to implement a new check system: For each job unit, a check was generated in a checks database, with statuses ranging from pending, confirmed, and refuted. Then I added a scheduler config which generated all the checks for each job unit that had to be run for that day at a certain time. The job runner server I implemented would read from the checks database every few minutes, grab all checks with unconfirmed statuses, and run or enqueue the corresponding job units using Elixir’s Exq library, and keep a list of all unconfirmed checks in its state. I then migrated all the studies to include functions that read in corresponding job arguments from a check struct, checked for data availability, and then ran the actual job unit. Sounds simple, right?
Wrong. The first version of this system that I deployed ended up crashing multiple servers. Aka every intern’s worst nightmare. The first, horrifyingly embarrassing mistake I found was that I read in the list of unconfirmed checks using “checks” instead of :checks (you’ll know the difference if you’ve ever used atoms in Elixir), which caused the same jobs to be rerun tens of thousands of times. Another major error was that the state was emptied whenever the servers were temporarily shut down during deploys, thus causing more duplicate jobs. Overall, I was utterly embarrassed and scrambled to get the fixes deployed.
This could have resulted in a traumatizing experience on my part, if it hadn’t been for my amazing mentors. During the whole fiasco, the senior developer I was working with sent me a DM saying he had once accidentally deleted an entire year’s worth of data at a startup company he had previously worked at, due to a silly mistake, and that I shouldn’t be worried at all. My mentor also DMed me saying that this wasn’t my fault at all, and that the team should have had a better system in place to make sure my deploys wouldn’t have been able to crash the servers in the first place. To sum, my team was incredibly forgiving and supportive as I executed the fixes.
So what did I take away from my internship this summer? Apart from learning two new languages (Elixir/React), I got quite good at using Git in an agile workflow, creating data visualizations, and developing full-stack web applications. But above all, I learned how important the people you work with are. It can really make or break your experience, regardless of where you’re working and becomes a deciding factor on whether or not you want to return to that company in the future.
Overall, my experience at the trading firm was much more tech-oriented than I had expected, but I did pick up quite a bit of knowledge about quant trading, cryptocurrency, and investment throughout my internship. If you’re someone debating whether to take take an offer from somewhere like Facebook vs a top trading firm, I’d definitely advise you to weigh your options (quick reference: http://qr.ae/TUN2b3) and go with whatever you find more interesting, because you’ll gain valuable knowledge no matter which route you take.