Surviving a startup minefield
You don’t always know what you’re getting yourself into when you accept a job.
This is the story of my first two months at a startup I worked with — let’s call it Transaction Software, Inc. In short, it was a brutal mess.
I’ve written my recollections here. While it’s now been some time since the events occurred, much of the story here is based on notes and logs I had kept as the situation was unfolding.
I’ve changed the names to protect everyone involved. As a disclaimer, everything I write here is from my perspective, my understanding, and entirely my opinion only. To be completely fair, I may be missing pieces of the puzzle that could change my perspective.
Ready for my nightmare? Let’s begin…
The beginning of the strange journey
Elliot, the CEO of Transaction Software, Inc., was looking for a senior front-end developer. He somehow found me on a job board and reached out to me.
During my interview with the CTO, Bob. I dug into the company and its history. I found out about the team , the technology, the problem space, etc.
The engineering team consisted of 2 junior front-end developers and Bob. There were multiple projects in the works, but Bob and the engineering team were currently focused on an effort to upgrade an existing transaction platform written in Ruby on Rails by a previous team and give it a facelift. The projects were already late and running up against some upcoming final deadlines.
The new front-end was written in Ionic 2 / Angular 2, and my first task would be to wire it to the Ruby on Rails system, which I was told had fully-functioning JSON APIs. I would be leading the front-end development efforts as a senior front-end / full-stack developer, guiding the junior developers.
I believed that I had found a good place to work: the technologies were recent, the domain was interesting, and I believed Bob would be someone who could mentor me.
I was wrong.
The truth hits like a freight train
From the very first day it was clear that the amount of progress had been…misrepresented. There was still quite a long ways to go. The Ionic 2 / Angular 2 application I was hired to wire up to the backend turned out to be almost entirely a mock. Much of the data was entirely hard-coded and flows through the pages scripted in a smoke-and-mirrors way. I wasn’t just going to be wiring up the apps, I was essentially going to have to redesign and develop most of the pages and flow from the ground up.
The code quality of the front-ends themselves was very poor. There were 3000 line SCSS files copy-pasted throughout the app. There were repeated functions everywhere. The naming was inconsistent. There were no patterns or libraries. It was messy and unclear.
The products we were building were intended to be used almost exclusively on the desktop, but for some reason we were building them using mobile technologies. I’m not one to pass up an opportunity to learn and get a “wide lens” of a project, so I asked Bob what the business reasoning behind using Ionic 2 was. Bob replied that he was running an “Ionic 2 / Angular 2 shop”, and that he didn’t want the other engineers confused when they also had to learn how to use Angular 2.
I pressed further —Bob’s reason didn’t make sense to me. Having to learn an entirely new mobile framework on top of an entirely new JS framework is a lot harder than having to just learn just Angular 2. Even the junior developers agreed — they had been struggling with Ionic 2 for quite some time and often spent significant amounts of time trying to do basic things with the framework.
Bob stated that we were using Angular 2 / Ionic 2 was there because the users would be using it on their phones. By creating an app we could provide offline queuing capabilities for the system in case connectivity was lost and that as a hybrid app it would prevent us from having to create both an iOS and an Android version.
Aha! A legitimate business reason…or so it seemed. I thought about the audience and the kind of system we were building — the business case of making it mobile just didn’t add up.
Our system needed always-on online connectivity to verify the validity of transactions and to provide immediate feedback on the result. Ours was not the kind of transaction we could queue and wait several hours to complete. The user had to know at the time of the transaction what the result was. There was no valid business case I could think of for offline queuing capabilities.
Our audience was also not primarily mobile app users. They were users at their desks on large-screen always-online computers, infrequent users, or users visiting the site for the first time and never visiting it again after completing their transaction. A vast majority of them would not be taking the time to download a mobile app to use our system.
So why were we making a mobile app?
The mobile focus leads to technical issues
Bob’s focus on creating a mobile app also influenced the architecture. The app was designed with the intention to load every transaction record on sign-in and store it in memory for offline support — searching, querying, reporting, calculations, etc. This data, along with all the other data in the app, was stored in a singleton reference that was then scattered throughout the entire codebase.
The problem —there‘s several, actually!
Problem 1 —the design was a giant ball of spaghetti
The first problem was the whole design was the question: how should we persist the transactions in an offline-queue capable way like Bob wanted? If we did pursue this direction, we would need some sort of mechanism to sync it sensibly — to keep track of changes, updates, successful and unsuccessful requests, errors, etc. To do so while dealing with a singleton modified and read from in every single file would be a nightmare.
I asked Bob what the plan was to sync and persist the data in a sensible way. He responded with the names of technologies like NoSQL and PouchDB in response without explaining anything beyond the name. I was confused — we were using Ruby on Rails and MySQL and had no intention to move away from a relational database — how did NoSQL or PouchDB factor into this at all?
Problem 2 — incredibly slow performance built into the design
The second problem was one of performance. Many users would easily generate over 70 megabytes of data a day! It was far too much to load even a week’s worth! Even with only a single day’s worth of test data, it took a good 10 seconds to sign-in! No sane user would wait that long.
I asked Bob about the performance issue and he responded that it “shouldn’t be a problem”. A reply like that was shocking — users nowadays barely wait 2 seconds to use a website, why would they wait 2 minutes? I even presented him with the math and actual numbers involved — he remained unconvinced. It was only later that he admitted that this was a problem, but he noted that he would solve this using Angular Universal. I couldn’t see how that was relevant — the data would still have to reach the client-side somehow.
It dawned on me that there was no business reason for using the technologies Bob chose. Even worse, our technology choice was actually going to become an issue down the line in accomplishing our business objectives — users simply wouldn’t use our system in the manner we were building it. We were designing and building a mobile app for something that was rarely, if ever, going to be used as a mobile app.
If the mobile side doesn’t work, at least the web side should, right? Wrong.
As the web side of the hybrid application, the picture was even worse. Ironically, for a project that took “mobile-first” quite literally, the web side was not responsive.
That wasn’t even the worst offender. One of our core functionality requirements was the ability for users to share their various pages across social media. However, in the SPAs this just wasn’t possible — any URL change would lead to a completely blank slate and the front-end would lose all state.
At the time, Ionic 2 didn’t support URL routing, and the Angular 2 router was undergoing frequent core changes. As a result, URL navigation didn’t route correctly. The browser navigation buttons (Back, Forward, etc.) weren’t supported — they would reload the application, as would any URL change.
Looking deeper into the codebase, I learned that all navigation was being done by programmatically and without history support. A link click would trigger a function that would crawl the DOM and look for a reference to a master Tabs element. Once found, the it would look up the current page’s tab index and add or subtract the number of tabs to move from the current tab depending on what page the link was supposed to navigate to.
In short, it was brittle, highly unclear, and not a good solution by any means.
No solutions — again
I asked Bob about a potential solution or design direction for this issue. Once again, he threw out the name of a technology without going into detail, claiming that he would use NGINX to solve it.
I didn’t understand how that would solve the client-side routing problem — having a manifest of the possible pages in NGINX was not a viable solution, and even if you made NGINX interpret the URL appropriately, the SPA would still needed to interpret the URL on the client-side and statefully keep track of it.
I asked for clarification, but Bob didn’t articulate his design approach.
He later produced a “prototype”, stating it had taken him a mere 15 minutes of development work to figure out:
It solved the issue of being unable to load a specific page on first load, but the client-side routing and history support remained unaddressed. To use Bob’s approach would have required us to write a custom client-side routing solution from the ground up. The resulting system would have been brittle, fragile, and highly unmaintainable — it would be re-inventing the wheel in the worst way possible. Since we were already months behind schedule, I believed it would’ve been a waste of the company’s time to do so, and certainly not a viable solution for any long-term use.
Those were just some of the technical issues that were plaguing the company. I wish I could say that the problems the company faced were purely technical, but I’d be lying if I did.
Bob had a way of managing and communicating that I would describe as passive-aggressive and controlling at its best. His management style didn’t go unnoticed — when he walked into the office (a less than 200 square foot closet), I noticed the junior developers would clam up and go quiet.
I spoke to both of the junior developers in private (they were very friendly and personable) and learned their true feelings towards the situation. Needless to say, it had to change quickly — everyone seemed to walk on eggshells whenever Bob was around.
I’m a stickler for effective team communication, so I set up Slack as a tool to give the team a communication outlet. This way, communication was still encouraged even if the atmosphere in the room itself proved too tense for the junior engineers to speak their minds.
Elliot gave me access to the Github repository for the backend to better acquaint myself with the product. Bob, who was in the office at the time, immediately demanded I be removed. When Elliot asked why, Bob stated that I hadn’t proven my velocity on the front-end code to be given access to the back-end code. Elliot reluctantly removed me, later asking if that reaction from Bob was a “power thing”.
No other engineer had access to any of the back-end code. All work was done by pointing our local front-ends to the URL of a shared staging environment. Bob’s reasoning was that he didn’t want “people’s PCs cluttered and confused with all the brittle legacy Rails back-end stuff”. He told me to base my attempts to communicate with the back-end on the front-end schema of the singleton object he had created (which, as I wrote previously, turned out to be wildly inaccurate on multiple levels).
This was a very strange, ineffective setup — even Elliot realized this, and he’s not even a technical person! Not having any documentation or visibility on how the APIs worked would make my job of actually integrating the front-ends and the back-end very difficult.
Making progress any way I can
Since I didn’t have much to work with, I decided working with the junior engineers would be mutually beneficial. I’d be able to help them understand better practices and the technology when I could, and they would be able to help me understand the various quirks and the general problem domain.
I was new to Angular 2, but the weekend I spent ramping up was more than enough to start contributing. Pairing helped us both, and we were even able to get past some issues they were stuck on.
During the time I was mentoring the juniors, I implemented a code review process through pull requests as a way to share knowledge and improve quality. One of my tenets of code reviews is to help guide people and teach them to arrive at the solution instead of giving the solution outright — it helps them grow as developers and ultimately leads to better decision making and improved code quality.
Bob didn’t seem to share my sentiments.
Later that day I received an email from Bob stating that while my mentorship was valuable, I was hired to code and that my continuing employment would be based on output from my desk. Bob stated that Transaction Software, Inc. was “still in the process of evaluating [my] performance as a senior-level software developer”. Read in his voice, I found it vaguely threatening.
I’ve never been threatened before with losing my job in my first three days of working. Concerned, I asked Elliot the CEO (who was CC’d on the email) about this, and he assured me my job was safe. I asked how I could possibly stay at a company if the CTO had already determined in the first few days that I’m not productive enough? Elliot finally gave me the rundown.
The org-chart is a mess
It turns out Bob was actually a contractor, and not the CTO as he had told me! Even more surprising: Transaction Software, Inc. didn’t actually have an engineering team at all! Elliot had selected the junior developers to hire, but Bob had convinced Elliot to hire them under Bob’s own consulting company, for “insurance purposes”.
Elliot paid Bob’s consulting firm extra to pay for the junior developers. In effect, Bob and the junior engineers were a consulting company working for Transaction Software. It was confusing — I was technically Transaction Software, Inc.’s first engineering hire.
Bob was previously attempting to negotiate a CTO position or an acquisition of his consulting firm by Transaction Software, Inc. I assume keeping everything in-house and under one’s control puts one in a better negotiating position for a buyout or acquisition, especially if a company relies on your firm exclusively for their technology needs.
Additional meetings with Elliot gave me a clearer understanding of what was going on. Elliot had started wondering why the projects were being delayed repeatedly. One key project in particular was already months in the making, and Bob had told Elliot time and time again that production delivery was “only two weeks away”. The front-ends had been delayed time and time again, and the clients for that project were already highly agitated and losing faith.
Elliot was paying Bob tremendous amounts of money and felt he was getting very little in return. I was hired, in part, to shine a light on what was going on — were all these problems a normal part of software development?
At the time I still felt the best course of action was to work things out with Bob to make positive changes gradually. Bob hadn’t demonstrated to me that he was experienced at leading or working with teams. He hadn’t yet shown me that he was particularly knowledgable about the principles of developing large web applications.
As time went on I continued what I was doing, replying politely and professionally to Bob’s various remarks and tried to set a good example for the team. However, I believed blindly following Bob would lead to the failure of the projects and the organization in a very short amount of time. I couldn’t do that — I had a responsibility to the company.
Turning the ship around
I spoke a lot with Elliot, explaining from my perspective the various technical, team, and cultural issues I was discovering. I proposed a new approach that would get the product up and running on a much faster and more aggressive timeline, in the hopes of meeting the deadline for a critical stakeholder demo.
I designed and diagrammed a workable architecture for the various front-ends. Instead of the giant singleton ball of spaghetti, I created a linear hierarchy of pages and components. I created a schedule that we could realistically achieve.
Elliot and I agreed to deal with Bob with “kid gloves”, tolerating any outbursts and passive-aggressive behavior until we were in a better position to make a decision regarding him.
A New Hope
I presented my plans to Bob and Elliot on Monday, explaining that with some focused engineering effort, the system could be done in less than two weeks of development effort in Angular 1 if the endpoints were in the state Bob said they were in. I also presented alternative approaches incorporating Angular 2 if needed, although I clarified that this would greatly extend the actual timeline.
Bob was visibly upset. We had a very long debate, with him making points that writing the code in Angular 1 would be generating untold amounts of technical debt because it was a deprecated technology. I agreed that Angular 1 was on its way out, but the Angular 2 / Ionic 2 combo he had built the product with just wasn’t ready for our use.
I argued that Angular 1 was stable, the problems were well-known and the solutions readily available. I greatly preferred the “devil you know” approach to risk management. I reasoned that with months of development time already wasted on the Angular 2 / Ionic 2 solution with no real progress, with more problems arising every day, and with deadlines far missed, it just made more sense to start from scratch and build it in Angular 1 in 2 weeks.
Bob argued that he had the solutions to the problems we were encountering. I begged him to tell me what the solutions were, or at the very least to point me in the right direction so that I at least knew he had a viable way across the river, metaphorically speaking. If there was a way forward, I would pursue it to the end. He didn’t articulate his direction and vision, only mentioning the names of various technologies.
Elliot stepped up and made an executive decision to go with my 2-week plan of writing the front-end in Angular 1. If we didn’t succeed, at worst we would waste 2 weeks, and what’s 2 weeks wasted when we were already months behind?
At this point Bob’s gloves were off — he was adamant I wouldn’t be able to do it and said I was living in the “land of rainbows and unicorns and fairies” and not the real world, where people had to deal with legacy code. I told him that I had dealt with monstrous legacy systems before, and that I understood the challenges and risks associated with them.
Bob said that he “washed his hands” of the project, and that it was entirely on me and that he would not be held responsible at all if anything went wrong. It wasn’t the best team-oriented attitude, but it was enough for the company to start succeeding.
I laid out the plan in detail — for the next week, we would follow my Angular 1 plan and wire it up to the endpoints needed for it. I made a list of endpoints I needed from Bob (less than 7), and we set to work.
Ace up my sleeve
I have to admit, I had a secret weapon coming in to that meeting — I had already built the entire front-end in Angular 1 over the weekend. It wasn’t smoke and mirrors, either. It had full data flow to JSON endpoint mocks and full user-interaction. Routing was fully working. The code was architected in a maintainable and extensible way. The styles were just like the design minus a few font-weights and bolds here and there. It was mobile responsive. The deployment and production scripts were built and working. Quality-wise, it was a production-ready front-end. All it needed was the URLs for the endpoints and some minor field changes in case there happened to be field naming changes from the anticipated schema.
The light reveals some dust bunnies
It turns out endpoints weren’t in the state we were led to believe. The front-end schema we were given to follow was wildly inaccurate — different field names, missing fields, different accepted schemas, and even different datatypes. I had to go back and forth many times with Bob to get the endpoints to function properly.
Some of the issues were normal synchronization issues that are a normal part of integrating with APIs. However, some were indicative of poor testing and quality assurance practices. One issue was opened and closed 5 times as various bugs kept coming up — it led me to believe that the code changes weren’t being tested thoroughly before being deployed to staging (where my UI was pointing).
Despite the difficulties we encountered, the Angular 1 plan actually worked! We managed to meet our deadline (minus some promised features due to missing / non-functional endpoints) and we presented the product to our stakeholders, who were very satisfied with the progress. They had almost written us off by this point, but the new momentum had renewed their faith in us.
Fixing the cultural problems
The sprint gave me hope that Bob would still be able to be a productive member of the team.
Although some of his actions and communication during the sprint could be interpreted as passive-aggressive (such as changing the names of our Angular 1 repositories and adding the word “temporary”), they weren’t outright antagonistic and the edge seemed to have dulled somewhat. The office didn’t feel as tense when we were working on this effort (although there were still moments), and things were actually getting done for the most part. There was actually a nice buzz of activity as we all worked together towards a common goal. It was a glimmer of hope for how the engineering team could be.
I played with the idea that Bob could be redirected and he could work exclusively on tiger projects or other lone initiatives without negatively impacting the rest of the team.
Unfortunately, it quickly became evident that this would not work out:
That was the straw that broke the camel’s back. Bob’s apparent lack of care and his refusal to cooperate or play nice with the team led me to recommended to Elliot to end our business relationship with him.
The Bus Factor
Unfortunately, this relationship was somewhat difficult to end: Bob had the keys to the kingdom. He was the only with any knowledge of the backend or access to the various production servers, the passwords, the critical technical accounts, etc. He had admin access to most of the systems, and in some cases was the only person with admin access.
During this time, Elliot had requested multiple times for Bob to sit down with me and train/provide deployment instructions, passwords, etc. Every time, Bob refused and responded that anyone who “knew what they were doing” would be able to figure it out in a matter of minutes.
It was likely a jab at how unskilled Bob thought I was. Multiple attempts to get him to sit down with us proved futile — there was always some reason that prevented the meeting— time, money, schedule, etc. We weren’t getting anything.
Even with the bus factor of one, we decided the best thing to do was to end it.
Bob is gone
After we ended our business relationship with Bob, we were basically starting from zero. No passwords, no instructions, no diagrams, or deployment instructions. We didn’t even have the latest deployed backend source code — the earliest version we had was from a month before I joined the company. We hadn’t even received any code or deliverables from the critical, constantly delayed project Bob repeatedly said was only “weeks away”.
Luckily, I was friends with the team at a consulting firm that was highly skilled in DevOps and system administration. We set up a meet, I explained the situation, and Elliot decided to contract their services. I worked closely with the consulting firm’s lead, Charlie, to figure out our server issues and get a handle on where we were as a company.
Starting with a crawl
Our first move was to retain access to our servers. We decided to start by resetting the staging server’s root password so we could do some exploration on how everything was set up. It was the least risky course of action in our minds —we didn’t want to touch production directly because we didn’t know if we could bring it up if it when down.
Good thing we didn’t touch production! The staging server’s application and its dependencies didn’t start up after we brought it back up. We had to dig around for a while in the command history, search through various installed applications, and dig through files until we figured out which commands to run on which files and in what order. We eventually figured out the commands we needed to bring the application back up properly.
The fires start up
As Charlie and I researched the environments and learned our deployment procedures, we received panicked messages from our biggest client, saying that records were showing up twice in the production environments, messing up key reporting. We traced it back to the staging server’s job runner.
Apparently, the staging server’s job runner was being pointed to the production database, which caused it to create invalid records. We quickly stopped the staging server and simply decided to bring the whole system down, unsure of what other traps were lying in wait.
Over the next couple of weeks, we utilized Charlie and his team heavily, obtaining control of our servers and figuring out various account and network issues. We set up a secondary deployment procedure because we simply couldn’t risk changing the initial production server’s deployment. There was a month of untested code in the latest branch, the dependencies were old and hard to find — some weren’t even available online anymore.
A terrible discovery
At the same time, I was resetting the passwords for the various accounts that were being used by Bob for Transaction Software, Inc. — email accounts, 3rd-party vendor accounts, etc. We didn’t have a list of them, but there were easily several dozen.
During this process we discovered that Bob had actually deleted Transaction Software, Inc.’s Bitbucket repository! The deployment setup was partially relying on it and the latest version of the code was on it! Worse yet, this had been done this AFTER we had decided to terminate our business relationship with him!
We lost everything in that Bitbucket repository, including any wiki pages, documentation, past / existing issues, and code versions. No passwords, no recent codebase, no roadmap. The latest version of the code we had available to us on Github was from before I joined the company.
I had never heard of a contractor destroying the assets of the company he was contracting for. I guess there is a first time for everything.
In a pure stroke of luck, I had discovered that the staging server had the latest source code on it. I tarballed it, downloaded it, and then pushed it up to a branch on the company’s Github. The work was saved — or at least some of it.
Like a bad penny
I wish we could say that was the last we saw of Bob, but unfortunately it wasn’t. Bob sprang up again some time later, this time with outlandish IP claims.
He claimed that all of the code and software he wrote was his intellectual property, despite multiple references in his past invoices that it was work for hire. He demanded payment of his final invoices, despite the fact that some of the projects he was supposed to have been working on were never delivered to us at that time.
Several back and forth and multiple attempts to resolve the issue without getting lawyers involved proved futile. Both sides eventually got lawyers.
When we did finally receive the code for the missing projects, it became clear just how much we overpaid. The critical project that was “2 weeks away” for months? It consisted entirely of an automatically generated rails scaffold with a few files taken straight from easily discoverable online code samples. None of it was usable for any real world, production use. It would’ve taken anyone, even someone who didn’t know how to code, mere minutes to recreate.
This was too much. Elliot wanted to sue him into the ground, no matter how much it cost. I recommended we move on. We had already incurred significant costs trying to fix Bob’s mistakes and errors. Wasting more time, energy, and money would only serve as a distraction. It was simply easier to stop wasting time on Bob. We had a company to build, and we wanted to focus on that instead of fighting silly legal battles or dealing with Bob anymore. If Bob used or further laid claim to our IP, we would defend it. Otherwise, we would ignore him. He wasn’t worth our time or effort.
We decided to fully pay Bob’s final invoices and left it at that — a painful lesson learned.
This was a story about my first two months at Transaction Software, Inc. It was absolutely brutal, and if I had known what I was getting into I wouldn’t have shown up to the interview.
However, this story has a happy ending. Starting that low meant you could only go up from there. After the initial rocky start, we planned and successfully executed a turn-around for the company, working hard and smartly at removing the old baggage and creating a new future. I’m proud to report that the company culture turned around completely, and it actually became a fun place to work for everyone!
You don’t always know what you’re getting yourself into when you accept a job!