Is a costly developer’s mistake worth firing? I guess not, but management was bloodthirsty…
In England, everyone is obsessed with soccer. The country is full of stadiums and amateur teams. One day, an English guy found me through my friends. He had some money and an idea for an app that allows people to manage their own amateur soccer teams, including arranging players’ field positions, making substitutes, booking stadiums, and interacting in chats.
It was a weird part of my life and now I can barely explain how I managed to survive at all. I was at university during the daytime, worked at night as a back-end developer with shifts until the morning, and I took android development courses somewhere in between. Surprisingly, I had plenty of energy and enthusiasm to do it and I was not merely present but was actually driven and inspired.
So, I accepted the project and it became my escape from all the things I’d gone through working in back-end development.
At school in our math classroom, we had a sign on the wall which immediately caught my attention. Later on, I would often discover the truth of these words:
The program doesn’t do what you want but what you’ve written
When I started university I immediately stated to professors that I want to be a programmer. I was assured that I would be. They added, however, that in order to become a programmer, I would have to commit to lifelong learning. Somehow, having interpreted it in my own way, I didn’t spread the studies over an entire lifetime, but instead, I compressed all the information to speed up the process: university studies, evening courses, and overnight freelance work.
My search for a part-time job led me to an interesting project: C++ development of a speech recognition engine. Together with a team, we fought for every percent of speech recognition and I was dedicated to actually getting the machine to perform with 100% results!
It was a perfectionist’s nightmare. I created my own solutions, used ones from open-sources, deconstructed them down to the tiniest particle, and then assembled it all over again. I customized the code throughout sleepless nights and was barely awake during the daytime lectures. I painstakingly untangled the endless code for the sake of achieving just a few more percentage points in the end.
Nevertheless, it did not work out. Only a couple of years later I found out that it was impossible. I stumbled upon an article about Facebook’s neural network that achieved up to 95% accuracy in face recognition. I felt sorry for Facebook. Even a huge company like this could not make it to 100%. Yet the article featured an amazing caption saying that humans are capable of face recognition with 93% accuracy. The goal was not to achieve 100% but to beat human face recognition. Are you kidding? I want my 100% results.
It was my first disappointment in the back-end development
Later I tried front-end. Everything I knew about it was within the stereotypes of website-making. It seemed terribly boring from the very beginning. I craved for something more. So, I set myself off for Skolkovo to find new things there.
The Skolkovo Master Program came with a good scholarship. But if you want to live at a higher rate in Moscow, then you have to earn more than that. I found an open position in a finance company. The requirements were writing the back-end in Python and taking some night shifts.
During the interview, I answered questions on multithreading and wrote a C++ code on paper (this one was hard). Additionally, they asked for some Python-related things and approved my application.
Previously I worked only in small teams. There were 5 of us working on the speech recognition project, while this company was full of developers. And all of them were better than me. I asked them loads of questions and sincerely enjoyed receiving feedback on my work. The first 3 weeks were all about digging into the craft. Sometimes I had to redo the same task and it wasn’t getting any better. My code had to be approved by at least 4 colleagues. Sometimes 3 of them approved, while the 4th one was attentive enough to unveil my flaw.
It was the place where I discovered a great review method: I passed the code to the Team Lead, he highlighted some fragments without saying a word on what the issues were and I had to find out myself. This way you have to start thinking, and that’s what actually develops you as a professional.
Yet even such supervision did not keep me away from the trouble.
The work was related to databases. The servers stored a massive amount of clients’ financial information (bank cards, transaction history, etc). Every 5 days I had to spend a night at work. I made sure that everything worked and replied to new emails and phone calls. I enjoyed the night shifts. The night shifts were paid triple the regular shift pay. I stayed on duty from 9 pm to 6 am and was paid a 3-days salary. And what else would a student need?
Apart from money, students need some sleep
All of the processes in this company were flawless. Check-ups, tests, reviews. Check-ups, tests, and reviews again. Though even systems like this can fail and one should prepare for a sudden alarm to interrupt the peaceful sleep in the middle of the night.
One day I came back from a night shift with a very strong desire to sleep. Yet I went to university, pretended to be a student there and In the evening I fell asleep before 8 pm. I wanted to sleep forever.
But at 4 am I was woken up by the person responsible for the entire company backlog. It was the first time he ever called me.
- Have you done anything to the database yesterday?
- No, nothing special.
- Get up and hurry to the computer.
Well, I’m an early bird, still, I was shocked by this call. My night shift was just the day before the accident.
First thing in my head: Sure thing, I fucked up everything
It turned out that an entire group of users disappeared. It’s not possible just to delete the users. I started searching. I looked through the logs to check the activity and see what scripts were launched recently. I spent 30 minutes to wake up, another 30 to get into the logs and understand which scripts were working. I downloaded all the scripts, looked through all the activity, and found out which one led to the issue, and finally located the error.
I launched the script in the test environment and the test users disappeared. I wondered how the script was tested. We had three development test stands. For some reason this time the script has never been run on the first and second. The third one was responsible for user database consistency, nevertheless, it did not show a sign of errors. Even more — our team had no test written to check it (so I put a note to myself to write one so that it wouldn’t happen again).
At the same time, a lot of colleagues constantly had team calls as they were looking for the same problem I did. At 6 am we realized that someone had written a script and tested it without taking one important detail into account. As a result, a group of users was scheduled to be removed.
So we lost about a million users overnight
We restored the database from the backup. Until we found the problem, users pushed new data throughout the night. To stop the flow of new information we informed everyone about the technical problem and forbidden new logins. We had to merge the databases. Restore the database and fill it with the new information of the last 7 hours. The company apologized and gave bonuses to the users who were involved.
By 7 am we fixed all of it. At 8 am the company fired the person responsible for this.
He was a good guy who worked well and always carefully checked everything. A couple of times he even reviewed my code and suggested really great improvements. It’s more like driving. The first year you are afraid on the road. Then you relax and become more confident and this very moment you get into an accident!
Would I fire a person for a mistake? “The program doesn’t do what you want but what you’ve written”. However, developers constantly make mistakes. All the time. Imagine how terrible this guy’s morning was. And why fire a person who now will check every single symbol of his code? But the management was bloodthirsty. During the following two weeks, the door location was shown to a few more people involved in this case.
I worked there for a few more weeks, already knowing that I want to leave the back-end.
Skolkovo was the place where I realized how interesting Android development projects are. We developed an application and connected it with Arduino. It was a quite simple task. We had to connect with Arduino via Wi-Fi or Bluetooth from the phone and pass the command. The application had a couple of screens with buttons. Arduino was connected with the smart home prototype and the buttons were to close and open its windows.
Android is full of reasons to fall in love with it
I really liked to work with UI. I could do fancy animations (now I like it the most). Google’s documentation was much better than anything I had seen before. I enjoyed Java much more than Python or C++. While I was taking Android development courses, I realized one thing:
It’s important for me to see the result.
It also became easier to explain what I do at work. Try explaining to someone what a back-end job is. Why is it important, what is it for, and so on? With Android, I simply take my phone and show the application I have made.
So, back to the soccer thing. I was given a project written by bad developers. Honestly speaking, it was miles away from a job well done. Instead of creating a function, they copied 700 lines of code again. Only refactoring took me two weeks.
I managed and really enjoyed the work anyway. I realized that I want to keep on doing it and started to look for a company to work for. Got a certificate from e-Legion’s Android school, successfully completed the coding test assignment, got an offer and now I’m the Head of Android Development.
In the end, I’m glad to know that no one would ever wake me up in the middle of the night to rush and resolve the emergency situation at 4 o’clock in the morning and write a list of people to be fired after that.
