Profiling Series: Profiling Skills > Intuition

Screenshot from the video embedded into the post and provided by TopMovieClips under Fair Use.

Let’s talk about life mistakes. More specifically, let’s talk about fateful life mistakes. And even more specifically, let’s talk about the mistake of underestimating the role of debugging and profiling in software engineering. There was this little story-joke, telling about a mom coming home from hard work and asking her three daughters whether they had done any housework to ease her evening load:

[Eldest daughter] Yes mom, you won’t believe, but I’ve done all the cooking!
[Middle daughter] You won’t believe, but I’ve washed all the dishes, mom!
[Youngest daughter, cryingly] Didn’t have time to do anything, mom… Was busy with cleaning up the havoc caused by the chicken soup spilled and plates shattered all over the place.

Little did she know that what she did was of an equal (and, dependent on a project, even of a higher) importance and complexity. Debugging and profiling. Activities within the repertoire of a software engineer which are good indicators of how seasoned she/he is. Despite that all too often you’re gonna see engineers who mostly stick to simple debugging techniques, never investing their time to master the right schemes and tools. Things are even worse for profiling, where some go so far as stating something along the lines of: “I feel the system, I feel how it breathes, and know exactly what happens if something goes wrong, so I don’t need extensive profiling to fix the thing eventually.”. Well, unfortunately complex systems are more subtle than that, and the number of possible internal disorders just cannot be simply mapped to different observable outcomes. Hence there will inevitably be times when they fail, and fail hard. I could probably go on with presenting the story of the notorious Therac-25, but there didn’t happen to be a good excuse not to come up with something I have gone through myself long time ago, feels like in another life. Although it’s not exactly a software system, it’s still quite interesting and instructive.

Captain America: Civil War, the scene with Tony Stark debuting B.A.R.F. as a therapeutic device to facilitate reprocessing of traumatic memories. Love that scene. For those who don’t know, here’s a little spoiler: Tony’s parents are going to die in a car accident shortly after the conversation presented in his deliberately adjusted retro-frames. Watch the scene, it’s worth it.

Tony Stark presenting B.A.R.F. at MIT when announcing his September Foundation Grant. The video is provided by TopMovieClips under Fair Use.

Sometimes I’m brave enough to imagine it’s me who’s presenting the thing with my own retro-frames, again, adjusted deliberately.

A family of four — mom, dad, sister and brother, getting into their car. The brother hesitates for some time, unsure whether he should go or stay.

[Sister] Are you not coming with us?
[Brother] Umm, Anna, I have passed the first stage of my interview, now I’ve got this interview problem, which I should solve and submit during today. I will join you guys at the hospital later today, OK? Please do not worry, everything is gonna be fine.
[Mother, getting back out of the car and approaching her son] She will miss you at the hospital. And frankly, you’re going to miss her. Because this is the last time we’re all going to be together. You know what’s about to happen.
[Brother] *nods obediently*
[Mother] Come with us. If you don’t, you’ll regret it.
[Brother] *readily gets into the car, taking a sit beside his sister*

[The scene freezes] That’s how I wished it happened. Binarily Augmented Retro-Framing or B.A.R.F. (god, I gotta work on that acronym), an extremely costly method of hijaking the hippocampus to… clear traumatic memories. Huh. [He blows on a candle and everything around him shimmers, then starts to dissolve. It’s a hologram.] It doesn’t change the fact that my sister never returned alive from the hospital… or all the things I did to avoid processing my grief, but…

She was diagnosed with thyrotoxicosis. Now, thyrotoxicosis is not a low-level diagnosis — it’s more like a symptom having a bunch of possible underlying reasons, some of which are: Grave’s disease, inflammation of the thyroid gland, excessive iodine consumption. One interesting possibility, which I was told about much later, was prolonged excessive stress, which sounded quite natural to me: we were both going through our second job interview rejection, and I remember how serious she was about getting a job offer, and how disheartened she got after each rejection. The physician who was conducting the case picked Grave’s disease as the base: he was able to convince us, although there were no explicit signs, like enlarged thyroid, or Graves ophthalmopathy, not a slight bit. Regular drugs were prescribed, and whatever it was, things started to improve. Overall the physician was probably right, and his guesstimates started to work as expected. Where we all failed though were these lines extracted from the Wikipedia article for Thiamazole, which should be read with Dr. Hawking’s synthesized voice to engrave deep into memory for the rest of the life of all the participants:

It is important to monitor any symptoms of fever or sore throat while taking thiamazole; this could indicate the development of agranulocytosis, an uncommon but severe side effect resulting from a drop in the white blood cell count (to be specific, neutropenia, a deficiency of neutrophils). A complete blood count (CBC) with differential is performed to confirm the suspicion, in which case the drug is discontinued.

I had gone through some stats back then: that particular side effect of thiamazole may happen in one case in two to three hundreds, hence the risk is unacceptably high (taking into account seriousness of the side effect). And yet no regular blood count tests were conducted. Furthermore, fever and sore throat were misinterpreted by the physician and all family members, especially me, who could plainly go through some docs online or even offline. Eventually WBC count dropped beyond the point from where medicine could bring her back (at least in the country we lived in). Put simply, she got poisoned by a common drug. In the 21st century. AD, not BC. How likely do you think it would happen, had we gone through regular blood “profiling”?

So, back to software engineering. Software systems will grow in complexity with time. One can anticipate disciplines like debugging, profiling, and benchmarking to massively grow out into separate steps within CI/CD pipelines of a growing number of products with time. Imagine debugging the pipeline right during its execution with the debug window incorporated within the web portal of the pipeline itself. How about getting a detailed list of CPU/memory intensive functions throughout the application, automatically compared with the previous run? Pretty sure you’re going to find such pipelines here and there even now; the thing is that those are not common and widespread. To achieve prevalence and build the right means, we’ll have to learn utilizing the building blocks first — standalone debugging and profiling tools, something a vast number of software engineers is surprisingly bad at.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store