Beyond the Black Box: The Risks and Rewards of Abstraction
Programmers are no strangers to the concept of abstraction and while another field might not know it as a word, I’m sure they use it every day. Abstraction is the concept of building a black box and reusing it, building on top of it while hiding information underneath. Think of it as a black box where you know what the inputs and outputs are, and you’re free to use it without needing to know what’s happening inside them. In Electronics, you don’t need to exactly know what’s happening inside an IC in order to use it as long as you know what it does and how to use it. For manufacturing, you don’t need to know how the machine exactly works in order to use it. In cooking, I don’t need to know the exact spice mix as long as I know how it’ll generically make my food taste.
It’s fair to say, without abstraction we would still be hunter’s gatherers. If you need to know exactly how each thing works, there is not much you can comprehend in your short lifetime.
This reminds me on an incident. I was on a trip recently with a couple of my friends when we were discussing the subject of cooking. One of my friends comment on my cooking style saying “You don’t exactly cook, you assemble. You get premixed spices, pre-cut veggies, pre-baked bread, and sauces; all you do is mix and heat those. That’s not cooking.”. To this, my other friend replies “Okay then, tomorrow onwards we will go to the farm and grow our own cereals, spices, and veggies.”.
Abstraction is basically a trade-off someone chooses for “Time and Effort” vs “Performance”. If you build everything from scratch, you might have the most performant system but you’ll also spend so much time and effort creating and optimizing something which already exists. Having said that, many times you might need to reinvent the wheel and build a couple of blocks for performance optimization. Also, it is just fun to understand how each block work and make it. There is fun in cooking from scratch and writing small problems in assembly and building computers from gates not to mention, it gives a very detailed understanding of things that might be used somewhere in the future.
Checkpoints when using abstractions
When you work in System’s Engineering, integrating various blocks and solutions developed by multiple entities and creating a few of your own, few things become important.
- Determining which parameter is important for your application and which parameter you can let go of.
- Evaluating the performance of various blocks so that when you do have to optimize your application, you know replacing or modifying which block will give most drastic effect.
- Using generic blocks which are available from various vendors/sources to derisk them going out of stock or losing support.
Abstractions as source of bugs
Abstraction introduces bugs when we don’t know what’s happening under the hood. If you just deal with abstractions, not knowing how things work can leave you in dark for the performance matrix and a solution that is highly suboptimal.
Recently I was working on an Embedded C project where I was facing a bug. My uint16 type variable was getting coupled with another uint16 variable, whenever the second variable changed, the first variable also unwantedly changed. The root cause turned out to be compiler optimization settings where the compiler was bit squeezing these two variables into a single 32-bit register to save memory. Closed source software/blocks are a huge source of bugs that can leave you debugging for a good amount of time without even concluding the root cause.
Dark Side of Ease of Use
There are two general approaches when someone starts to learn programming. One is to start with lower level programming language like C and build there way slowly to C++ and Python. Another is to directly learn Python. There have been debates on which of the approach is better with relevant points raised from pro Python side like “Why should I learn C if I’m just going to use Python here after?”, “It’s so difficult to understand the syntax of C compared to Python”. To answer this, learning from scratch does give programmer overview of how things happen under the hood giving them better chances of solving bugs like
These issue are further amplified by the rise of graphical programming platforms like LabVIEW, NodeRED etc. Where things are even more abstracted.
With the rising age of AI, there have been increasing debates around the pros and cons of it. Though I am pro AI, one issue I do see if we become increasingly reliant on AI is the future generation won’t know how things work at all and will introduce tons of bugs and sub-optimal applications in the market. As the barrier to entry in any market become low, the number of low-quality applications increases drastically. This is evident enough on the play store with 1000’s of apps to do the same task making the paradox of choice even greater not having any metric for the quality of the app.
Conclusion and Call to Action
I watched a video recently on how all of the human race forgot the method of creating an important compound for decades and had to spend millions in recreating it. Though the root cause over there was a lack of documentation, I could potentially see similar incidents rising in frequency in the future. There was an article mentioning recently how only a couple of dozen people around the world truly understand how RSA encryption works. The algorithm, is so important that our whole internet infrastructure is based on it, if something goes wrong, there soon might not be anyone who knows enough to fix it.
With the rise of ease of use tool, educating the next generation for basics is becoming increasingly difficult and is a problem which needs to be solved soon.
Another shift in mindset would be on moving towards more open source software giving developers further understanding of what’s happening under the hood.
Finally, without black boxes, no development would be completed. Having said that the performance matrix and test results while releasing any block would be essential to ensure that the abstraction layer is working as intended. The trade-off between the convenience of abstraction and the necessity of understanding the underlying mechanisms must be carefully managed to prevent potential system failures or inefficiencies. The performance metrics and testing outcomes not only validate the functionality and efficiency of these ‘black boxes’, but they also provide valuable insights for future improvements and adaptations. Therefore, although abstraction can significantly accelerate development processes, rigorous testing and comprehensive understanding of performance data should never be overlooked.
Credits and References
- A blog on Embedded System optimization where closed source software cost a company lots of effort in optimizing their solution
