What Does Mastery Look Like in Software Engineering?
Compilation of a QnA on the Hacker News Forum
This is a brief analysis on the the question popped up on the Internet’s Intellectual Dome — Hackernews (news.ycombinator.com) on the mastery of software engineering. Here is the discussion for your reference.
The perspectives are all over on a general mastery spectrum, some developers are very much taken on reducing complexity of the codes, some focused comprehension of the technological breadth and depth, some are beyond any sole technical measures — considering a measure of mastery into expertise in all relevant parts and processes just as well as the technical proficiency. Some developers even provided a higher level optics to factor-in all opinions. Although technical proficiency is what comes to everyone’s mind when we talk about mastery in software engineering, you will be amazed at the areas of focuses which the comments had explored. It’s an intense discussion with personal experiences, what they feel is right, why they like it certain way and unproductive experiences they’ve experienced. While HackerNews is a mecca for developers all over the world to browse through the technology news and discuss various topics under AskHN tag, you should know these observations and my compilations aren’t scientifically rigorous, it’s a clash of ideas and a narration of an exploration in to the unknowns of a mastery.
Tools and Technical Proficiency
According to a contributing developer, achieving levels of mastery is based on developer’s ability to develop a library or a tool which is used by the developer and there should be capacity to imagine new tools and visions to advance the knowledge [1].
While it’s true, software engineers do tend to focus on the expertise of frameworks on the application side of it rather than the theoretical side of it. Diving deep into the Language, frameworks and libraries we are using needs a disciplined, focused, deep work in to the language and it takes a lot of time.
But there’s also flip side to that particular focus. According to another developer who is commenting on such technical proficiency talks of the flip side, says that (Macromedia/Adobe) flash was a technology at the peak of his time at an ad agency where the developer developed websites, games and was really very proficient at it but since unable to switch over successfully to React soon enough, which are component based development platform with different Javascript than flash’s Actionscript, his proficiency is all but became a background knowledge [2].
Flash which doesn’t fit any programming or deployment paradigms we work with these days, it definitely would been a hard transition but the point remains as the time put in to specialising one self in a technology is a time lost on general understanding in many. Today, many languages differ on syntaxes converges on many layers above application like CI/CD, Devops, Containers and cloud. Unlike flash days, Today skills would be much more transferrable across a lot of paradigms and silos. While no developer lays down any hard and fast rule on depth vs. breadth question on technological expertise, we can assume it’s up to the developer to take it up as necessary in consideration of oneself’s interest or working organization.
While senior roles may demand generality by having to lay out the solution options in a bigger tray for a problem, putting your focus on a speciality has a reward of it’s own, — imagining trade-offs. And, What really are trade-offs?
Regarding technical proficiency as the ladder to mastery, comments in the discussion has an interesting resonance — tradeoffs.
Mighty Trade-offs
If you can do something better with trade offs in technologies, solutions, stacks or even algorithms on their weaknesses or strengths, you are correctly imagining a solution curve in a problem space says a contributing developer [3].
“expert engineers are able to avoid work that does not provide value and this could be their greatest skill”
That imagination is not a given but gained. To know a trade off, you should be well aware of what’s right, what’s wrong, when is it right and when is it wrong in a proposed solution, so that you know what to compromise, what to keep and what to add. To do that, It takes a lot of experience, in a lots of different settings, with a lots of different ideas and a lot more of trial and error. According few other contributors, Most of the proposed solutions are wrong and a few are correct with different tradeoffs. So, Novice engineers will struggle to get something done, running with the first solution that comes to mind, letting confirmation bias drive them (WKPDA: Confirmation bias, is the tendency to search for, interpret, favor, and recall information in a way that confirms or supports one’s prior beliefs or values), while Intermediate engineers can get things done but fail to see the tradeoffs and an Expert engineer can identify possible solutions with different approaches [4][5].
Many trade offs are thrown out in the discussion — Should we compromise accuracy for cost or complexity and What situations do we do that?, Some situations may demand speed over precision, while another situation may demand performance over extensibility. Some solutions might be easier to change later than others. It’s also depends on the business your organisation is running, According to a developer, You can’t master Software Engineering without mastering interfacing with the rest of the organisation. I don’t care how intellectually pure your implementation is or how fast you deliver things. If it can’t easily connect to, use and get used by the rest of the business, it’s a folly [6]. The developer comments, That’s what an expert engineer does: ask the right questions and identify the right trade-offs, and they’ve become efficient in that process from experience [7].
The proposition of mastery in tradeoffs aren’t the only defining factor in gaining advance expertise in software engineering. An expert engineer can be a step ahead with involvement in problem itself. According another developer, Expert engineers have the ability to think about a problem in a broader context than the engineering problem at hand [8]. Exploring the problem, asking the right questions at the problem, even refining and streamlining a problem. It could be the starting point of the journey into the solution curve of a problem space.
While trade offs and problem definitions resonates with the contributing developers as a factor in mastery, another metrics is floated around as a measure of a sorts.
Shipped to hours worked
What do these people have in common, a developer asks,
John Carmack, developer of Wolfenstein 3D, Doom, Quake and many other games.
Jeff Dean, currently head of Google AI, His work spans across MapReduce to TensorFlow.
and Bram Cohen, Author of BitTorrent Protocol.
The common characteristics across these highly accomplished programmers who are leading a legion of programmers in their respective fields, as developer outlines, are extremely effective at getting things done, hardworking, and specially laser focused on shipping [9].
It’s a metrics which you aren’t going to find in many programming skills development posts or guides. What would be laser focused shipping?
With extremely high ratio of value shipped to hours worked (With less hours working, producing more value), expert engineers are able to avoid work that does not provide value and this could be their greatest skill [10]. The very best ones can direct a whole team away from work that’s not valuable, a developer commented. However, as that metrics seems sensible as it is, it did have a wall around it.
That is a position of privilege that you’re able to push back against non-valuable work, I’ve seen many good engineers that have this quality but gets overruled and their true value goes unrealised as a result, replies another developer [11]. Although many has commented that position of privilege is very rare in organizations where they completely trust your opinions, provide you the freedom to do away with what you consider non-valuable work and not affect their organization goals, there were ways commented of how that privilege is earned. A developer comments, that position of privilege is earned, when a senior engineer knows what the highest-value work is and is influencing/driving the roadmap. A Senior Engineer says ‘no’ more often than ‘yes’ and backs it up with a ‘why’ [12].
Expert engineers enables organisations accomplish more with less and keeping it from falling into different pitfalls, however this is hard, responds a developer, because it gets harder to convince management of future issues if they haven’t manifested yet [13].
While this metrics can seem very interesting, a laser focused shipping metrics might prove counter-productive for an aspiring junior engineer whereas exploration is a necessity. But there’s another interesting twist in software mastery which many developers had discussed.
Spiralling Complexity
Mastery is all about managing complexity, many developers agree in the discussion [14][15][16]. Masterful code has some kind of straightforward simplicity, everything is predictable and boring. You can see how programmer was thinking and how well program structure was planned, a developer commented [17].
“software engineering is way too diverse, fragmented, chaotic, and ever changing”
An expert engineer is in control of the complexity arises in all walks of development. Another developer comments on the complexity as a mastery metrics, They are able to figure out what the irreducible complexities are, whether it’s in implementation, maintenance, algorithmic, scalable complexities, and what tradeoffs these complexities could be addressed in different components of an application. Masters can more easily put the complexity where they want it, having more control over trade-offs and they will have an easier time estimating the required work needed to implement a project, as experience gives them a good intuition for what the irreducible complexity of a problem is [18].
An expert engineer helps others to understand the code faster by identifying and removing most of the accidental complexity. Accidental complexities can range from small structural issues to large scale architectural shake-ups. But a master works through that, continues another developer, they are able to build systems such that the cost of adding more value to the system remains constant as the system grows in complexity [19].
As much as this discussion on mastery goes on, there was another idea on the sea of comments which really will make you think about all these in a different perspective.
Journeyman
There’s no mastery in this line of work, says few developers. Mastery is difficult to quantify, especially in terms of code or engineering cleverness. There is just too much change and evolution in this field of software development for that to ever be true[20]. My goal is simply that I continue to improve, says another developer [21]. It doesn’t matter how much effort I put in to keep up to date with thousands of moving pieces in this industry, It seems like every single action that we take comes with trade-offs and that we can never make an ultimate good choice, software engineering is way too diverse, fragmented, chaotic, and ever changing [20].
With many developments on the daily basis, in the libraries and tools we use, frameworks we develop in, languages we are comfortable in, many best practices and automations we see every day, it’s actually natural to be overwhelmed. As much as we would love to see these developments portraying millions of engineers committed in to this industry and communities, there should be something stable and fixed to hold on to, a line we should establish for our journey in to these engineering wilderness.
“Find as many solutions to given problem as you can”
While another developer says mastering software engineering is doubtful, there’s a ‘Journeyman’ level of software development and engineering skill [22]. We are to go through a journey, as a developer describes it, there’s no such thing as master but it’s the proverbial journey. There is however a baseline at the other end … if it’s not functional and maintainable, there’s a definite lack of mastery [23]. Another developer takes a philosophical and scientific approach to question the premise of mastery, Can anyone claim to be a “master” of an area in which there are such large known gaps in everyone’s knowledge?. And in some ways, it’s much harder to “know” that an engineering approach or paradigm is “right”. How can we make types track the “important” invariances of a system? How can we convince ourselves that a distributed system can guarantee certain properties, or that a modification to that system doesn’t break those guarantees? [24]
That certainly makes sense.
The wise Owl
In the discussion on the forum, many a wise counsel was offered as to how to ascend to mastery if there ever was one, and many are interestingly what we come across in our daily lives and some are obvious but can seem like a reminder.
A developer motivates, You don’t need a ton of experience to practice it. What a mentor did to our team: ask the entire team list all possible solutions to a problem, including the obviously bad ones, then have us whittle down the list based on pros and cons of each until we reached consensus on what to do [25]. While this can seem a heavy state of affairs for the mundane problems, it could bring out the beautiful subtleties, burry worrisome fears on implementations and even with time, to be able to calculate the trade offs mentally.
You can never truly be perfect, you can only approach it. Understanding your limitations is just as important as understanding your capabilities. Depth vs breadth is hard and you only have so much life to go so deep into so many domains, a developer commented. Working with experts, Writing and deleting lots of code, Participating in code reviews and Reading even more code from others will definitely help you there [26]. Another developer advises, You need the experience of shipping a lot of programs and thus you should optimize your career around shipping as many good programs as you can [27].
A tone of reliability sure rings in many comments, the best engineers produce fewer bugs than they fix and rarely end up in a “whack a mole” situation where fixing one thing breaks another thing [28], while another commented, Consistently and reliably delivering good software solutions. Consistent, It wasn’t just a fluke or luck, you know what you’re doing and can repeat it [28].
Software engineering is a team sport, many seems to agree, literally [29][30]. You need to know how to write design documents, test plans, architecture plans, progress reports and documentations in all relevant areas of engineering. If you can’t effectively communicate your designs or strategically communicate even why someone else’s thing is bad or good, you won’t be able to convince others that you are right [29].
Find as many solutions to given problem as you can, a wise developer says [31].
Many online resources were referenced in the discussion:
https://faculty.washington.edu/ajko/papers/Li2019WhatDistinguishesEngineers.pdf — What distinguishes great software engineers?
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ — The Joel Test: 12 Steps to Better Code
https://archive.is/NCWmz — Wolfenstein 3d iPhone development By John Carmack
https://www.infoq.com/presentations/Simple-Made-Easy/ — Simple made easy by Ricky Hickey
https://www.ted.com/talks/linus_torvalds_the_mind_behind_linux/up-next — Ted talk of Linus Torvalds
https://www.youtube.com/watch?v=JjDsP5n2kSM — Jonathan Blow — How to program independent games
https://twitter.com/Gankra_/status/1046438955439271936/photo/1 — Mozilla Senior Roles Job Description
https://www.bti360.com/what-ive-learned-in-45-years-in-the-software-industry/ — What I’ve Learned in 45 Years in the Software Industry
All comments, contents referenced are copyrighted to YCombinator.
