The Expert Programmer

I’ve been curious about the differences between novice and expert programmers for quite a while. One day, given time and experience, we can consider ourselves to be an ‘expert’. In the tech industry, becoming an expert happens rather quickly. Namely, after about 3–5 years you can be promoted to a senior programmer without any real test of your knowledge and ability. Yes, companies do this for many reasons:

  • We don’t have money to give you a raise so we’ll give you a promotion instead.
  • Instead of taking that tempting offer from another company, we’ll give you a new title.
  • You are now responsible for mentoring other developers.
  • You’ve been around long enough and now you’re the guy who knows where all the skeletons are hidden.

Beyond retention strategies, there is great pressure to ‘be the best’. Every company is competing for a limited supply of talent and each developer wants to get their own piece of the money pie. So, we endeavor to become experts.

The Young Savant

The tech industry is obsessed with the young prodigy. The fresh face straight out of some top university that can ‘effortlessly’ churn out hundreds of lines of code. With no mention of quality, let’s assume that this code is bulletproof. Without extensive guidance and mentoring, how can a young person have enough perspective to tell whether this code will satisfy the deeper business requirements? In other words, how can a young person have enough perspective or wisdom to make important decisions for themselves?

Surely they can produce code. Perhaps lots of code. But how can we be certain that this code will work or simply be useful in the future?

It has been well researched that we simply do not ‘mature’ until we are about 25 years old. Namely, our brains are unable to understand long term consequences until we simply grow up and our brains become ‘activated’ to do so. In the meantime, we’re walking around with a limited perspective on the long term consequences of our actions. I imagine if you’re reading this article you’ve moved past the short term reward period of your life into a more balanced perspective of a fully functional brain. So why does our industry fetishize youth? Is it preferable to churn out an excess of code without thought for the future verses some careful planning?

If I were a manager or HR professional, I may mitigate the risks of hiring young yet energetic programmers by deliberately creating a system of checks and balances. Namely, having a more senior programmer around to check against the innocent yet un-implementable ideas of less experienced programmers. This is exactly what Matt Briggs argues in his article The Role of a Senior Developer. He goes so far as to state that your project is doomed without having a senior developer on the team.

Spectrum of Competence

More importantly, we don’t know what we don’t know. Especially as a novice, one simply lacks the necessary exposure and practice when it comes to more complex technical skills. Nevertheless, we all begin in a state of unconscious incompetence. After we are exposed to a particular concept or idea, we move into the uncomfortable space of conscious incompetence. But without further effect and energy it’s easy to remain here. Worse, you might not be aware of your true level of skill. With pressure from industry to be brilliant at everything, the wrong behaviors may be incentivized in favor of more productive behaviors like acknowledging when you don’t know something.

Still, moving from conscious incompetence to conscious competence is where the work begins. There’s no way to develop a skill without practice. Having knowledge about a subject is one thing but demonstrable skill is entirely a different story. Even having a demonstrable skill does not necessarily equate expertise. When you finally move into the last state, unconscious competence, you have fully internalized the knowledge and skills necessary.

Which may explain why a domain expert gives such cryptic explanations about how they’ve obtained their own skill level. If they have moved past the stage of thinking about the particular skill, they might have difficulty articulating how they arrived there. Or they might be too busy and rather give some glib answer to get back to the task at hand. But having expertise doesn’t mean you’re well equipped to explain your own skill acquisition or have the necessary ancillary skills to pass them along.

The famous quote attributed to Michelangelo characterizes his level of unconscious competence.

When they asked Michelangelo how he made his statue of David he is reported to have said, “It is easy. You just chip away the stone that doesn’t look like David.”

Perspective

There’s one Alan Kay quote that has struck a chord with me since I’ve heard it. Maybe it’s because it sounds smart or if you simply change how you look at a problem you can make headway.

Perspective is worth 80 IQ points.

I would argue that novice developers lack important perspective on developing software. Indeed, the industry has noticed and lamented how inadequately prepared for the rigors of software development young grads are. Is it simply lack of experience and perspective? Perhaps. Or perhaps there’s the hard knocks of life that only come with trial and error, and really failing in the adult world. In the age of rampant grade inflation and ‘everyone is a winner’ it’s easy to skirt by any real challenge to one’s self esteem. One could ask instead, is it because I’m a bad programmer or is it because I’m inexperienced in this area? Did I approach the program in a suboptimal way? Or perhaps this isn’t my strength. Do I even know what my own strengths are? Well, this is a problem you can solve.

Strengths Finder

When I stumbled upon the Strengths Finder it was so obviously good I spent little time debating whether I should spend the $20 and take the quiz. In a nutshell, the Strengths Finder asks you a series of questions that help you identity your own strengths. You’ve probably heard the phrase ‘play to your strengths’ but it’s unlikely you have identified your true strengths. I know I didn’t. If you’re curious, my strengths are:

  • Deliberative
  • Strategic
  • Intellection
  • Learner
  • Input

In retrospect, some of these strengths seem obvious. Others, like deliberative, are not so obvious. If you’re curious about your own strengths I encourage you to discover them by taking the quiz. It is easy to have others, perhaps innocently, characterize your own strengths for you. Or perhaps you’ve come up with them on your own. Instead, this approach is backed by research and decades of experience. Guard yourself against wasted time and effort by developing the wrong strengths. Your career success and growth is dependent upon playing to your real strengths and developing expertise in your real domain.

The Real Expert

So much has been written about experts it almost seems redundant at this point. You’re probably a well-optimized reader who’s consumed many articles, books, videos, etc but I can tell you that the definitions of expertise I’ve read about have not been satisfying. From my own experience, I’ve viewed both brilliant technologists and people who’ve clocked in decades of experience. Surely, as it goes, the developer who’s repeated the same year ten times is not the same as a developer who’s expanded beyond their comfort zone. Yet, there isn’t a good model that I’ve found. A model that is repeatable when it comes to developing expertise. Is it simply knowledge? Wisdom? Perspective? Expertise? Or some combination?

Moreover, it’s far easier to point to something and describe the symptoms rather than the underlying process that produces the so called expert. We can generalize the symptoms here: smart, capable, experienced, technical, and productive among many other qualities.

Peter Norvig has detailed one approach to developing expertise in just about any area. 10 years of deliberate practice. We’re all in such a rush to develop the essential skills that it seems we’re running in place just to keep up. Yet deliberate practice without understanding the desirable final state would not be of much use. While a gifted developer can intuitively devise a solution, without 10,000 hours of deliberate practice it’s unlikely you will develop deeper mastery.

Our field is a bit odd. Do you think carpenters sit around at some carpentry conference and mythologize the rare gifted young master carpenter? Or civil engineers applaud the heroic efforts of a young yet capable engineer who’s solved some sticky problem? Perhaps every industry has its myths but for our quasi-scientific endeavor we readily consume books, articles, and videos that ‘promise’ master level proficiency in record time. It’s as if, somehow, on the way towards becoming a better developer we’ve simultaneously sidestepped decades of pedagogy, neuroscience, and cognitive psychology seemingly overnight. A real accomplishment indeed.

Deliberate Practice is Hard

Yet unguided deliberate practice is hard. It requires stepping out of your comfort zone. It requires humility as you have to admit you don’t know how to do something. You have to get used to failing. It also requires some specific goals on how you can achieve the desired state. Perhaps you’re lucky enough to have a mentor to guide you. If not, then you’ll be in the wilderness and have to fend for yourself.

More importantly, not all practice is equally valuable. Some tasks may be hard but may be irrelevant to your goals. Perhaps you’re banging your head on a problem that can be easily solved by a shift in perspective that comes from a more experienced mentor. Or a longer coffee break.

This is the reality. You have to put in the work. The work on becoming a master in any craft requires lots of trial and error and practice. That’s the tough part. How can you sell yourself on the prospect of doing more work after you’ve come home after a long day at work? There are no easy answers as you have to make tradeoffs between your personal life and professional life.

Characteristics of an Expert

There has been some research done on the differences between novice programmers and expert programmers. Here are some of those observations:

  • Expert programmers not only have available more knowledge schemas, strategies, and rules applicable to solving programming problems, but they perceive and remember larger “chunks’’ of information than novices. [1]
  • Experts read programs for flow of control execution, rather than line-by-line (as text). [1]
  • Experts form detailed conceptual models incorporating abstract entities rather than concrete objects specific to the problem statement. [2]
  • Their models accommodate multiple levels and are rich enough to support mental simulations. [2]
  • They bring to the task their previous experience, in the form of knowledge of the commonly occurring structures (combinations of the primitives) in the domain. [3]

If I had to summarize, I would say that a true expert has developed the knowledge, strategies, and mental models necessary to excel. More importantly, more than skill, you have to develop your own ‘cognitive toolkit’. These are the perspectives, strategies, and lines of reasoning needed to make any headway towards mastery.

The Long Road Towards Mastery

I’ve resisted turning this into an easily consumable top ten list of things you can do to become an expert. If you’re anything like me I read those types of articles, feel better about myself, and then quickly forget to implement what I’ve learned. The journey, the real journey, is of course more complex than any short article can hope to illuminate. However, I can propose some questions you can ask yourself:

  • What do I want to get out of my career?
  • What will I give up to achieve what I want?
  • What resources will aid me?
  • What level of technical expertise do I want to achieve in my target area?

It is unreasonable to expect that you can become both a world class programmer and a world class violinist. Not impossible, but highly unlikely. You simply have to choose. There isn’t nearly enough time in your lifespan to obtain both skills and retain them. However, you can decide to become ‘decent’. Becoming decent at something may be sufficient enough to achieve your goal. Josh Kaufman has written at length about how to do just that. Considering how quickly technology changes, becoming ‘mad decent’ at a particular skill may be enough to keep your career afloat.

However, to develop true world class expertise requires nothing more than a favorable disposition, great determination, and roughly 10 years.

References

[1] — On the cognitive effects of learning computer programming — Roy D. Pea, D. Midian Kurland — 1984 — New Ideas in Psychology

[2] — Mental imagery and software visualization in high-performance software development teams. — Petre, Marian. — 2010 -Journal of Visual Languages & Computing.

[3] — The Programmer’s Apprentice Project: A Research Overview — Charles Rich, Richard C. Waters — 1987


Originally published at benjaminspencer.me