The Fall of The Software Engineer, The Rise of The Programmer Technician
Most of the people I know are software engineers or computer scientists of one form or another. Most of them are very experienced and come from a time when to be either of those things required a very serious computer science education. Of course this is still the case for being a computer scientist, but seemingly not so for programming. Many of my well seasoned colleges lament the decline in skill and education of their younger peers. I also have seen this shift in the skill set and education of entry level programming candidates both on-the-job and in the classroom. It seems as if software engineering as a discipline and profession has given way to “hacking” and “cowboy” developers who know very little actual computer science. Has something changed? If so, what?
If you go back to the beginnings of computer science, it was not called computer science and there was no computer science department in any school. Computer science as a discipline was a subset of mathematics and/or engineering. The first computer science department was established in 1962 at Purdue University, and it was another 15 years before there was even a just a small handful of University computer science departments. Well into the 1980's, computer science was still mostly considered a sub-field of mathematics or general engineering. However, we had computer programmers as early as the 1940’s. Who were these early pioneers and what did they do?
The first programmers were mathematicians. They weren’t usually called programmers. They were analysts, engineers, mathematicians, or just generally “those really smart women over there in the refrigerator” (most of the first programmers were women and most computers were in “cold rooms”). These “programmers” had to have extensive formal education and knowledge in mathematics and/or engineering. Most were more closely related to scientists than what we think of as programmers today. Over time, and by the late 1960's, the field grew and the people involved in programming computers became known as software engineers. These were very highly educated people (still mostly women), usually with advanced degrees in math. This continued for some time.
By the 1980's, computer science departments and degrees were beginning to take hold in colleges and universities. Some consider this the golden age of computer science. Personal computers were showing up in homes across America, movies like WarGames intrigued people and made them want to learn more about computers, and rumors about something called “The Internet” were scaring the hell out of people. During this period we saw the rise of the formal and traditional software engineer. It was a coveted and respected title, and only those formally educated in mathematics, computer science, and/or electrical engineering were allowed to carry it. This also lasted for some time.
By the end of the 1990's we began to see the first of whom we now simply call “programmers,” “coders,” and “developers.” These were people who instead of formal University degrees had credentials like military technical training or an associate’s degree in hands-on programming, or were simply self-taught thanks to the Internet. Slowly this new group entered the field of software engineering with new titles such as “programmer” or “developer” as opposed to “engineer.” We also saw an explosion in computer related technical fields such as network specialists and system administrators.
Even up to the early 2000's, as a hiring manager, I almost never saw any application for a software engineering job where the applicant did not have a related four-year degree. If I did, I would have thrown it out without thinking about it. Fast forward to the late 2010's and at least half the applications I saw were from people with no formal degree. Now as a professor of computer science I regularly have students who drop out after a class or two because they got a job as a programmer of some sort (usually not a good job, and not a job for which they are ready, but a job with the title “developer” nonetheless).
The title “software engineer” has either fallen out of favor or is reserved for traditional and senior roles in large tech corporations. Now it’s usually “developer” or “programmer” that you see advertised as a job title, or the unfortunately absurd and comical “hacker,” “code-ninja,” or “code-guru.” We also see the ridiculous “full stack developer” title thrown around, as if that has some special meaning (it doesn’t). A degree is no longer necessarily an indicator of fitness for a programmer job, and in fact no formal education is necessarily required. Standards seem to be in sharp decline, to the point where anyone who can type is now a “programmer.”
For many old schoolers, the typical conclusion and argument is that standards have simply declined. It’s obvious that the demand for anyone who can program (even rudimentarily) has skyrocketed. The typical argument is that someone has to do it, so we lowered the bar just to put bodies behind keyboards. On the surface it makes sense. Very few people are fit for four year degrees in the hard sciences and fewer still for advanced degrees in disciplines like math, engineering, and computer science.
Obviously it’s become about simple numbers. The demand for developers is absurdly high, the number of people who are capable of getting a four year degree in computer science is relatively low, and we need programmers in the field right now. We have to fill the demand somehow, so we just lower the standards. Apparently, now we just let anyone “engineer” and we blindly hope for the best. Seemingly, true software engineering is disappearing while “coding” and “hacking” with little formal education is taking its place. That’s a simple answer which many formally trained engineers would like to believe (including me). However, like most simple answers, we should look deeper before making a conclusion.
What really happened is computer technology matured, modularized, and productized. We used to require highly trained, formally educated software engineers in all aspects of computing because the number of computers in a business or institution was… one. One very big, very complicated, and very expensive computer. There were no networks as we think of them today and no Internet. The individuals who worked on these machines had to be all-in-one scientists, engineers, and mathematicians. The rigors of these early jobs were incredibly intense because the technology itself was immature and largely experimental. As computer technology matured it became modular, productized, and packaged (like any technology). No longer would one person have to be all-in-one mathematician, engineer, programmer, network administrator, and database administrator. Technologies became plug-and-play and so did the jobs for them. Technical roles (like the technology itself) became broken up among many focus-trained specialists rather than a few highly educated generalists.
This happens in all fields. The first days are reserved for the visionaries and the most talented and educated. The early days are then populated by only the most serious minds with formal education because each person has to have full knowledge of the entire field. Consider the early days of the automotive industry (before it was an industry); only those with a serious understanding of metallurgy and engineering could build, design, or fix an automobile. Building or fixing an automobile in the early 1900's could literally require knowledge and skill in blacksmithing, metallurgy, fabrication, engineering, physics, and chemistry (formal education or not, you still had to know a lot). In the 1930–40's, demand for automobiles skyrocketed, the assembly line was invented, and modular parts were created. Today, the average worker on an assembly line producing automobiles has little to no knowledge of the engineering and metallurgy that goes into an automobile, let alone the computer science also now needed to control the vehicle. Similarly we have automotive technicians that have little to no knowledge of engineering or metallurgy (or computer science). They use diagnostic equipment to tell them the problem, they order parts online, and they swap out the parts. These technicians and assembly line workers are trained and skilled, but in a completely different fashion from a mechanical engineer or a metallurgist. This has now happened in the computer science and software engineering fields — because it had to happen.
Just as in the automotive example, the demand for skilled workers in programming skyrocketed. At the same time, computers and software became modular, packaged, and productized. We have also seen explosive growth in the areas related to computer science such as networking, system administration, encryption, databases, electrical engineering, etc. We simply cannot afford to have everyone in the computer field be a highly trained computer scientist and mathematician, just as we could not build or fix cars if every assembly line worker and mechanic was required to have a master’s degree in mechanical engineering. It just doesn’t work that way.
More and more, what we have today behind computer keyboards are analogous to the skilled and trained automotive technician (as opposed to a formally trained mechanical engineer). The “coder” or “developer” is more and more someone without formal or extensive computer science education. Instead, this is a person that has been trained specifically in programming as a technical skill, and also has some foundational computer science education, but primarily as it relates directly to programming. Programming today no longer takes place in a free-for-all complex and experimental environment that requires extensive and formal computer science education. A lot of programming today requires something analogous to the automotive technician, the ability to identify the part that is needed, know where to find the part, and plug it in properly. A day-to-day programmer no longer needs to know how to design and construct a sorting or searching algorithm and analyze its performance mathematically before implementing it. A programmer simply finds the sort or search “part” and plugs it in.
What we are seeing is not the demise of the true software engineer or computer scientist, what we are seeing is the rise of the programmer technician.
These new skilled workers are those with basic computer science knowledge, but only as it relates directly to programming along with hands-on training in programming.
True Computer Science and Software Engineering: Alive and Well
Computer scientists and true software engineers have not been supplanted by “coders,” “developers,” and “hackers.” We’re simply re-organizing our professions and positions into what is a maturing industry. Thanks to computer scientists, engineers, and mathematicians in the first place, the programming technician (i.e. the modern “developer” or “coder”) is now possible. Building software is now modular and mostly pre-packaged and prescribed thanks to advances in computer science that have led to extensive software libraries, package managers, dependency managers, integrated development environments, software-as-a-service, infrastructure-as-a-service, distributed code repositories, and of course the Internet; all of which started with PhD’s and engineers in computer science research labs. Furthermore, programming languages like Python have abstracted away details to such a degree, almost anyone can do it with just basic computer science knowledge.
Computer scientists are still hard at work building new theories and new technologies. Formally educated software engineers with degrees in computer science are still building those theories into reality. However, now as those new technologies filter down to those on the front lines; it’s the programmer technicians who do the bulk of the day-to-day heavy lifting in software development.
Computer scientists, mathematicians, and software engineers who are formally educated with advanced degrees in computer science are still here, and the demand for them is higher than ever, they’re just a little harder to spot as a proportion of the total number of people in the field. Formal engineers and scientists use to make up 100% of the people in the industry. Now they make up a smaller and smaller percentage. Not because they are being replaced, but because thanks to those formally trained computer scientists and software engineers there is a new high-demand job created; the programmer technician.
Not only has the true software engineer and computer scientist not fallen, we’re still here and doing better than ever. We just have a new colleague we should welcome with open arms, because let’s face it, they are in the trenches doing most of the work now.
Now It’s Your Choice
Thanks to explosive growth in technology, and the work of those early pioneers in software engineering, an entire generation of people can now have lucrative and engaging careers in what used to be an almost inaccessible field. Just as the automotive and manufacturing revolution of the 1930–40's led to the creation of the middle class and an unprecedented rise in living standards, the IT revolution of the 2000's is doing the same thing. We’re seeing the democratization of computer science and this is a good thing.
The choice is now yours. Almost everyone can be part of the continued technical revolution and secure a well paying and engaging career in technology. The choice now is not if you can be in the tech industry, now it’s about where and what level in the industry do you want to be and how do you get there? PhD’s in computer science are still researching and coming up with new theories every day. Those with advanced degrees in math and computer science are still testing and implementing those theories into real and practical technologies. Formally trained and educated software engineers with computer science degrees are the software architects and leaders in the field. Programming technicians (i.e. the modern coders, developers, and programmers) are on the front lines putting the parts in place and assembling and maintaining the systems, while analysts and administrators support them.
Unlike the early days where only the most highly trained and educated could even think of entering the computing field, now you can choose your education level, and choose or change your place at will by advancing your education. However, just to be clear, if you want to code for a living, you still can’t get out of attaining some foundational knowledge in computer science, and you still have to have an affinity for the skills and lifestyle. It’s still not easy. It’s just that now it’s your choice where you start and how far you will go.