If you’ve looked for a software job — or tried to hire software engineers — in the last 5 years you’ve probably used the term full stack to describe yourself or your organization. It basically means a ‘generalist’, but that is just more jargon; what does it mean to be a generalist? What does an engineer with the designation actually need to know? Everything in the realm of software? What kind of software? Are we talking web services, mobile, micro-controllers? No one knows all of that.
Full stack is a reference to a group of components that come together to create a fully-realized piece of software. A database by itself is not terribly useful without an application sending it data, and so software components are layered — stacked — together to create functional applications. Web application stacks are normally thought of as the combination of UI, application code, data persistence, and webserver. A popular modern stack is MERN (MongoDB, Express, ReactJs, and Node.js), and the one I started my career in was LAMP (Linux, Apache, MySQL, and PHP) — a classic. Notice the more modern stack no longer concerns itself with the operating system.
When I worked on a previous employer’s Windows, IIS, MSSQL, .NetFramework stack (the WIMN stack?), we prided ourselves on all being Full Stack Developers. Teams could be shuffled to meet deadlines because everyone there could do any part of the stack. We all knew at least a little front-end, a little MVC, and a little MSSQL. Sure, some people knew a lot of front-end, or used other server-side languages, but the theory held. Once you are familiar with the concepts, writing endpoints in Rails is not altogether different than in .Net.
I remember interviewing a candidate there; when asked what she thought it meant to be full stack, she faltered, and replied it was “my ability to work in any part of the stack”. While technically correct, it wasn’t the answer I was looking for. I was wanting specifics about how she had experience adding buttons to pages, building CRUD endpoints, and debugging slow database queries because at the time that was everything a developer needed to know to be highly effective. The rest of the important steps to running production software— deployments, build pipelines, monitoring, security, end-to-end testing, server acquisition— those were other roles, not Software Developers. I think on that memory now because if I were to be asked that question today, and gave the answer I had expected then, it wouldn’t be what I’d want to hear now.
I spent most of December of 2019 building the full stack of a small .NETCore web service to give Kabbage users some new functionality. It took me maybe three days to write the front-end, back-end, unit tests, database migrations, and a Dockerfile. At this point I would consider it ‘dev done’. It had everything it needed to fulfill the business’ software need and you could one-click-run-it locally. Then I spent another 2 weeks doing everything else to get it deployed. The stack isn’t just the database connection and web tier anymore. It’s the (virtual) infrastructure, security-policy enforcement, and deployment pipelines.
Terraform declares and orchestrates our infrastructure; a fleet of docker containers, EKS to host and manage those containers, Ambassador to control ingress and load-balance the nodes, and Route53 to set up canonical names to ambassador. Helm to deploy to EKS, migrate the database, and run our integration tests. Bamboo to build the image and kickoff deployments, Artifactory to hold the built image. RDS to manage the database servers and security groups to control their ingress. Vault to manage our credentials, kubernetes config maps for non-secrets. Splunk to log, Jaeger to trace, Datadog to monitor, intricacies with Jira and the way we track changes that must be configured for new projects!
The curious reader might have followed a few of the links in the paragraph above, all of them go to the enormous (excellent) documentation for these services. I don’t think anyone expects me to be an expert in all of them, I don’t think someone could be. I’m not saying a dev is expected to be a lonely island in a sea of tooling. When there are questions, or an error in infrastructure manifests, we have many people who have more experience in certain tools who can offer advice and preferred practices. The management of these tools at a company level — the EKS clusters themselves, the Artifactory account, the bamboo build servers , etc— still need dedicated teams, just less and less as the management of them is abstracted by tooling.
At some point Ops became DevOps became Devs. I don’t think this is a good or bad thing. 5 years ago, getting a new web app stood up could take half a year or more as you tried to wrangle the priorities of multiple teams and requisition physical servers. With infrastructure, testing, deployment, everything as code, it takes a month or less if you’ve got templates to work from. It also has given me a great deal of agency over my work; ownership of the whole process. The ability to decide how many replicas, how much compute, which style of storage makes sense for my data. But it’s also an enormous amount of tools and best practices to be familiar with, and I don’t spend nearly enough time with any one piece to consider myself proficient.
Case in point, I recently took out a PostgreSQL database by accidentally filling it with expired users. I was configuring Vault with an incorrect revoke statement to rotate my database credentials. Turns out, you have to drop permissions before you drop users, because the permissions are dependent objects. I had never deleted a user from a database before, let alone configured user life-cycles, and it didn’t occur to me that permissions were like foreign keys.
As a .NET developer, configuring database security had always been Someone Else’s Job, but now it’s within the fluid boundaries of full-stack development. Besides the immediate problem of making incorrect assumptions about the specifics of user-deletion in PostgreSQL, this raises problems with hiring and career growth. How do you hire for a junior Full-Stack Developer? If they need to be familiar with the whole process of getting an idea to code to production, isn’t that actually a senior developer? If all your devs are full-stack, if each can work on any part of your software, how do you know when they’ve demonstrated senior-level prowess?
Kabbage is hiring, and we try to attract a wide variety of people by describing as much as possible about what you could be working on. We don’t expect anyone to be an expert in all or even one of those things. Just familiar enough with core concepts to figure the rest out, and to be kind while we all learn as we go. Nowadays when I interview for web developers, I want to see that they know how to learn. Are they able to articulate their reasoning? Are they actively interested in web technologies? If these are demonstrably true, then knowing some or all of our specific stack isn’t a requirement.
To me, today, a Full-Stack Developer is anyone with a storied history of googling error messages.