Managing software teams in academia and industry
Teamwork, ugh. Is there anything more frustrating to either participate in or oversee, especially when it comes to software development?
That’s the sentiment I usually get, regardless of the context. Students in classes gripe about their groups. Teachers have trouble managing groups. Developers have trouble learning to be great teammates. And engineering managers grasp at guidelines to help them manage the unmanageable complexity of people working together.
I’m not going to pretend to be an expert on software project management, in academia or industry. That said, I have spent a lot of time managing software teams. I managed a team of eight engineers at AnswerDash for two years. I’ve managed teams of researchers. And nearly every class I’ve taught has involved group work, including my undergraduate Design Methods class (which has design teams of 2–4), my undergraduate Software Engineering class (which has software engineering teams of 3–5), or our six-month capstone (which has teams of 3–4, a team of instructors, and a team of TAs). I also explicitly teach software project management in my software engineering course, so I know the literature on what does and doesn’t work.
So why is teamwork so hard?
What is teamwork?
There are many aspects of teamwork that I want to address, but first, I think it’s helpful to provide my informal theory of what shapes software engineering teamwork:
The fundamental challenge in software engineering teamwork is maintaining a shared understanding of what is being built and why, while simultaneously evolving what is being built and why.
This statement is full of complexity. For example, how can a team maintain shared understanding? Why is it important that teammates know why something is being built? And when these two things are changing, how can teammates ensure their understandings are up to date? Lurking inside these questions are critical challenges in communication, coordination, the social relationships necessary for ensuring these activities are efficient and fruitful.
With this organizing framework in mind, let’s discuss about some of the major challenges in organizing software teams.
Forming
There are so many challenges in forming teams:
- Teams have to have the right expertise for the work, but the necessary expertise changes as projects evolve, requiring learning.
- Because no one individual has all of the necessary expertise, individuals have to communicate with each other to make decisions, exchange knowledge, and ensure whatever is being created is coherent. This means they have to be willing and capable of effectively communicating with each other.
- Individuals have to be sufficiently motivated to do both of the above, and if they’re not, work won’t happen.
In higher educational settings, all three of the above are a mess. Students don’t really have much expertise yet. Most students, since they’ve spent a decade working alone in school, are still poor communicators. The only factor you can really control is ensuring that students are sufficiently motivated to do the work, and do the necessary learning to do the work. Even then, students have numerous other responsibilities (other classes, jobs, personal lives), so projects only get a small portion of their motivation. And finally, they’re motivation can be quickly extinguished, since the struggles with all of the above can be frustrating, disheartening, and ultimately demotivating. Fortunately, students are also predisposed to learning, so as long as you explicitly help them develop these skills, students can make steady progress.
With all of these challenges, how should one form teams? I like to think of it like building a puzzle.
- Know what expertise the team requires and make sure the group has it. That means monitoring what expertise is missing and having good ways of evaluating expertise. (I wish I had some recommendations, but we don’t really have good assessments for most knowledge in computing, in academia or industry).
- Understand what motivates the individuals in your team, or the individual you might hire. Is a student in the class because it’s required? Why is that undergraduate interested in contributing to your research? Is the developer you’re interviewing interested in the salary, the product, or both? Knowing why an individual is motivated is key to keeping them motivated. Just ask them.
- Can the team have not only a functional conversation, but even an enjoyable one? If you’re bringing a new person onto a team, are they going to make those already enjoyable conversations frustrating and slow? If you’re team can’t have a conversation, or specific pairs of individuals in your team can’t have a conversation, your team is screwed.
Balancing all of the above concerns is the hardest part of the puzzle, because rarely can you satisfy all of these constraints. You may have to decide between having the expertise you need or having a team that communicate, and then devise strategies for working around a new team member’s deficiencies. The art of being a great manager is being aware of all of these factors and continually erecting workarounds as a project changes and as people change.
Communicating
Communication as the fuel of teamwork. It’s the foundational skill that enables coordination, learning, knowledge sharing, and decision making, and if people on your team can’t do it, it’s going to fail, over and over.
That said, even when people have baseline communication skills, the way a manager arranges their communication can make things better or worse. Take tools, for example. What’s better: email, meetings, or Slack? Should teams communicate in-person or remote? In my experience (and in research), in-person communication always has the best chance establishing common ground and preventing misunderstandings. Research suggests this is because it is so high bandwidth: people communicate not just with words, but non-verbal behavior, and even emotionally through faces. Though some media do a reasonable job approximating these channels (e.g., video chat), none come close.
All that said, synchronous, collocated meetings aren’t always feasible for people’s lives. If you have to settle for something synchronous but remote, just be aware that there will be misunderstandings, and you might have to give extra effort to preventing them by giving the team’s communication more structure. The best trick I’ve learned is restatement: when someone says something, say “Let me restate what you said to make sure I understand it”. This trick makes explicit what we implicitly do face-to-face (otherwise known as “grounding” in communication research). Of course, you can do this in collocated meetings too, but it’s less necessary.
If you have to communicate asynchronously (as with email, Slack, or other messaging platforms), you need even more structure. I find Slack is best at providing this, because it ensures there’s appropriate, shared historical context for a conversation. Email threads hide the context in a mess of indented, quoted text.
Coordinating
If you think communication is hard, coordination will have you stumped. Coordination is about dependencies, and in software, about the communication dependencies that emerge from technical dependencies in code. These dependencies require shared decision making, they result in blocking (where one person has to wait for other decisions before they can proceed), and if you want your team to be productive (who doesn’t?), they require careful planning to maximize the pace of shared decisions and minimize blocking.
I find that there are two critical aspects of coordination. First, it’s making sure that each person in a team knows when they’re making a decision that affects someone else on the team. That means each person needs to have some model in their head about who is doing what and how those activities are connected. If they don’t, they’ll make a decision that breaks someone else’s work. Tools can help with these, because when we express dependencies in code (especially statically typed code), compilers and type checkers can tell us when we’ve made a decision that is incompatible with another decision. If you’re using a programming language for which this isn’t true (anything dynamic), your team members need to be even more careful to check who’s work is affected by their decisions. As a manager, you’ll want to monitor for these coordination breakdowns for signs of deeper issues in your teammates’ coordination.
The second thing to manage is broadcasting availability. Everyone should be able to reach everyone else, because you never know when a decision is going to require their input. Setting up Slack with people’s phone numbers, email addresses, and other contact information ensures that teammates can reach each other when they get blocked. Many teams will just rely on meetings, even scheduled ones for this. While meetings work, they’re rarely scheduled for the right time, which means your team will spend a lot of time being blocked.
Finally, people have lives. And if you’re working with students, they probably have other classes and may even a job or two. They can’t be available all of the time. That lack of availability is something to plan for, by setting up practices like requiring students to check Slack, or their email, regularly.
Managing
So far, all I’ve talked about is the work that the team does. What about you, the manager? The best managers I’ve had in my life have excelled at being three things:
- A resource
- A motivator
- A mentor
You’re a resource because you probably have expertise and resources your team needs to make progress. You’re a motivator because you’re the one setting deadlines, creating incentives, enforcing rules, and setting vision. You’re a mentor because you’re probably guiding the individuals in your team beyond the specific project you’re managing, to future projects and careers. In fact, the core source of motivation your teammates have is probably their future, so nurturing that is also key to keeping them motivated for the current project.
This is also key to retention: after all, if someone is gaining valuable mentorship from you, they’re learning, and they’re enjoying their interactions with a team, they’re not going to want to leave.
In some cases, in addition to these three roles, you’re also a teacher. After all, most teams don’t have all of the expertise they need, and so they’ll be learning, and you’ll be overseeing that learning. If you’re the one with the expertise, you’ll probably be teaching them, which means being a great teacher, which I’ve written about elsewhere.
Common Questions
Up to this point, this discussion has been pretty abstract. Let’s get concrete, and address some common questions I’ve received:
How can I identify good recruits? You have to know what expertise and personality you’re looking for and devise ways of assessing whether a recruit has it. Research hasn’t been very helpful in providing these assessment tools, so you’ll have to devise your own. When it comes to interpersonal skills, trust your instincts. When it comes to skills, focus on activities that directly demonstrate competency, not proxies like coding puzzles.
Is it a good idea to have a “trial” period with a recruit? It’s certainly reasonable to hire someone for a period of time, and let them go if they aren’t working. That said, if you’re going to invest in someone, take it seriously: help them to become the person you want them to be. Don’t be the reason they fail.
How do I onboard someone to an existing project? Expect a lot of learning. Depending on how long the project has been going, it may take months of full time effort before they understand enough to make meaningful contributions. Give them an authentic task in the project to find out what they need to learn, then help them learn it.
How can I develop shared ownership for an existing project? The only way to do this is to have defined the project as something shared from the start. Otherwise, if it has an owner, you’ll inherently be taking something from someone if you attempt to share it later.
What tools are good for small software projects? These days, a popular project management stack is Git, GitHub, Slack, Trello. They’re all free, and they do most of what you need. That said, they aren’t magic: you have to define process and coordination around these tools to have any success.
What’s the value of in-person meetings? They’re great at developing trust, creating shared understanding, and preventing misunderstandings, but with the right practices, you can do this remotely too. Research shows you have to start with in-person meetings though (Šmite et al. 2009), which is hard if your team is always changing.
What I do
I manage software teams in three contexts.
In my classes, I carefully design teamwork scaffolding. That means giving teams some of the key ingredients I’ve described above rather than having them devise them on their own. This is critical because they’re still learning many of the basics of communication and coordination; requiring them to learn these and more advanced skills in process design would only lead extraneous cognitive load, preventing any learning from happening. You can see the types of scaffolding I provide in my software engineering course.
In my lab, when I have a team to manage, I collocate the team, I define clear ownership over what we are building (usually giving ownership to a doctoral student), and frame myself as the person with final sign-off on design and engineering decisions. I also carve out long blocks of time in my schedule (especially in the summer) to ensure that I’m available as a resource, mentor, and motivator whenever it’s necessary. Finally, during the academic year, I do simple scrum meetings on a weekly basis, but then reach out to students halfway through the week to see if they need help getting unblocked. During the summer, we scrum daily.
In industry, my approach was more meta. There’s enough complexity in a full-time engineering team that you need to constantly reflect and refine process as things evolve. In a startup, you might iterate on process weekly; in a larger company or more established product, things will necessarily be more slow to change (and to ship). You can learn a bit more about the process challenges I faced in my recent ICSE 2017 paper.
Reflect on your practice
There’s no way this post is enough to make you a great manager. You could probably read ten more books on the subject and still have few of the skills necessary to thrive.
I find the best way to learn is to reflect on your practice using all of the fundamentals above. Why is that student not motivated? Why is the team so slow? Why aren’t those two developers working together well? Understanding the answers to these questions for the specific people in your team is the only way to successfully address these problems.
If you want to learn more about software engineering, I’ve written a short book summarizing the research literature. It’s probably not the best resource for improving as a manager, but it has even deeper foundations than this post does, and might give you a better foundation for analyzing your own process.
If all else fails, I love the “5 whys” approach to problem diagnosis. Wait for your project to fail, then iteratively ask “why” questions about the failure until you find the root cause. For example, lately I’ve noticed my doctoral students don’t usually work in the lab, meaning I can’t have as many ad hoc meetings with them to unblock their work. This is despite the fact that I block off time to be available to unblock them. I haven’t figured out why yet, but let’s imagine I have:
- Why aren’t they in the lab? I haven’t made it clear when I’m around.
- Why haven’t I made it clear? It’s clear on my calendar, but I forget it’s not clear on their calendar.
- Why isn’t it on their calendar? I never invited them and I only briefly mentioned once at the beginning of the quarter when I’d be available.
- Why didn’t I send an invite? That’s a good idea, I really should!
That was only four questions, and I found a potential solution to my problem! Through a little reflection like this, you really can solve any team management problem.
Loving teamwork
If you get good at everything above, teamwork and software project management can be a real joy. I’ve practiced enough in my career that I think I succeed more than I fail, but I still fail a lot. Every single one of those is a chance to learn.
