Engineer Q&A: Patrick Gilbert, DevOps Engineer

Tell us about yourself! What’s your name, what do you do at Optimizely, and how long have you been here?
My name is Patrick Gilbert, I’ve been here for a year and a half. I’m a DevOps engineer on the E-squared group. “E-squared” stands for “Engineers for Engineers”.
What “DevOps engineer” means tends to vary depending on the company you work for, who you’re talking to, or whoever is defining it at the moment. To me, it’s a combination of software engineer and operations specialist that focuses on not focusing on any particular thing.
A software engineer often has a primary language they work in, an application they’re gonna build; that’s their day to day. I, on the other hand, maintain a dozen or so tools and systems. Additionally, I lend support to product teams at Optimizely in all stages of software engineering projects, from design and building to releasing and operating.. DevOps, to me, is more about being a Jack of All Trades than being specifically good at any one thing.
In the year and a half that I’ve been here, I’ve written more application code than I ever have in my entire career, and I’m happy about that! I actually do enjoy software development. It’s nice to get that big payoff, the satisfaction of a completed project, in the end.
Tell us a bit about your trajectory to get here.
The short version is — I got into tech because it paid better than what I was doing before. I was a salesperson at a landscaping store. It was a good job, but it didn’t pay that well, so I got into IT.
I’d been doing IT on the side for years, and finally figured out how to break into it as a mainstream job, and then I was provided with the opportunity to join a TechOps team (the operations side of system administration) — a lot of software, a lot of process management, but not a lot of hardware. That introduced me to Bash scripting, Perl, Python, Jython, and Java. I was a Java software engineer for a few years, then I went back into DevOps because I found that in software engineering, moments of gratification would often be separated by prolonged periods of development. I found that to be frustrating, and it was difficult to keep myself motivated. DevOps work, in contrast, can often have a faster turnaround and produces grateful coworkers, which is rewarding.
So what makes someone want to be a DevOps Engineer?
I think you have to love technology in all its forms. If you have a specific love for a software language or a way of doing something, you probably want to be a software engineer. If you have a generic love that goes across all of that stuff, and you want to solve problems that bridge those technologies, then you may want to be a DevOps engineer.
I’m a junkie for feeling like I’ve accomplished something in a short time. I had a coworker in ops in the past who said to me: “I live for the outage”. He didn’t want outages to happen, but resolving other kinds of situations just never made him feel as proud or accomplished as resuscitating a failing production service. If a software engineer is a surgeon with a specialization, then a DevOps engineer is like an EMT.
If you want that challenge of leaping to the rescue and, to some extent, being a hero (I dislike that designation; it suggests that you’re the sole person who can fix the problem at hand — never a good state to be in), then DevOps is a good career to get into.
What’s it like to be a DevOps Engineer at Optimizely?
I get to help people a lot, which I really enjoy. Yesterday I started out digging around in AWS infrastructure, then got pulled into Google Cloud Platform. Following that, Jenkins build problems, then Python and Serverless Framework code reviews. I’m given opportunities to be a part of software design reviews and implementation decision-making. It’s engaging. It’s not the maintenance and cleanup crew I’ve experienced elsewhere.
Optimizely is unique for my role because I’m not directly responsible for all of production. The developers are responsible for the services that they maintain and deploy into production.
In previous DevOps roles, it didn’t matter how many engineering teams we had; all of production belonged to my team. When engineering teams would create new software and release it, the last moment they ever looked at it was when it left the development and testing environments. After that, it was my problem.
And there were problems. At one previous company, when bugs were discovered the product team was not expected to investigate the issue. That responsibility, instead, fell to my team. Upon understanding the issue, we would then judge the severity of the issue and file a bug report. If it was a severe bug, I was given the power to sit next to the engineer and say “You will work on this bug now,” and torpedo what they were working on! I found this arrangement dysfunctional because it was unfair to everyone involved. The software engineers could not be expected to meet promised deadlines when we interrupted their work. My team, held responsible for the health of production, did not have the resources on the team to fulfill that requirement.
At Optimizely, I’m more directly responsible, and primarily responsible for enablement of engineers; helping to do the job of managing and maintaining their production services instead of doing it for them.
How is the team structured?
E-squared is part of the Developer Productivity squad. The squad is more generally responsible for operations, QA, build and release, and incorporates embedded DevOps members of other teams like Greg and AJ, who work on the Data Platform squad, even though their expertise is in DevOps.
An example of how DevOps work affects the entire engineering org: Greg found an issue with how our AWS accounts were organized. He documented and socialized the issue, and started us on a path toward coherently-arranged accounts with refined access controls.
Good as that sounds, it also highlights one of the weaknesses in the E-squared team: because we’re not embedded in the teams, it took us longer than it otherwise might have to recognize the awkwardness of the previous setup. We’re not as close to the day-to-day pain our software engineers experience. To mitigate this effective “blindness”, we conduct surveys, join planning sessions, join bug bashes, and come to standups. We’re the best team for bug bashes!
How does your work fit into the team?
I’m a Senior DevOps engineer, so part of my job is to lead larger initiatives on the team. But I also spend a large amount of my time mentoring members of my team and members of the engineering org at large. You can call me over and I will help you! I’m a resource for you to use. We are engineers for engineers… or IT for developers. We’re your support desk! Please take advantage of that support.
What are you working on this quarter?
This quarter I’m working on secrets and config management. One of the challenges of any engineering project is not just that you’re going to be creating a new process or application or what-have-you, but that you have to do it in a way that’s both backwards compatible and not production impacting. Configuration management is absolutely no different.
Bespoke tooling is fun to write and horrible to maintain; it should never, ever, ever be part of the primary solution to any problem, unless you’re an operations team of more than double digits and judge the benefit of having purpose-built tools to exceed the tremendous costs to build and maintain them. The stuff coming out of Netflix and Spotify — they have large operations teams that have the time, skill, knowledge, and money to spend on creating those projects, like Spinnaker, Chaos Monkey, etc. These are really interesting projects that do interesting things, but each requires a dedicated team of maintainers, just as any nontrivial project does.
What are you most excited about in your work here?
I am most excited by the fact that we are building more stable, easier-to-use, simpler-to-understand systems across the board that our engineers do not have to study to be able to use. The goal is to reduce the knowledge transfer required to start using the systems, tools, and processes, because they’re intuitive and obvious. We’re slowly getting there, but it’s a process! The cohesiveness of our group has been a large part of that, because we’re all on the same page more or less and we’re all pulling in the same direction.
I really love my current team. They are unanimously passionate about what they do, and they want to do the best work they possibly can. They each have their own particular skill set, and we come together to form VOLTRON. We all have strengths and weaknesses, and we overlap in a way that makes us all stronger together. We also help each other recognize where we each have room to improve. I have 1:1s with every member of my team to facilitate this.
It’s a quirky group. The style of person that gravitates to DevOps is a little rebellious, definitely individualistic, and still willing to be a team member without losing that. A sample of folks whom I work with: you’ve got Panzer with his passionate outbursts. Laura brings a calm to the group. Eady and Joy are our two skilled managers and resident goth ladies. They’re all very unique in their own ways and they’re comfortable in their own skin, and they’re comfortable being wrong and learning from it.
I feel like a lot of engineers are uncomfortable being incorrect about something. It’s OK to be wrong because that’s how you learn. DevOps is definitely a process of learning all the time.
What’s your favorite thing about Optimizely? What’s unique about Optimizely?
Favorite thing: equality. I think of the equality across the org — I never get a feeling here that one group is better than the other. If I really wanted to sit down with someone from the business teams, I totally could. I’ve had lunch with David, our CFO (he lent me his manga!) and with our CEO, just because they came and sat at the same table.
Being DevOps at other companies I felt like the red-headed stepchild of engineering. I was “not good enough” to be an engineer. It was kind of this weird space of not being a “proper” software engineer. Here, the democratic mindset is very different.
