Popmenu’s culture of cross-functional collaboration
Turns out, teamwork actually does make the dream work
We compete with each other, not against each other. We fiercely support each other from day one and genuinely celebrate each other’s successes. As a true team, we have zero acceptance for brilliant jerks or laid back free-riders. We don’t pass the buck, but we fix it together!
Popmenu Careers Page
Software engineers know that technical expertise is key to providing value to our customers. While technical skills are undoubtedly important, the ability to work effectively with other teams and stakeholders is just as crucial. We recently faced a challenge at Popmenu that drove this point home. One of our core products is called Boost. This is a consultative marketing service where Popmenu Marketing Consultants create customized marketing materials, including social media content, for restaurants. We experienced repeated disruptions with the integration to Facebook/Instagram that caused the Client Marketing team difficulty in posting for customers. Since the social media tools were built in the early days of Popmenu, the alert messages and error flows needed some work. Only by collaboration between the Client Marketing, Support, and Engineering teams were we able to begin tackling this challenge. Ultimately, the progress we have made towards solving this predicament has not been only through technical solutions, but also the power of teamwork.
Key Teams: Partnering with Engineering
Support and Support Engineering (SSE)
Through our partnership with the SSE Team, we have bench strength to provide high technical proficiency, so the Support team can focus on client communication and case management. Having a team of experts to translate and prioritize the client experience into the clear problem statements for our Engineering team creates scalability and efficiency. We’ve leveled up the client’s experience while identifying trends and bugs more quickly, a huge win as we continue to build complexity in our product.
— Stephanie Jourdan (Manager of Support)
The Support team forms the cornerstone of any effective incident response because they are on the front lines of customer communication. This situation was no exception. They played a crucial role in speaking to customers and helping troubleshoot the issues. It’s essential to have a Support team with sharp critical thinking skills, so they know what can be addressed directly with the customer and what needs to be escalated. Once they decide to escalate a ticket, they send it to the Support Engineering team (a team that sits between Tier 2 Support and the Software Engineering squads). When they escalate tickets, they provide screenshots and detailed explanations of the problem, which helps the Support Engineering team get a head start on tackling the issue.
I got a chance to see this firsthand because, at Popmenu, software engineers have a rotation of spending 2 weeks at a time on the Support Engineering team. This is a great practice, because it allows software engineers to experience what it is really like on the ground. I observed that there is an efficient flow of information from the Support team, through the Support Engineering team, and into the Software Engineering squads. Coordinating this much communication back and forth cannot be easy, but the team functions like a well-oiled machine. They can also troubleshoot many tickets on their own, including fixing certain bugs in code. This saves the Software Engineering squads a lot of time and gets solutions to customers’ problems much faster. All in all, the Support and Support Engineering teams play an enormous role in all aspects of Popmenu’s success, and this situation has been no exception.
Client Marketing (CX)
A trusted bridge between CX and Engineering is crucial to delivering an optimal product and service for customers at Popmenu. When our CX team identifies trends in disruption of service related to possible product bugs, we outline these with data and present actionable information to our Engineering team — and our Engineering team always takes swift action to find a solution.
— Savannah Greene (Manager of Client Marketing)
The Client Marketing team has been indispensable in the resolution of this challenge. After all, the entire purpose of social media posts for a restaurant is to market its food. And, as mentioned, we offer a marketing service called Boost, where the team creates custom social posts for restaurants. So any interruption in social media integrations adds inconvenience to their already extremely busy schedules.
Individual Client Marketing team members have communicated frequently with Support and the Support Engineering team as issues come up. I have noticed that they are highly effective at making screen recordings with detailed explanations of any problems they encounter. This undoubtedly takes extra time, but it makes an enormous difference to the ability of the Support and Engineering teams to troubleshoot and fix any problems. The leadership of the Client Marketing team was also essential in communicating with the Engineering team to prioritize fixes. They knew what changes would make the most impact and made sure to share them with the Engineering team’s project manager. Ultimately, the excellent communication skills of the Client Marketing team have made addressing this issue much easier and more efficient for the Engineering team.
Having the individual engineers rotating through the SSE team has built both camaraderie and understanding of what and how the SSE team works. The Engineering teams have embraced the rotation process, aided the SSE team’s maturity, and made it easier to assist the Support team and our customers better. We were lucky that while encountering issues with the Facebook integration, we had an Engineer from one of the teams that worked directly on that integration, who took the problem very seriously and escalated it to the team.
— Olga Norton (Quality Assurance Engineering Manager)
Engineering Contribution
Contacting Meta Support
The problems with social posting were not coming from our site. Instead, we ran into various issues with Facebook’s Graph API. We have noticed that we get many times more 500 “unknown” errors from Facebook than any other social media integration we support. That is frustrating, but at least that can be mitigated by retrying later. Another huge problem is that Facebook is prone to banning or restricting people’s accounts without any transparency as to why. Facebook offers an Account Quality checker, but often people are restricted from posting even if their account is supposedly in good standing.
We started making repeated attempts to contact Facebook Support several months ago, and surprisingly we got a reply recently from our Meta account manager. Since they are a massive company, it’s understandable that it is difficult to get through to them, but it seems like persistence pays off in that regard.
Making Errors More Actionable
A piece of feedback that came directly from the Client Marketing team is that the errors they were seeing on our frontend were not easy for them to take action on. We tackled this in several ways. One of the simplest ways was to make error messages more explicit and detailed. As an example, Instagram accounts must be a business account to post using our platform, but the error message simply said, “Oauth failed — try reauthenticating your account.” This message was too vague and even misleading as to the real cause of the problem. So we went through dozens of possible error messages and made sure they were as actionable as possible to the user, and the Client Marketing team has said this has helped greatly.
Another thing is that there was no notification when posts failed. This is a problem because often, the Client Marketing team schedules posts months in advance. It is unrealistic for them to go through and make sure scheduled posts actually went out for every client. Based on this feedback, we implemented an email alert system for post failures. This was a straightforward thing to do from a technical perspective, but we only thought of it with the input from Client Marketing.
Monitoring and Logging
The final thing Engineering has done to tackle this problem is to add more monitoring to help us get ahead of any troubles before other departments or clients even notice a problem. This is a work in progress, but we are using Datadog monitors to alert us via Slack when the social integration error rate is high. We also added a Datadog dashboard for use within our Engineering team to keep track of errors and see how they are trending over time. At Popmenu, the Engineering team also uses Sentry, which has helped us gain insight into exceptions related to social integrations.
In addition, we added a Looker dashboard to give non-technical stakeholders insight into any social integration errors. Compared to Datadog, Looker offers a much more intuitive and visually appealing dashboard. Our team found Looker to be more time-consuming to set up than Datadog dashboards, but it was worth it in the end to have a place that anyone who is interested can quickly go to and explore the data regarding social integrations on our platform.
As this story shows, continuous communication with other departments is key to keeping the Engineering team on track to deliver the solutions customers need. As software engineers, we can sometimes fall into the trap of thinking that complex technical solutions are the only way to improve our software. And yet, in this case, the technical solutions the Engineering team implemented were not anything groundbreaking. Thus, it is important for software engineers to realize that communication skills are just as essential for success as technical skills.
Thank you to Savannah Greene (Manager of Client Marketing), Stephanie Jourdan (Manager of Support), Olga Norton (Quality Assurance Engineering Manager), and Michael Holloway (Engineering Manager) for their insight and collaboration on this piece.
