GraphQL at PayPal: An Adoption Story

Photo by Parrish Freeman on Unsplash

State of GraphQL today

We started our GraphQL adoption journey by building our checkout experience. We saw tremendous benefits in adopting GraphQL when our checkout app built with GraphQL became our guiding light.We built more apps, provided infrastructure support, launched a public GraphQL API, and provided trainings and learning materials across the company. We also set up a standards body, provided a GraphQL tools fanny pack, and built sample apps to help teams get started.

Today, GraphQL is being used by several production apps across PayPal. It is now a default pattern to use GraphQL for building new UI apps. Many existing apps are in the process of migrating to GraphQL. GraphQL is being used by common platforms such as Identity, Payments, and Compliance to provide a consistent experience across all PayPal products. Our API developers have started using GraphQL to build APIs. Braintree released its public GraphQL API.

With the help of GraphQL, we have been able to bridge the gap between a backend for frontend (BFF) apps and backend API capabilities since GraphQL can work as an orchestration layer for downstream APIs, perform functions of the backend API, and act as the API interface for UI apps. We are moving towards a unified GraphQL Gateway to support one graph through the company.

Why did PayPal need GraphQL?

When we chose GraphQL, we were looking for a technology to help us solve the following problems:

• Overfetched data: Our REST (representational state transfer) APIs sent a part of the response a client needs and some extraneous data. Since the server in a REST API dictates the shape of data, our UI teams spent a lot of time filtering and parsing data on client side, often using libraries such as Redux to format and store data. With GraphQL, clients are able to ask for a set of fields, and get exactly those fields back, thereby removing the need to do data formatting and reshaping on the client side. This greatly increased the time it took to ship UI features and make our apps lightweight.
• Avoid multiple round trips: Often, in order to call one endpoint that required a certain parameter, such as /getProfileById/{id}, we would have to call additional endpoints as a pre-req, such as getUser{username} to returnparameters such as id. This was a problem because we were making multiple round trips to fetch one piece of information. GraphQL helped solve this problem because it allowed us to fetch everything we need in one round trip.
• Keeping clients up to date: We heavily used API versions in REST APIs. Anytime we had a breaking change, we would publish it as a new API version. The problem we were facing was that when we created a new version, clients that integrated with older versions wouldn’t receive these updates without reintegrating with the new version. Sometimes the documentation or parameters would have changed in the new version. With GraphQL, we are able to send out updates and clients no longer have to worry about keeping up to date with versions. Since all updates are published to one endpoint in GraphQL, clients can fetch updated resources when they need it without having to reintegrate with a new version.
• Freedom to use any programming language when integrating: Historically, Braintree did not have a public API. We supported server SDKs and client SDKs. The challenge was that we did not have server SDKs in all languages. A lot of merchants did not want to use SDKs for whatever reason. We decided that we needed a better API for merchants on the server. This new GraphQL API offers great control, flexibility, portability, maintainability and freedom to choose any language when integrating, and provides extensibility of our global payments platform. You also get updates from the API as soon as they are released, without having to update an SDK.
• Uniform experience: Every flow in PayPal has its own NodeJS app and each team had their own implementation of ReactJS. We wanted to provide a layer that would provide a uniform frontend experience while giving us a backend to orchestrate the APIs.
• Ease of integration for those without prior domain knowledge: In our Identity team, we wanted to provide a uniform experience when using our services without needing domain knowledge of the PayPal system. We wanted to take control of the identity of all of our adjacencies and provide a secure way to convert a PayPal subsidiary account to a PayPal account.
• Field and method level instrumentation: We have internal instrumentation tools that show metrics of how long an endpoint took and what params were used, but it was hard to find which fields were being used. Without this information, we could not know if a field was safe to delete or if it was still being used. With GraphQL, we get field level instrumentation and a clear picture of which resolver took how long, common errors, and which fields were called. This field level instrumentation helps to intelligently deprecate fields that are no longer being used.
• Inconsistent developer experience when integrating with APIs: In REST APIs, different teams would have different conventions for the same variable, such as user,username making it harder to understand APIs. With federated GraphQL and one graph, all teams share the same schema, so it makes it a lot easier to identify duplicates and consistently name variables.
• Easier testing: GraphQL tools such as Apollo Client make it easier to add GraphQL queries within UIs such as React. It helps to keep code co-located and helps in debugging and separating concerns. It makes for a clean developer experience and improves testability of code.
• API Exploration: We spent a lot of time going through API documentation and figuring out which endpoint to use for a particular field. Once we had an endpoint, we would copy the urls and try them out in Postman. If we missed a parameter, we would go back to the documentation and search for the parameter again. It made playing around with APIs harder and more time consuming. With GraphQL, we got tools like Playground and GraphiQL that could be used to not only explore APIs and navigate through the documentation, but also make requests right in the tool. This makes the development process a lot smoother.

Photo by drmakete lab on Unsplash

How did we begin adopting GraphQL?

PayPal has an extensive suite of REST APIs that power the core functionality of apps and sits very close to the database. GraphQL is used as an orchestration layer in our apps. It sits between the frontend UI app and backend API layer acting as a backend for frontend (BFF). This means that a UI app talks to a GraphQL endpoint, and the endpoint figures out which downstream service to call. New features can be built directly within the GraphQL layer. Some teams have chosen to use GraphQL as a pure orchestration layer, while other teams use the GraphQL layer as a business logic layer.

The Checkout team was the first to pioneer the adoption of GraphQL. Mark Stuart shared our journey of adopting GraphQL in the Checkout app in this blog post in detail. When the Checkout team showcased their app, the benefits of orchestration and improvement in developer productivity really clicked with our engineering teams. This sparked interest across PayPal, and teams started developing demo apps to utilize GraphQL for their apps.

It was the new shiny thing. Everybody got excited about the hype, but what stuck out most for the teams was how easy it was to orchestrate downstream APIs and create a unified experience for clients. With GraphQL, all of the downstream complexity could be hidden and clients wouldn’t have to worry about figuring out which piece connects where. It made for a more coherent experience for clients.

Teams started building products, presented them in our tech showcase, and got other people excited too. Soon, everybody was curious. Once we got leadership on board, we were really able to take off.

How did we scale adoption of GraphQL?

As we started scaling our GraphQL adoption, we realized that every app was trying to solve its own GraphQL problems. Often, teams were solving the same problem and reinventing the wheel. We realized there was a need to unify these efforts under one umbrella and hence, we established a standards body. We provided support for tooling, front-tier and mid tier sample apps, error handling techniques, and learning resources.

We built tooling to help support GraphQL adoption:

• We established GraphQL standards for defining GraphQL APIs internally. These standards define naming conventions, GraphQL types, header standards, directive standards and error handling techniques.
• All GraphQL operations are required to specify special directives that describe all authorization requirements for queries, mutations and fields.
• Common libraries including plugins for logging, directives for data-classification, middleware for Apollo and playground, CLI, error classes, and Apollo graph variants.
• Template sample applications for frontend and backend.
• Learning resources to help onboard teams onto GraphQL.
• Slack channels to help answer common questions and create an internal GraphQL community.

Having a standards body and tooling was great and helped teams set up their graphs faster. However, we noticed that some problems persisted. We noticed that the individual graphs were deviating from the correct way of doing things, such as authentication. We lost a little control of auth rails in individual graphs. We also realized that having multiple graphs makes schema sharing harder. We wanted to provide a centralized point of entry, govern schemas together, model the data in a global way, and provide a way to reuse types. This is what led us to build a one graph gateway using Apollo Federation.

What advantages are we seeing with GraphQL adoption?

The fact that we were able to orchestrate adjacencies and convert the identity of a PayPal subsidiary into a PayPal account is a matter of pride. We initially launched our Braintree API and we were able to get it done really quickly. Time to ship was faster and GraphQL was able to report what piece of schema went where. The main advantages we have been seeing with GraphQL are:

• Developer productivity: Tools like GraphiQL and Playground are really great to play around with the API and explore documentation at the same time without having to check out any additional tools such as Postman.
• Access to entire schema: Since all operations (queries and mutations) are at the same endpoint, it makes it much easier to have access to all operations supported by the API.
• Team collaboration: Building UIs in parallel with GraphQL API helps increase team collaboration. Since GraphQL schema needs to be built upfront, backend and frontend engineers work together, thereby reducing silos.
• Paradigm shift: Since GraphQL mandates a design first approach, we thought about GraphQL in the context of enabling business use-cases, and built out the API with the customer in mind.
• Faster shipping: We are able to get features shipped faster. We were able to get rid of a lot of plumbing that made it harder to provide feature updates and keep feature parity. Before, we had to ship out an SDK in every language that our merchant worked in. Now, we can provide one single GraphQL endpoint that merchants can integrate with no matter which language they use.
• Simplify unification: Internal clients and adjacencies no longer had to worry about the complexity of the internal systems and figure out which API to call. The GraphQL layer hid the complexity behind the scenes.
• Analytics: instrumentation on how long a single request for a particular field took.
• Exposure and hiring: many people in the community are excited about GraphQL and it has helped us attract talent to PayPal. Our teammates are excited to share their learnings in the community and have been speaking at conferences, writing blog posts and making instructional resources. Vishakha Singh gave a talk on building a faster checkout experience at PayPal with GraphQL. Rohit Basu talked about working with Kotlin and GraphQL .We had multiple talks in our public conference JS @ PayPal about how we are using GraphQL in various apps.

What challenges are we facing?

Photo by Possessed Photography on Unsplash

We are still working to create a standardized approach to challenges in GraphQL technology such as error handling, authentication, file handling, and batch processing.

Teams are independently building their own graphs, which leads to duplicated efforts, different ways of handling and surfacing errors, and deviation from the standard way of handling authentication.

We are still working on incorporating internal tools. Since a lot of these tools rely on the status code of the API response — 200s, 400s, 500s, it has become harder for us to map GraphQL responses (all 200s) to these tools.

GraphQL at PayPal grew tremendously fast. Many teams built their own way of handling errors, solving GraphQL problems, and instrumenting to the internal logging system. After it grew, we provided support by adding internal plugins and middleware to normalize error handling, instrumentation, and reducing internal network chattiness, but we wished we built support sooner.

Our adoption of one graph approach has been slow. Teams have to change a lot of behavior of how they are currently making apps in order to get on the one graph, adding process and time to deliver. The challenge has been to tell people that now we have rules to add to the graph but keep them motivated to use one graph. Joey Nenni gave a talk at JS @ PayPal and talked about our approach to one graph and a potential solution to overcome this challenge.

How did we convince our engineering and leadership teams?

Our frontend developers saw the benefits of using GraphQL immediately. It was also easy to convince backend developers who were working on UI teams. They understood the power of orchestrating with GraphQL. For core platform APIs teams, we haven’t fully convinced them yet. When we introduced GraphQL concepts, sometimes we were told that REST can do that too. Yes it can, we can replicate what GraphQL does with REST as well, but then at the end, we are just recreating GraphQL. We haven’t gotten a full buy-in from developers across the stack, but our REST and GraphQL APIs can live together. We learned to not boil the ocean.

Photo by Christoffer Engström on Unsplash

To convince our leadership, we knew that it wasn’t enough to focus on the performance of GraphQL APIs. The performance of an orchestrated GraphQL API depends on the APIs it consumes. A GraphQL API is only as fast as its slowest downstream API. Instead, we focused on developer productivity and time to ship a product. We demonstrated that using GraphQL can help build products faster, improve team collaboration, and make it easier to explore documentation. When we introduced GraphiQL and Playground tools to our teams, they immediately saw the benefit in having a GraphQL endpoint and playground tool to fire off requests while exploring the documentation.

We demonstrated how GraphQL helps improve the productivity of internal and external developers, how GraphQL helps reduce the time to ship features, and how we were able to hide complexity from our clients. With GraphQL, we didn’t have to work on multiple SDKs for each platform. We built the API once, and we were done. Without GraphQL, we had no idea which fields were being used by the merchants and which endpoints were called. We did not have instrumentation on KPIs such as first integration to production. With GraphQL, we were able to demonstrate our learnings, instrumentations and field level insights.

How can you start adoption at your own company?

1. When first speculating whether GraphQL is the right technology, it is helpful to build a demo app that can demonstrate how GraphQL can fit in your enterprise architecture.
2. Bring the team with you — demo your app and showcase the benefits of using GraphQL, your experience of adopting GraphQL, the advantages you have seen, and the struggles you are facing to help others in the company.
3. Set up enterprise for success — set up a working group that will help to establish standards. Set up learning resources, provide guidance, and build tooling and support for GraphQL.
4. Get teams on board — showcase the advantages of using GraphQL from a productivity standpoint. Everyone wants to ship products faster and make it easier to integrate with APIs. GraphQL provides exactly that. Demonstrate to your teammates and leadership how easy it is to build a new feature with GraphQL and how easy it is to send updates to existing clients without having to publish a new version, while still being backwards compatible.

Big thank you

Photo by Wilhelm Gunkel on Unsplash

This article has been made possible by the contributions of our valuable team members. Thank you to Mark Stuart, Joey Nenni, Walmik Deshpande and Miriam Goldberg for agreeing to be interviewed and sharing their experiences. A big thank you to Mark Stuart for leading the adoption of GraphQL in PayPal, motivating me to share my GraphQL experience, and being an inspiration to our developer community.

Come join us!

Are you interested in helping out with our GraphQL journey? We are hiring across the board!