Shared Tech, Not Central Tech
When we started N3TWORK we made a conscious decision to not form a central technology group. Game studios often try to build central technology or platform groups to provide core technology for game teams, but that rarely works well. The best technology is developed against actual requirements for shipping games. Central tech teams end up considering potential, future requirements and balance those with requests from game teams. This results in misalignment between the game teams and central tech teams and neither side ends up happy. The game teams feel their needs are not being prioritized correctly and the central tech team becomes frustrated because the game teams don’t use the features they develop.
Instead we build the right tools and systems for each game, but we take the time to consider how that solution might be extended to support other, future games. We may not take the time to do the complete engineering required to make the system fully reusable, but we avoid building systems that are inherently un-sharable. This approach has allowed us to organically distribute large amounts of technology across the company, with each team contributing back fixes and improvements over time.
The majority of our shared technology exists on the server side of our software stack. We have several services that coordinate gameplay and live operations which are shared across all of our games. Each service provides a well defined API and a client that can be integrated into each game stack. The functionality of each service solves a fundamental need for a game, such as remote configuration deployment, push notification delivery, A/B testing, event scheduling or analytics. Games are welcome to develop their own solutions to new problems, but often adopting or extending an existing service solves the game’s need and it can be deployed and operated quickly.
Each service is built on a common software stack and has a common devops model, which helps engineers quickly contribute and extend the systems with additional features. We maintain unit and functional tests for each system to allow teams to iterate and verify that changes will not break existing games.
Some systems extend to interact directly with our game clients. For these systems, such as managing downloadable content, we directly fork the client code so each game can make any needed modifications to fit their exact needs instead of trying to manage a single shared repository across game clients. Client bug fixes and updates are exchanged between games as needed. This removes the need for immediate, direct coordination across games, but allows improvements to find their way back into each game over time. On the server the exposed APIs are more stable and we are able to share and iterate on live services. We maintain a staging environment for all shared services and we can verify any possible breaking changes across games before deploying to production. Once features have stabilized we share binaries between game servers to connect to our various services to further ease the integration across games.
Additional benefits from shared technology extend to live operations. Because our games use the same underlying systems, the live operations team can use very similar workflows to create new content and events. As new tools are built by a game team to improve that workflow, those tools can be shared between teams to improve how we operate across the company. Equally, individual people can help on multiple games.
Ongoing operations of our shared services is distributed across our game teams. We do not have a formal devops team to manage each service, but some engineers do spend more time than others focused on improving scaling and realiability. Ultimately we benefit from sharing knowledge between the team about how the systems work so more people have the ability to correct a problem when it arises.
We have been able to achieve a great level of software reuse and knowlege sharing across the company by developing from a game first perspective. This bottom up approach allows us to continually improve the systems we build and make improvements available to all games we make, but without a formal central technology group. Our systems do only what our games need them to do, which leaves them more focused and easier to maintain. We also encourage teams to pursue new ideas that will benefit their game knowing those new ideas will be ready and available for the next game we make.
