Be Lazy! Running a Games Team
Mike Bithell posted some hints and tips about tools and process the other day, and it reminded me that I have a collection of similar tips about running a dev team.
As a little bit of background, I came to games after having run development teams creating software for interactive TV (red button apps and web infrastructure) and for educational games (the Fireman Sam CD-ROM! and about 40 other titles). I was running these teams in the early days of Extreme Programming — which mutated into Agile, over time — and while there was a load of nonsense in there, there were also useful gems, and we were forced to learn in the trenches (while under constant assault by Those Who Set Release Dates).
So here are a few thoughts drawn from that experience, modulated by how the games industry actually works.
Usual caveats — some games, and some studios, won’t be a good fit for the general advice I’m giving out here. But you can always ignore me. Your mileage may vary!
What are you trying to do?
You’re trying to make a game. And to make money out of it, so you can make another game.
So take the shortest path you can to achieve that.
Stand on the shoulders of giants
You bring something unique to your game — whether it’s the setting, the art style, the narrative, the mechanics, a particular procedural algorithm. But for anything else, it’s probably been done before, so why do it yourself?
Adopt existing software, tools and libraries. Even if they cost you money, they will cost you far, far less in the long term than the ‘free’ thing you intended to write in-house. (It isn’t free. It costs you development, testing, documentation, debugging, porting, and support time.)
Use a game engine such as Unity, Unreal, CryEngine, GameMaker, Cocos2D, Torque, or anything else which can give you a leg up. Use external libraries — Havok, Bullet, FMOD, Wwise. Use asset store packs.
The usual objections are:
- ‘But they cost money’ — Most game engines don’t, these days, until you start making money. If you write every single line of code from scratch, you may never finish your game. Or it may be so buggy that no-one ever pays for it. So isn’t it worth paying a small cut?
- ‘But my code will be better/faster’ — You can stop reading now, Mr Carmack. For all others — is that really true? Are you better at writing physics code, than, say, the team behind the free Bullet physics engine, contributed to by people at Sony, MIT, AMD, and Google, and tested by hundreds of thousands of people over various platforms and configurations? And even if your game code is faster… are you going to write a level editor to go with it…?
- ‘But I like writing code’ — Great! That’s a reasonable thing to say. But here’s the thing — do you want to write an engine? Or do you want to create and publish a game?
- ‘But you can’t get at the source code if there’s a bug’ — That’s no longer true. Major engines offer source-code access if you really, really need it; other engines are entirely open-source. Many libraries are open-source. And do you really need that access? Many more people will be testing that engine, finding bugs, and a whole team will be working on fixing them, across all platforms.
- ‘But it can’t do everything I want it to’ — Could it, though, if you wrote a few systems on top of the out-of-the-box engine? Or if you downloaded a few additional extensions?
- ‘But nobody else has ever written anything like this’ — If that’s really true, then that’s the unique thing you’re selling about your game. You may have a novel procedural algorithm, or a unique set of mechanics. But does that really mean you have to write everything — core loop, renderer and all — from the ground up? Or can you simply write your unique code and drop it on top of something that already exists?
There is a tendency for coders to adopt the attitude that they need to have written every line of code in a game so that they can have full control of it. I know, I went through this phase as a coder myself. As you get older, you realise that what you really need to do is the minimum amount of work possible to get the desired result — or you may never get the result. Be lazy!
There are many, many drawbacks to writing it all in-house. Not the least of which is ‘Anna got hit by a bus. She’s in hospital for the next month. No-one else knows what this code does, and we have a bug, and the game launches next week…’
Regarding tools, a tool probably already exists to achieve what you want to achieve, whether it’s a voxel editor, a sprite atlas creator, or a localisation system. Look for existing tools first before rolling your own — you can have something that works now without a crappy hand-rolled interface which bugs you (and costs you time) for the whole project. Sure, you might have to do a bit of integration on those tools to sort our your pipeline, but that’s a relatively straightforward task by comparison to writing a whole tool.
But! Do remember to check out the tool or library first. Make sure other people are using it; preferably a large number of other people. This is a particularly relevant thing for the Unity Asset Store, which has a load of undercooked half-working things on it. Caveat emptor!
Build something that works right now
Ignore the fancy graphics, the epic story, the ten thousand pages of design document. Start with a dot that moves across the screen — something playable — and iterate from there. Get your basic game mechanics in, and working, with temporary graphics. Make it fun. If it’s not fun now, it never will be, however many bells and whistles you add. If it’s not fun, throw it away and do something else. How much time have you wasted, compared to how much time you would have wasted on writing design docs for 50 levels before you found that out?
Iterate. Make a fun thing. Test it. Make it more fun. Test again. Show it to other people, see if they find it fun. Repeat, extend, add new ideas, throw them away when they don’t work. Get the core of the game firmed up.
If you’re going to write a full design doc, make sure you nail the core loop & mechanics before you write it. Otherwise your design doc will be wasted as you go ‘these mechanics suck, shall we rework?’
If you’re going to develop a game story, don’t do it in isolation before you’ve proved your mechanics. You want your story to support the mechanics and to gel with them. Sketch your story, sure. Don’t nail it down until you know mechanically how the game will work.
Once it’s fun, expand your core game into a vertical slice — a level or two to test out how much fun all the core mechanics are, and to test out your intended look and feel through near-final graphics and sound. Not only does this allow you to prove your concept before you start modelling every single level, it gives you a useful marketing tool for talking to publishers, platform holders or investors.
Try to build systems, not levels. Once you’ve got the core mechanics, you want to be able to go ‘Yep, this works. Now let’s have fun scaling it up across 50 levels’.
Always have a game that works
Never get into a state where you go ‘yeah, by the end of the week we’ll have finished replacing the particle system, but the game won’t build right now’. Your game should always be buildable, and always be runnable, ideally without bugs. If a big feature is going to take time, send the coder off to do it on another branch. (You are using version control, right? More on that later.)
Don’t let bugs hang around to be fixed in the future — fix them now.
This is because:
- You never know when Biz Dev Guy might turn round and say ‘I need a build to show a publisher by this evening!’
- A broken build might mean your art team or sound team grinds to a halt.
- You don’t want bugs to proliferate. The bug-fixing phase is always horrible at the end of development — you can make that easier for yourselves by keeping them under control.
- Each time your dev team hits an unfixed bug, it’s likely to slow them down. All that time adds up.
Always have a game that’s fun
Never get into that state where you go ‘yeah, this bit kind of sucks right now, but if you bear with it, you’ll find the next level fun…’
That’s no good. Keep it fun, all the way through. It’s great for showing to other people, it keeps you and your team excited. Focus on the fun stuff first! You never know, maybe you’ll never need to care about the dull stuff like proper physics & collision detection. (See: Goat Simulator)
Or in other words — don’t put your eggs in one basket.
Plan and build for portability
By the time you release your game (probably several years later than you intended), maybe your intended target — the amazing Amstrød Joytopus-7 console — will be obsolete! What will you do then, if you’ve coded it specifically to address the Joytopus’s 7 custom co-processors in glorious 7-bit floating-point?
Or what if a platform-holder comes in with a wad of money saying ‘Forget the Joytopus, we want you to put it on the new Nøntündo console as an exclusive’?
You need to be thinking both ‘Hey, it might not be on the platform I intended’ and also ‘Hey, if it’s successful, we might have to port it to a bunch more platforms in a hurry’.
This doesn’t mean that you need to write everything for all models right now. What it does mean is that you should be intelligent about things like your input code — making it so that if you suddenly need to accommodate the Amstrød Joytopus-8’s new Mind-link controller, you can.
Input’s a good example. It’s likely these days you might want to think about a few different ways of controlling your game — via touch, via keyboard, via controller, and maybe via motion controls. So keep your input code separate, in case you need to rip it all out again or to add alternate code.
Same for any platform-specific stuff like rendering, audio, file access, achievements, that kind of thing, if you’re writing your own code for all that. (Wait — why are you writing your own code for all that?)
Plan and build for localisation
This is for very much the same reason — you may plan to release your game only in English (International), but then suddenly you get an approach from a pair of polar bears who want to license it for Arctic (Hungry). You want to be able to say ‘Yes, and please don’t eat me,’ at this point rather than ‘Ooh, err, we might have to rewrite a bunch of core syst — aargh!’
Separate assets from behaviour
Maybe your game just isn’t destined to succeed, and you need to build something else. Or maybe it’s wildly successful, and you want to make some DLC. Or maybe it’s time to start that next game.
If your mechanics and behaviour are easy to separate from your assets, you can easily add new levels, reskin the game for the sequel, or turn your cutesy platformer into an atmospheric horror platformer with a minimum of fuss. Or steal some core systems from this game and move them to game #2.
Whereas if your mechanics are so tied into your assets that the game stops working when you replace Fuzzywuzzy Unicorn with Trip Hazard: Serious Action Hero, you’ve got a problem.
But! Build only what you need right now!
Don’t build your game systems with multiple future-proof features ‘so that they can be reused if someone wants to add feature X, feature Y, or Z’. Because then you’ll actually need feature Q, and it’ll turn out that you have to throw away the code that supports X,Y and Z.
Don’t listen to your coders. They’re wrong on this one. They enjoy writing code to suit hypothetical situations. Sit on them. I’ve been guilty of this myself more times than I can count. Code which is overcomplicated ‘to support future features’ is always rewritten when someone comes to add future features, and it takes twice as long to write. Do the minimum you need to make the game work right now. Be lazy!
This is a hard balance to strike. Mostly it’s about code encapsulation — my input controller example above as a case in point. Don’t write your console game with a bunch of generic touch code “in case we someday need to support touch”. Do put your controller input code all in the same place, so that if you need to rip it all out and replace it, you can.
Remove the hassle
Streamline everything that you can. It’ll take a little time up front, but will make your life much easier as development goes on. You’ll probably need a coder or scripter involved (although Actions in Photoshop or Macros in Word are also useful). Editors like Unreal and Unity allow pretty comprehensive in-editor scripting and automation; and Python is great for gluing together command-line tools or transforming data.
This stuff doesn’t need to be overly complex – it can simply be a set of useful command-line batch files accessible by everyone on the team.
The key thing here is to think about what happens as you go through day-to-day dev processes: ‘was that task a pain in the ass’? If so, make it less of a pain in the ass.
Say, for example, you need to import a bitmap into your engine.
The simplest thing to do is to document. If to import a bitmap you needed to go and look up how to do it — say, to look through the source code, or to look at command-line options for an external tool, or to ask Anna — then write down how to do it (in a place where the whole team will know where to find it) so that you never have to ask Anna ever again. (Remember that oncoming bus?)
But if the operation was non-trivial (it took five steps) and you are pretty sure you’re going to need to import more bitmaps to the project, then automate it! Write a batch file, or extend Unity with some import code. A good rule of thumb is ‘Is this the third time I’ve had to do this?’
Automate everything you can
Is a process complicated, or multi-step? Does it have a bunch of hardwired numbers that need to be put on a command line? Are all the bitmaps at 90% quality, and you have to remember to set that at import time? Automate it!
This is because:
- You never know how many of these things you’ll have to do. (‘We’ve added another 50 levels! A bitmap for each level!’ or ‘We’ve added a whole new platform!’)
- You never know how many of these things you’ll have to redo. (‘We’ve decided to downgrade the quality to 60% across all levels!’)
- You never know how many of these things you’ll have to get intern Dave to do. (‘What’s a bitmap?’)
- You probably can’t remember the settings (‘Did we say 60% or 70%? Did anybody write it down? Shall I check the last one, or just guess? Was it 75% to fit in the memory capacity on the Nøntündo Sp00n?’)
Make your game easy to run
If someone has a copy of your testable game, make it simple to run. Don’t require command-line access, or for the screen resolution to be reset, or particular drivers to be installed, or for it to be copied across from a USB key. Keep it really, really simple. Remember your Biz Dev Guy will have it, and you know what he’s like with technology. If a load of steps (drivers, copying etc.) need to be carried out up front, make that happen when the game icon is double-clicked.
Make your game easy to package
Remember Biz Dev Guy? He suddenly needs a copy of the latest build with that particular new feature in it by tomorrow morning. It’s 8pm now. Do you want to have to drive to the office and do it for him by checking out the latest source tree, configuring everything, test everything, then look up the FTP site details and uploading it, bearing in mind it’ll be 1am by the time you’ve finished? Or do you want to be able to say ‘I’ll set it going now — you’ll get an email when it’s done’ and just click on a web-link?
Use a build system of some sort if you can — something like Ant, Make, Maven, Incredibuild or a series of script files— and make it a really simple thing to trigger and publish a build. Not just for BDG’s benefit — you’ll be doing it every few minutes yourself just prior to release.
Some engines aren’t set up well to do complete remote builds, but at the very least, make it a single batch file or single menu item command to package your build, update version numbers, and export it to a folder.
(Ideally, of course, you want to be able to say ‘hey Biz Dev Guy, a fully tested latest build is already available at the usual download link, just like it was the last 10 times you asked’ — we’ll talk about Continuous Integration a bit later on.)
Make your game easy to test
The game’s writer, Bob, wants to download and test the game at his personal mansion in the Seychelles to look at a particular cut-scene in level 73.
Cool, you’ve got a build-and-package system, so he’s got the latest copy. That’s great.
But now he has to play 72 levels to get to that cut-scene — and like 83% of game writers he’s terrible at any kind of hand-eye co-ordination, so can’t get through level 1. Oh dear!
And now the game’s crashed in a place it’s never crashed before.
“What happened?” you ask Bob.
“I hit the spider with a flannel and the game just quit to Windows,” he says.
“What colour flannel?”
“I think it was blue… no… green? A sort of bluey-green. I can’t remember. Anyway… I’ve got to go, my manservant has my jacuzzi full of champagne warmed to the exact temperature I need to cure my writers’ block.”
Development builds of your game should make it really easy to:
- Jump to any point in any level.
- Allow cheating (e.g. God Mode)
- Robustly save a copy of the game state right now, so it can be restored, or dug into for debugging.
- Robustly load a saved game state, so you can get back to a thing you wanted to test. (Also, save/load code is often really fiddly, and 20x more fiddly if you leave it until late in the project – this means it’ll work earlier.)
- Save game logs and, ideally, core dumps or stack traces (ask your coder).
- Report info on the host system, such as graphics and sound drivers.
- Allow screenshots.
- Report all this stuff to your bug database (you do have a bug database, right?)
And it should be really easy to turn all of this stuff off for a full release. Unless you want all your players to have access to God Mode.
(It should also be really easy to hide it in a build, but access it through some sort of menu option or Special Button. You will need to be able to do this at a moment’s notice on the show floor at E3 while your comrade desperately tries to juggle three flaming chainsaws to keep a queue of thirty disappointed customers entertained.)
Add more debug options
Make it really easy to do things like change game resolution, turn on / off player damage, move the camera around independently of the player, play everything at half speed, show all the physics collision, turn on logging options…
Given our principle of ‘do only what you need right now’, you only really need to add this stuff when someone on the team finds they need it. But be prepared to keep adding to this stuff all through development.
Ideally, let this be changeable on the fly at runtime. At worst, let the game be started with a bunch of these options already turned on — a particular configuration. You could specify them in a config file! Which leads me to…
Make data files readable
Where you can, use readable text for things like your level data, not crunched binary files. Plain text, XML, or well-structured JSON. Huge advantages:
- It’s readable, obviously. You can look at a data file and go ‘yeah, that’s right’.
- Something not right in a level? You can work out if your editor tool is producing garbage data just by opening the text file and looking. Or if it’s your game code at fault.
- Source control systems (yeah, yeah, we’ll get to those) are much better at dealing with text files — merging away conflicts, giving you a way to work out that it was Yevgeni who added that data line that broke all the feline NPCs.
- You can easily automate changes that need to be made to all data files (search and replace!)
- You can easily write script-based preprocessors to make changes to your text data files, or even to generate them from other types of data (say, automatically import data from a spreadsheet).
- You can easily search for particular pieces of data (‘Which levels refer to Trip Hazard?’)
- You can hand-edit files if you haven’t got your editing tools ready yet.
But! say the coders… these files are far too slow to parse, and far too big on disk!
Firstly — that’s unlikely, in this day and age. Secondly, you can always write a tool to take the text and pack it down to binary, if that’s what you want to do. But make that an automated thing. (And test it. A lot.)
Include a version number
Clearly show a version number with every build, and increment it with each new build you make (make it automatic!). Show it in the game, on a menu somewhere, but also include it somewhere on the disk build in an obvious file — as if the game has crashed, you can’t read the number on the menu screen, can you?
No, not the shoot-with-paintballs kind… here are some things you can think about putting in place to make life easy for your team.
Backups & version control
Make sure everyone is using version control. Use SVN, Git, Perforce or whatever else. If it’s overly complicated for some reason (annoying command-line syntax) then use a client, or automate it away with some batch files. Make it simple, easy, and natural. And then you get all the benefits — version tracking, the ability to work out whether Anna or Yevgeni broke the build this time, the ability to go back in time before they broke the build so you can carry on working, and a host of other useful features like branching which you should make your coders and/or release engineers care about.
Ideally, don’t use your own server. As it will blow up at an Inconvenient Time. Use something with loads of backup and support who know what they’re doing. If you are using your own server, back it up offsite too.
In fact, back everything up. Source files, design docs, the works, anything you’re keeping on your own computer or in your office. If you’re using Dropbox or Google Drive for stuff, that’s great, there’s built-in redundancy there. Don’t keep all your core game files on a USB key. Don’t think ‘oh, I’ll back it up next week’. Do it now!
You can think about continuous integration (CI) tools like TeamCity or Jenkins. It’s not something to fit every team — it’s more useful for bigger teams, and some dev tools aren’t a great fit for it. If set up properly, the theory goes something like this:
- Anna checks in a code change to the central repository.
- A build machine somewhere goes ‘ooh, a code change’ and starts a full compile of the game.
- The game has a compilation error. The build machine starts flashing red alerts and mails Anna a rude message saying ‘oi, you broke the build — fix it!’)
- Anna fixes it, and checks in the fix.
- The build machine rebuilds the game. This time it all works, so it starts to run a bunch of simple automated tests on the game (i.e. ‘does it start’, or ‘does it fit on a cartridge’ and so on). It suddenly doesn’t fit into the required memory footprint for the Nøntündo Sp00n, so the build machine sends Anna another rude message.
- Anna tries another fix, until it works, and the build machine is happy with the tests. The build machine copies the successful build to an output folder somewhere, where Biz Dev Guy can download it and show it to Nervous Investors.
- Overnight, the build machine takes the latest successful build and runs several hours worth of tests — stability tests, loading and exiting every single level, loading random levels and auto-pressing controller buttons, all sorts of things. At some point, the build crashes. The build machine sends all the logs and screen captures to Anna, blaming her for the new instability… it all worked fine last night.
- Anna fixes it again. This time, it all works.
Repeat until you have a game.
(There’re a few steps in there that involve automated testing. That’s a whole other article. It can be hard to set up, but it’s invaluable to have stability tests running constantly if you have a spare machine. Doing this stuff early means your life will be much easier when it comes to platform certification before release.)
Issue Tracking & Bug Tracking
Have a shared database of some sort which keeps track of the team’s TODO list, and of any bugs reported by testers & clients. Your producer should have introduced this anyway, in an attempt to keep everything on schedule. If they’re not doing that, fire them and get a better producer. Use something like Jira, Hacknplan or Trello or similar.
Coding & Naming Standards
Have coding standards. It doesn’t really matter what they are, so long as you have them and stick to them. This covers things like naming of variables, functions, files, location of files, all of that sort of thing. Your aim here is that when that bus finally hits Anna, any other coder should be able to look at her work and figure out immediately what’s going on. (To be more kind to Anna — it’s also so that when your wildly successful game company hires coder #75, they can quickly understand what the other 74 were doing. Or so that when you export your codebase to the poor third-party porting team, they have a clue… )
Naming standards for files are also important, because if artists call some files things like
Sprite-Fred.jpg and others
Untitled-75.jpeg, it can be terribly hard to do any sort of automation. Or even to just look at the file and figure out what it is. Keep names readable.
The other reason for naming standards in code is that you can automate other things… like documentation…
Have a Wiki, or a collection of Google Docs — some central Repository of Knowledge that everyone can access and edit and add to. Keep it organised. Use it. Don’t let the info go out of date. If you spot something out of date, fix it now. No, right now.
And as I said way up the page when I was talking about removing friction — if you’ve had to figure out how to do a thing that other people will have to look up again (including you, having forgotten how you did it), document it.
If you’re going to document code, document it actually inside the code, or it’ll never be kept up to date. If you want a web version of it to refer to, use one of the many autodoc systems (such as Doxygen) out there to generate that.
Don’t write down the same thing in two different places, because if you do, one will be out of date with the other as soon as you can blink.
Project management system
Agile? Waterfall? Postit notes on a wall? What works?
It turns out that it doesn’t really matter. So long as you have some sort of system in place rather than none at all.
In the first part of this now-ridiculously-long article, I mentioned using off-the-shelf components and libraries to give you a leg-up.
The same is true about all aspects of game dev. If you aren’t an expert in the thing you’re trying to do, then get help from someone who is.
This could be paid consulting, or hiring in an outsourcer. They cost money, but rather like buying an off-the-shelf component, they will probably cost you far less than it would for you to learn how to do the same thing, and you’d probably make a much worse job of it.
Are you really as good at PR as someone who’s had years of experience at it? Are you as good at QA as a QA team? Are you as good at writing or localisation? How good are you at talking to the public in front of a stand at a games show?
Even if you are… can you juggle all of it with everything else you’re trying to do?
But even if you don’t want to spend money… just talk to people. Ask around. You can find advice on forums, get it from other devs, on Twitter, via web search. Most of the problems you’ll hit have been hit by other people… you are not alone. Which leads me to:
One of the best things you can do for you, your game, and your studio is to be a part of the industry.
Go to games shows and conferences, see what other people are doing, it’ll inspire you. (Let your staff go too — can’t afford it for the whole team? Run a lottery.)
Go to Game Jams and local talks and meetups. Join organisations who can support you, such as Ukie and BAFTA.
Meet people and talk to them. Be open — help them out if you can, and you’ll find that help reciprocated. Talk about what you’re doing. Talk about the industry. Talk about how excited you are about your game; enthuse other people.
Show off your game. Put the controller in someone else’s hands.
They’ll find a bug. Thank them.
Then put it in the bug database. 😉
Got networking? Put that in first. Before everything else. Trust me.