AMD Crimson with Fury

I enjoy playing games, and while I was very injured with bilateral hand fractures this year, a friend suggested I try out something which I hadn't seen since the 1980’s. A remake of the pioneering classic vector graphics game “Elite” lovingly hand-coded by David Braben and Ian Bell. Elite: Dangerous is a modern-day re-imagining of this classic game:

1980’s game “Elite” originally written by hand for a computer made by Acorn called the “BBC Micro”

Most reviews of Elite: Dangerous generally agree with each other that the game itself can be a little “same-ish” after players have experienced the universe for several hours, although they tend to agree there’s nothing quite like the experience it offers. Although there are big plans for expansions in “Horizons”, in the base game the experience offered basically involves flying about, travelling between star systems and various space stations, essentially talking and gawking, and plumbing the depths of space-station BBS’es to load your cargo hold with every sort of imaginary space-trading goods, from electronics components, to cryogenic-frozen non-player-characters (who've voluntarily sold their digital selves into virtual imperial slavery). It’s kind of like a more British version of Star Citizen you can actually play right now, and it looks like this:

Needless to say with views like this from my starship cockpit, I was instantly hooked!

So I discovered after a few days that I could combine my hand therapy (both my wrists were injured in a skating accident in July this year) with playing Elite: Dangerous while using a Logitech finger-trackball. With this as my excuse, I was soon playing a few hours per day, and Steam tells me the total is approaching 146 hours. Then, around August, the troubles began. Frontier Developments, the company headed up by Braben, released their latest update to Elite: Dangerous (1.4.1) and suddenly, the game was a slide-show instead of frames coming at the screen fast enough to give the impression of full-motion video. The problem came about with a confluence of releases for things. Updates to the game itself, but more notably Microsoft’s release of Windows 10. As well as Advanced Micro Devices (AMD) release of their new Radeon video card driver: Crimson. Elite: Dangerous players submitting tickets en-masse and complaining loudly in forums were encouraged, and I was among them, to continue their complaining so Frontier Developments could “see the formation of a pattern” — of players plagued by these performance issues, particularly those using Windows 10 running AMD Radeon hardware.

Firstly, it was widely believed that the good people at Frontier were working on it, and working their fingers to the bone to solve our “poor performance in Supercruise” issues But then, in the spirit of dealing with things swiftly and decisively, the buck was essentially passed to AMD. I’d like to say “Please don’t be sore at Frontier Developments” here — I'm absolutely sure, because I assume they’d like to continue enjoying the benefits of being fully employed game designers and programmers, that behind the scenes they were doing the very best they could. Games companies don’t ignore their players who are consumes of their products. Frontier Developments were interacting with players, by stating the only thing they can responsibly respond to as a game company — find a pattern of people being affected by a section of code, and attempt to rectify whatever is causing it as quickly and quietly as possible. But when it became obvious to them that might have been less-than-responsible for these particular poor performance issues, Frontier Developments released a statement to the effect that they were “working closely with AMD” to solve the problem, and suddenly the bug appeared on AMD’s Radeon (then known as Catalyst) driver list of “known issues” as number:

  • [59475] Elite: Dangerous — poor performance may be experienced in SuperCruise mode

Now, as players of paid-for games, we are essentially using non-free software. In an extremely competitive marketplace where companies have, as a top priority, the need to keep the running engines under the hoods of their fancy graphical stylings fairly secretive. So the only useful thing players can help with to solve these sorts of problems when they crop up, is do as the developers ask — everyone has to individually submit support tickets. Submit them, again en- masse, along with as much detailed information about how to reproduce the bug as possible, as well as any system information developers request like Microsoft DirectX Diagnostic files. This is the only way that “patterns of players not being able to play” (most likely because of some suboptimal, possibly buggy shader code somewhere in the executable) begins to appear so that developers can get down to the nit & grit of debugging work. This results in rolling out incremental changes, which hopefully eventually fix everything, so that players can get back to zooming happily around a virtual galaxy (rather than unhappily begin requesting refunds)!

For some, this feels overly like stone-walling. To outline a simple scenario as to why the community responds with outrage at being affected by poor performance issues, consider this. Every time a game company rolls out an update which causes massive performance hits on certain sets of hardware configurations, this impacts their credibility as a company which is able to deliver a high-quality gaming experience. If the game company is charging top-tier rates for the experience they offer, they can scarcely afford to do this. Somewhere in the code, someone has written something suboptimal, and while communities are generally forgiving of mistakes, sharing enough information to “name and shame” individual software developers or engineers who’ve made one is an impossibility — communities aren’t that forgiving to entities upon whom punishments like firings can be enacted.

However, there is another problem which works the other way. Simply by being too open, transparent, and communicative with player-communities, game companies could do irreparable damage to their business by sharing too much information. As presaged before, what’s under the hoods of most games is kept as tightly under wraps as a formula-1 race-car engine, and with good reason. No game company with MMO-capable multi-player servers as part of the gaming experience it offers wants to deal with hackers and “Lizard Squad” -like groups of DdOS’ing script-kiddies, whose attempts to “play the way they want to” ruins the fun experience for the majority of the community. So players experiencing these issues just have to do as they’re asked. Again we’re back to the developer request for players to individually submit their support tickets and hope. Meanwhile, they must simply deal with shoddy in-game performance, and trust that eventually these wrinkles will be ironed out.

But in bad cases, sometimes things get worse before they get better. For example, with Elite: Dangerous, when AMD released Catalyst version 15.20 things, went quickly from bad to worse. Elite: Dangerous became completely unplayable and the game experience swiftly went straight to hell. Players wondered aloud and on forums. Perhaps Frontier Developments’ code-base just wasn't agile enough to keep pace with AMD’s driver update schedule? Players began to make forum posts about why the issue had been allowed to persist for so long, and repeatedly asking what they could do to diagnose the issue and help solve it. It was the players who put out the most information by poking around under the hood of the game’s engine and drivers (using tools like GPU-Z), discovering that for whatever reason the game’s executable was saturating a single core of AMD’s graphics hardware instead of paralleling the tasks to the hundreds of GPU cores on the cards. The companies remained completely silent as player-whine reached flame-out point and people literally began posting they were switching to Nvidia so they could play Elite: Dangerous, and that they would never buy AMD hardware again.

So clearly for many people this has become a serious problem which required a speedy resolution, and beyond “the problem is more complex than we first anticipated” neither Frontier Developments or AMD have made any further statement as to how they might resolve it eventually. Indeed, it hasn’t been able to be resolved yet — and may still take some time to resolve. Since the release of the new “Crimson” driver which replaces the 15-year-old “Catalyst Control Center”, AMD have vacillated from position to position, claiming they’re doing everything they can, to suddenly having “clerical errors” which simply remove issue [59475] from their “known issue” list (in the apparent hope that removing it from the list will make the problem itself vanish into thin air). After these things persist for such lengthy periods, one can be forgiven for starting to think this has very little to do with bugs, and rather it has much more to do with inter-corporate politics replete with enterprise bargaining. With this in mind, I’ll try to frame things from a different perspective.

Let’s start with what we know. Frontier Developments made Elite: Dangerous, and it performed well under AMD’s Catalyst drivers version 14.4 and previous (and possibly up to version 14.9). Then, for reasons unknown, and for which AMD and Frontier decline to relate to players, Elite: Dangerous began severely under performing following the roll-out of certain updates. The other thing we know is that much like Nvidia’s “GeForce Experience” drivers, AMD’s Catalyst drivers are HUGE (some 300mb to download now and counting, a massive jump up from 14.9 which was a “mere” 148mb download!). What possible reason could a company like AMD have for writing such a massive driver-set? Surely, the base drivers themselves would come to only a few megabytes compiled?

Finally, the last thing we’re aware of, is AMD “scrapped” Catalyst, to replace it with a completely new driver called “Crimson” (which according to Frontier Developments was going to solve all of our Elite: Dangerous problems — and very unsurprisingly did not). AMD made much noise about Crimson when it was an upcoming release, claiming they were reviewing everything from scratch, but technical details were incredibly sparse on the ground. Turns out, while the marketing material abounded, the actual driver itself changed very little (particularly according to Linux community sources).

Now, let’s consider how AMD have been treating their customer base as a marketable resource. This is evident in the way they have released Raptr, upgraded it to “AMD Gaming Evolved” then integrated this program into the installation process for Catalyst (and it has remained integrated into Crimson, which like Catalyst bundles all AMD’s Radeon drivers into a combined architecture for all users of AMD’s product-lines).

Why would AMD bother doing that? Well, just like Nvidia with “GeForce Experience” (occasionally a bad experience for some), AMD has recognised the potential for exploiting their entire user-base to the greater glory of the almighty advertising dollar! Not only can AMD Gaming Evolved be exactly like Nvidia Geforce Experience in terms of advertising more directly to players of games about more games (and — of course — the flavour of the month’s “most advanced” gaming hardware to play them on). This kind of system-tray specific TSR software scans the systems it runs on, ostensibly in order to “optimise” games for improved performance. But it’s particularly murky exactly how this process operates, and there’s a lot more going on behind the scenes. If I were some kind of jaded, cynical person, I’d say the possibility exists there are profiles for each gaming executable running inside the enormous driver code, along with optimisation routines which communicate back-to-base. If this were happening, perhaps gaming companies would have to “work closely with” much larger, design/manufacturing companies for this privilege (even if they are somewhat “weightless” in this era of lights-out, hands-free hardware design/manufacturing silicon chip fabrication).

At any rate, the closed-source, architecturally-unified, feature-combined graphics driver Catalyst is huge. About the same size as Nvidia’s 300+ megabyte equivalent. We don’t know, but this software could be filled this with little recognition functions which recognise whichever game is executing, then selectively adjust performance tweaks accordingly. They are certainly filled with reporting code which communicates information to their publishers. Particularly for popular games, these applications are able to inform their software publishers which users are playing which games. In fact they proudly advertise they are doing so by awarding points for each minute of gaming accomplished while the software is running. Geforce Experience and AMD Gaming Evolved constantly talk to to their publisher’s websites (especially if an error is encountered), just like Steam constantly communicates with Valve. This is mostly a good thing, sending off crash reports and error dumps to be analysed. Except design/manufacturing companies aren't publishing software, they’re publishing software drivers to run their hardware.

Armed with the knowledge of who is playing what, and the average price-per-unit of each game delivered digitally from Digital Distribution (DDN) companies like Valve (or directly to consumers from game companies like Frontier Developments), hardware design/manufacturing companies could potentially use “strong-arm” business tactics to force game design/studio companies to work closely with them to provide optimisation services to players. Essentially, this would take the form of larger companies “politely asking” smaller companies to pay to “maintain optimisation” of hardware for their products, to ensure their products are advertised in the larger companies software. Longer term, only popular games can “ensure performance” (which basically means flipping bits in recognition code so games actually play the way they originally did, before monthly driver-update schedules roll out “feature upgrades” to the drivers which could be considered as what degraded the performance in the first place).

In this circumstance, what players would notice is that following driver-updates, games would suddenly under-perform. Following this, optimisation software might suddenly “discover” optimisations to apply to an under-performing game, which would then perform as it used to. This would provide graphics card manufacturers with a fantastic excuse to make an absolute truck-load of money out of nothing other than drivers provided for old hardware, even that which was licensed to other manufactures, which could have been purchased by end-users anything up to years ago.

Nobody on a board of directors would seriously infer that such abhorrently unethical behaviour by large companies occurs, especially in their relations with smaller companies — unless it got out of control (i.e. was found to be verifiable and true, and not mere hearsay from some random person, buried in the depths an obscure ranting article concerning graphics card performance in a particular game). Because, let’s face it, if this was actually what was happening, it would mean that some middle-manager in some driver development department could suddenly be pulling in millions of dollars for some company (in cash payments from other companies around the world). No executive-level manager worth their salt likes to put the brakes on such go-getting initiative unless they absolutely must (i.e. someone with scruples is about to blow some kind of whistle).

In the world of business, a large company acting in such an unconscionable way probably wouldn't bother to enact such heavy tribute from a small company if their games weren't popular. If they did so, their products would become known as “under-performing”, and their bottom-line would consequently suffer from vastly reduced sales over time. But in a large company’s interactions with other large or “big-name” companies, (or companies which are offering a premium product which is rapidly gaining traction in terms of user-base over time), this could pose a unique and dangerous problem in terms of business relations. How much can a smaller games company like Frontier Developments afford to “co-operate” with a much larger design/manufacturing company like AMD? The larger company’s resources must dwarf the smaller one by a large margin, so there would be definite limits on how much of the smaller company’s overall solvency and agility they can control without things getting out of hand and businesses going to the wall and/or filing for bankruptcy.

Now, I’d like to emphasise hat I don’t know what is happening. I don’t know why Crimson is so huge and so terrible and as yet hasn’t been an improvement on Catalyst. I don’t know why AMD’s (or Nvidia’s) priorities in terms of releasing driver updates seems so piecemeal and weird. I only know that Crimson is apparently far larger than it needs to be, and that AMD have never satisfactorily explained why it is so large, and instead of releasing a small and proper driver which more people seem happy with than not; why they decided to go all-out in creating and marketing an entirely new product. I don’t know why Elite: Dangerous suddenly began under-performing following driver these updates after Microsoft’s release of Windows 10. I don’t know why software companies are so close-mouthed about what exactly is the root cause of this performance problem. Again, what I have outlined are simply guesses. I am not making any pronouncements, and I am not an investigative reporter or any kind of professional journalist.

But I’d like to point out that because no-one acting in any official capacity is inclined to share any insight concerning why players are suddenly plagued with performance hits in games which used to run well; the only way to stop people thinking the worst (see above) is for software developers and engineers to step up to the plate and fix certain problems. That, or official representatives need to go public with cogent, believable statements, on the record, about their assessment of what’s causing these performance issues. Such statements would have to go beyond the standard by-lines of “this is a more complex problem than first imagined, and inexplicable difficulties have been unavoidably encountered”, because remaining close-mouthed and refusing to discuss the issue isn't quite akin to admitting fault or admitting to being strong-armed into shelling out cash in order to get frame-rates turned back up by optimisation software, but when things drag on for months at a time and larger more popular games leap-frog over “known issues” consistently, things can get so protracted, it certainly starts to get close.

After all, all commercial companies must understand: in the absence of veracious information to explain strange or unusual circumstances, people will invent their own explanations.