The World’s First Mass Produced Full Body Wearable Virtual Reality System
Lessons Learned from Creating the Army’s Dismounted Soldier Training System (DSTS)
From summer of 2011 until the summer of 2013, I was a contractor serving as the Chief Engineer of the U.S. Army’s Dismounted Soldier Training System (DSTS). This is the Army’s “fully immersed first person shooter”. These days 2,000 to 4,000 grunts suit up each month and use the system for training and mission rehearsal. It was a royal pain to build given what was available in 2010, when the contract was in competition. Since then, I’ve seen a wide variety of commentary towards what a VR system — several using DSTS as a measuring stick for what should and shouldn’t be done for VR solutions.
I’m off to travel to the Silicon Valley Virtual Reality Expo in mid-May to hunt down possible solutions for the military for both virtual and augmented reality — my clients are asking for this. I thought I would give my two cents of perspective towards the design of DSTS — the first mass-produced full body immersive virtual reality system — as an open letter to the exhibitors so they can better understand how the military makes decisions in this business. I hope this will save some time and help the VR industry move forward.
The Army’s PEO STRI office alone spends $5 billion a year on training solutions yet it seems like VR private industry wants nothing to do with it because dealing with the government is frequently tantamount to experiencing pain at glacial speeds. But the prize of dealing with it is a steady, stable flow of significant revenue — just ask defense contractors Lockheed Martin, Raytheon, and General Dynamics who collectively earned more than $100 billion last year alone. Google, Facebook, and Twitter earned roughly $80 billion. The biggest difference is that the defense contractors have been around far longer: Lockheed (est. 1912), Raytheon (est. 1922), and General Dynamics (est. 1899). This is long-term revenue that can provide a stable base of funding.
When I’ve talked with vendors at conferences, through email, and on the phone, I hear the never-ending banter of “all you need is _____ and you’re done” or “yeah, we can do that”. In the lab, everything works fine, you see. Once you put something into the hands of the populace — or in our case, grunts all around the world — it’s a very different picture. Making a VR system that works is one thing. Making one that survives in the hands of users is another.
I am not an optimist or a pessimist. I am a realist. When I go to this conference, I would like the exhibitors to understand a bit about what we’ve learned from designing, prototyping, manufacturing, deploying, and maintaining this system.
Background of DSTS
The military has three kinds of training systems: Live, Virtual, and Constructive. The Live and Virtual variants are the most apropos to training soldiers on foot. The Live variant literally has soldiers going outside and practicing in large areas with fake towns, actors to pose as the bad guys (aka “OPFOR”), and a bunch of cool pyrotechnics (aka “battlefield effects”). This is exceptionally expensive (e.g., one two-week exercise at Ft. Irwin will run upwards of $20 million each and they do this ten times a year).
The smaller scale of Live training is practicing in “shoot houses”. These are the equivalent of a bunch of cheap walls to simulate the inside of a building. Targets will pop up and you shoot/no shoot. This is cheap but the training only covers a small (but important) subset of skills soldiers need to learn.
Commanders and non-commissioned officers (NCOs) want their soldiers to have a wide variety of training — ranging from patrolling a long road with possible IEDs to entering a sprawling town where snipers could be anywhere. They wanted to be able to make it dark, dusty, or whatever other effects they felt were necessary to get the soldiers prepared for war. Better trained soldiers result in far less casualties for both sides.
The military has historically turned to Virtual training methods mostly for aircraft and vehicle training. These are a lot cheaper than using the real thing and much safer. However, they are still very costly in that they require dedicated buildings to house — costly in terms of money, maintenance, and the fact that this space can no longer be used for anything else. The Army felt it was time to take the plunge and make a Virtual trainer for “dismounted soldiers” or “soldiers that are not in or on a vehicle” — but do so with a system that can be set up anywhere (no dedicated building) and packed up and set up again on a whim. The time seemed ripe. After all, more and more soldiers were heavy into first person shooter games, head mounted display technology was promising, mobile computing was on the rise, and there were a plethora of motion tracking technologies to choose from. All you gotta do is throw them together, right? Just make sure that the system can:
- Operate anywhere in the world, including universal power and adherence to the various nations’ wireless laws.
- Be able to simultaneously track the arms/legs/bodies/weapons for nine soldiers in the same room. By the way, making a single player VR solution usually runs into trouble once other players are added to the mix — the safety issues explode because more people generally equates to more accidents (e.g., running into each other, swinging a weapon into another player, dropping something on the ground for others to slip on). The Army is very precise and hardnosed on safety.
- Be able to daisy chain multiple systems together so you can have a lot more than nine soldiers at one time… like over 200 (Army company level).
- Put up with the ungentle nature of grunts handling the equipment and have a low failure/breakage rate.
- Include three additional workstations. One is to control the exercises the soldiers run. One is to run the artificial intelligence of the NPCs. One is to provide after action review (AAR) — basically two large screen TVs that can play back an exercise like a movie. This allows soldiers to learn from what they did or didn’t.
- Include five more workstations so other soldiers that aren’t suited up can participate, for example, as vehicle drivers or to soul-jump into NPCs for greater realism.
- All workstations must be high-powered enough to run 5,000 human NPCs in the game engine at the same time, in at least 30 frames per second, with the CPU never going over 50%. I realize that today’s framerate standards in the commercial industry have increased since then.
- Nothing can weigh more than 164 pounds (military standard “two man lift/carry” restriction). It seems like a lot until you have to factor in that this includes whatever weather-proof shipping cases that must be included and you are trying to cram as many things together because…
- …all packed parts cannot, collectively, occupy more than 100 sq ft in room that has no more than 800 cu ft of volume. No pallets or forklifts should be required, although some posts did have them available. This requirement was critical to ensure that the various Army posts could receive a system with little or no notice.
- It’s all got to fit in 40 feet by 40 feet of space when it’s all laid out.
- And 1,013 other requirements (literally). As a comprehensive Army training system, it would be a severe understatement to say that it’s nothing more than a VR first person shooter.
What could go wrong? Vendors regularly pitched to us their HMDs, motion tracking capabilities, mobile computing solutions… and, well, a bunch of stuff they thought would work because of what they heard from someone “of influence in the Army” or what they inferred from pictures/videos. There was always one or more gotchas that would prevent their solutions from integrating well — primarily for safety, maintenance, and/or performance in numbers. That last one, performance in numbers, essentially meant that they had never tried their solutions with nine people in the same room (an Army squad size) let alone forty-two (an Army platoon). They always felt that it should work until they actually tried it for the first time — which none of them had. The experience left me exceptionally dubious about any VR technology I come across.
When we set off to make this thing, we had a team of about 25 engineers that worked for IDI and a laundry list of sub-contractors and vendors. There was roughly 80 people total. We had to design, develop, integrate, prototype, test, secure supply chains, manufacture, train, maintain, and begin deploying the 104 systems — all within a year. Now that the system has been deployed for almost three years, I look back on it and think about the hundreds of things that I would have done differently; however, I am proud of what was accomplished even if, at a minimum, DSTS represents a significant step forward in advancing VR technology.
The Design
I have to be really careful here because there are trade secrets that people want to keep secret, so I will apologize in advance if details aren’t to your liking. I’ll go from head to toe and then I’ll jump into the support equipment.
The Head
The head mounted display (HMD) really isn’t head-mounted. It’s helmet-mounted. The idea is that soldiers would come in with their own equipment, including their ACH, and get suited up. Using their own ACH was imperative because each soldier’s ACH is form-fitted to their head. This is critical to ensure that the HMD remains firmly in place. If the ACH wiggled at all, you are headed down a quick path to motion sickness. Besides a loose helmet, another major contributor to motion sickness was caused by dirty HMD optics (i.e., don’t smudge them with your fingers) — although this was technically causing eye strain, soldiers would call it “getting sick”. In all of the dialogs I’ve heard with commercial VR displays, no one seems to address this. Please do. Asking consumers to simply throw their mobile phones in <device X> means they’ll be contending with dirt, scratches, etc.
Headphones… these were a challenge in that their requirements made it nigh impossible to find a suitable solution. They had to be noise cancelling, so the soldiers could be acoustically immersed. They had to somehow fit in the profile of the ACH itself (earbuds are unsanitary to reuse between soldiers). You’ve already reduced the number of commercially available headphones down to about a dozen models. Then came the hard part: Finding headphones that were certified to be stored/operated in the Army’s stringent range of temperature and humidity. 99% of headphones out there don’t even bother to provide said information because consumers don’t need it. The only thing left were the exact headphones that soldiers use in their vehicles; the good news was that soldiers knew how to put them on already and this was exceptionally helpful with providing intuitiveness.
Then came the microphone. When soldiers talk, they are either talking to each other face to face, they are using a radio, or they are using both (although the latter is usually by accident, but it was a requirement just so the realism would be there). The microphone solution had to allow for soldiers to talk freely as if they were live and then, with a click of their radio, allow them to talk through their radio. All of this must occur through touch, because soldiers’ faces are covered by the HMDs. The lesson learned for consumer VR providers is that you should only provide controls to consumers that they can use without looking. Soldiers are very used to where they put their radios and this kept the solution intuitive. This brings us to a good segue into what soldiers should expect the first time they use DSTS.
The Army’s requirement was that first time soldiers should be suited up and familiarized with the system within 45 minutes — remember that you’re asking someone to try something for the very first time. As a developer, I spent no less than 300 hours suited up. I could put the suit on and be in a scenario in less than two minutes (I timed myself frequently to find ways to reduce this time further). The moral of this story is that it doesn’t matter how fast a developer can do it — it matters how intuitive it is for first-timers. Familiarization includes tutorial levels you would find in a game, but in the Army it also included a long list do’s/don’ts for safety purposes. The faster you get soldiers into these tutorials, the faster they get accustomed to the system. Remember that soldiers were expected to perform all of the tasks they would normally do in real training within 45 minutes of seeing the system for the first time. See Army Field Manual 3–21.8 and its 602 pages of goodness to get an idea.
The Body
What are those black cables all over the body? What is the backpack? Why aren’t you using wireless? Etc.
About that backpack… It’s mostly a portable computer. The plurality of its volume is used to hold four high-capacity rechargeable batteries. Aside from what you’d typically find in a computer, it’s also a custom power distribution system. What? The HMD, headphones, motion sensors, and other parts of the suit need to draw power from the backpack — the details of which wouldn’t equate to a plain Jane USB system. The power distribution system had to be exceptionally efficient because it is a requirement that each suit can run for at least two hours under some pretty demanding computing conditions — this solution actually greatly exceeded this requirement. Each of the components require stable streams of power and so, whenever anyone tells me to “just use a laptop”, they are over-trivializing the problem in many ways beyond what I’ve described. For example, you must be able to roll on it — this is one of the 1000+ requirements — and that usually breaks laptops. If you get beyond that, good luck with heat, heat dissipation, and… I’m excited to see where mobile computing is and where it’s going.
We could have used a configuration that would put all of the computing off the soldier (i.e., wireless video, wireless trackers), used body tracking methods that don’t require on-board power (e.g., optical markers), just use a small battery to power everything else, and ditch all of the black cables, right? There are specific reasons why we didn’t or couldn’t — and it largely was dependent upon the quantity of soldiers that may share the same space, how the Army maintains systems, and legal issues (an overloaded statement that would require a TLDR warning). My message to the VR community is that the design of what you do can largely be constrained by scaling and laws (e.g., FCC 47 CFR 15), including all consumer products — not just Army. When I ask about the laws, it seems that 90% of the time or more I get deer in the headlights or “we’ll get around to that”. You can’t sell your stuff in the U.S. without compliance.
Finally, regarding the body, I will add that soldiers have the ability to grab and use virtual equipment from their bodies. The suits also have RFID antennae and transponders at the forearms and tops of the hands. Soldiers are given passive RFID tags that would represent their grenades, binoculars, or whatever piece of equipment they wanted. Soldiers would clip these tags wherever they would normally keep the associated gear (experienced soldiers keep the same gear in the same place on their body). Even though they can’t see the RFID tags because of the HMDs, they know where they are because they expect them to be where the associated real life gear would be found.
The Weapons
The DSTS included seven M4s, two M320 grenade launchers (attached on the M4), and two M249 SAWs. All were modeled to have the same weight and weight distribution as the real weapons (with Army, please don’t call them “guns”… you are outing yourself as a noob). For us civilians, the M249 outed us as weaklings because they were heavier than they looked — and they looked heavy. The Army required this realism; consumers think they want it until they actually have it. With first person shooter games, it’s always faster and easier to move your hand/thumb than it is to swing a weapon with your body — and you don’t break a sweat. The DSTS weapons had replica movements as well (e.g., bolt release). Although soldiers were blinded with HMDs, they knew how to handle these movements without looking because it was second nature to them.
The two most common questions I got with the weapons were, “What is the deal with the joystick/buttons on the forward grip?” and “Why don’t you have recoil?”
Starting with the joystick. It’s used for the soldier’s locomotion. The motion sensors on the body are only used to capture the poses of the soldier (e.g., arm movement, crouching, going prone, body rotation). The Army required a solution that would fit within 40 feet by 40 feet of space — and it must allow the soldiers to have an infinitely large space to conduct the exercise (e.g., an entire fake town such as McKenna at Ft. Benning).
Why not just have it free range motion? Because once the 40 feet of space was up, you had to use what’s called a “wall switch”. Basically, the soldier’s HMD changes its screen to motivate the Soldier to want to run in a different direction in real life, but straight ahead in the virtual world — we thought of many scenarios that would force soldiers to collide or get out of each other’s way (i.e., safety issue or it would disrupt training). If soldiers had to traverse multiple floors, they could run into each other in the real world — even though techniques of showing “ghost images” of other soldiers crossing their paths. Ultimately, the solution was good as long as you didn’t go beyond the 40 feet by 40 feet area and you didn’t have multiple floors. This is pretty much a shoot house — and the Army has many of those already.
The Army is big on safety and they couldn’t (understandably) stomach having the possibility of soldiers accidentally running into each other at full tilt or having to unnaturally stop for a wall switch or ghost images. It was a bit hairy with nine soldiers at once, but make it a platoon of 42 and you’re in trouble quickly.
What about omnidirectional treadmills? The smaller ones won’t allow the Soldier to crouch and/or go prone without the weapon colliding with something on the treadmill. The larger ones were too large to fit within the required area (they are also very expensive to buy and maintain). What about the big hamster balls? You can’t go prone because their curvature. They, too, are also expensive to buy and maintain. When it comes to solutions of this type, there is also a great importance placed on being able to run full tilt, stop, and turned directions very quickly — doing such was impossible to come by with the treadmill solutions I’m familiar with. Most of these treadmills rely on motors that, if they failed, would be a safety risk for the soldier using (e.g., running full tilt off the treadmill without knowing it because you’re wearing an HMD). More moving parts = High failure rates. Treadmills are making great strides and I believe they will overcome these issues soon.
Back to the joystick. I don’t like it. At. All. But it has no safety issues whatsoever and this fit well with the brass. No soldier will go into real combat and yell, “Where’s my joystick?! How am I supposed to move?!”
Overall, however, DSTS was not meant to be the end-all be-all of training. It was meant to cover many of the bases that Live training could do, but it could do so anywhere at a fraction of the cost. If anything, it would induce squads and platoons of soldiers to become more organic (i.e., used to working collectively as a team). If the Army were to wait for a system that could do it all, it wouldn’t happen for decades (I hope I eat those words). Current VR technologies are a great way to fill the gap, have the technology mature more, and bring down the costs.
Let’s talk about recoil. Where is it? Not in DSTS. Why? The Army didn’t require it at contract award. Later, they asked about it. It quickly because another “All you need to do is _____” moments from vendors. It’s not there because of a potent mixture of cost and safety. I summarize it by saying that vendors seemed to only think about the solution on the weapon itself and not the overall picture of logistics, power, and how soldiers actually use their weapons safely in a VR environment (and it had nothing to do with the actual recoil motion itself). You will see safe appear frequently in Army training.
Feet
The only comments or questions that I’ve heard regarding this part of the body is, “What’s that circle used for?” They were called “haptic pads”. They are very cushy, durable pads that prevent soldiers from unintentionally — and unsafely — wandering around the room. It’s a safety item. It has to be thick enough so a soldier can feel it through combat boots. These worked very well and were very intuitive for the soldiers.
Non-Wearable Equipment
The Army required a total of eight other workstations to be used for various purposes for flexibility (e.g., to add more players as desktop users), although the entire system could run fine using just one — which has always been the case for tradeshows that IDI would attend. These were equipped with the top processors and video cards at the time.
So, why not laptops? Remember that requirement about these workstations needing to run 5,000 NPCs at the same time and still have minimum framerates, etc.? You could find laptops that might meet these requirements at the time, but desktops of greater capability were far cheaper and more flexible to upgrade — the latter being very important for the Army.
This system has a bunch of components that are never seen in the media. Most of these components, such as FIPS 140–2 Level X compliant wireless access points and universal voltage power strips, are so esoteric to DSTS that they wouldn’t apply to consumers. What I would leave you with in this regard, was there was a lot more that the government required than what you will see in the videos or at conferences. For each part or method, there was usually detailed rationale as to why it is what it is. If you weren’t there for the development, you are speculating at best.
Game Engine
The media is confused as to what game engine DSTS uses because of the information available before deployment. Is it VBS2, VBS3, Unreal, Unity, or CryEngine? It was shipped to all of the sites with VBS2/3 only and that’s what you will find in videos that show soldiers using the system live (some will show Unreal).
Final Thoughts
DSTS made great strides in helping determine what works and what doesn’t work through use of thousands of hours of harsh use across the world — and not just through trade shows, developers, and a few curious consumers. Just like all things, it too will end and be replaced with something else. It is something that myself and everyone who had a hand in it is proud of because it forced a painful bridging of integration gaps of various VR technologies to provide a deeper understanding of what works and what doesn’t.
A special thanks to the awesome people I worked with to make DSTS: Greg W., Clarence P., Floyd W., Dan S., Robert S., Scott J., Shereaf H., Tim L., Peter J., Steve K., Rick N., Mike C., Jeremy K., Kiel E., Dino C., Jaime C., Jeanine W., Ross M., Aaron S., Andrew C., Mike B., Chris D., Matt K., Dustin G., Sean O., Juan V., Morgan M., Jamie M., Anton S., Dan Q., John H., Jason E., Cherylynn S., Samantha D., and many more — but most importantly, my very patient wife, Lori. Apologies for some of the fuzziness of the pictures — most are screen caps from my videos. All other pictures are public domain as published on Army websites or are publicly found on Youtube. All requirements stated in this article are approved for public release via Army document PRF-PT-00532 dated 3 Oct 2012.
