In the Beginning: How VR was Born, An Interview with Fred Brooks
Since joining IBM in 1956, Fred Brooks has been part of many of the most breakthrough computer technologies — working on some of the early computers. From the first generation of electromechanical computers, to vacuum tubes and finally modern transistors. Meanwhile becoming one of the most influential computer designers of the 20th century. Brooks has received numerous awards, including the National Medal of Technology in 1985 as well as the Turing Award in 1999.
Brooks is known for his book The Mythical Man-Month, which popularized and coined the term ‘Brooks’s Law,’ which states that a late software project is actually delayed with the addition of manpower. Brooks, as project manager, changed the IBM System/360 Series from a 6-bit to an 8-bit byte, thus enabling the use of lower-case letters. “The change propagated everywhere.”
The pitfalls when designing for Virtual Reality
What are some of the challenges of designing for virtual reality?
FB: In 1965, Ivan Sutherland gave a phenomenal speech [in] which he proposed virtual reality (VR). The key concept is “don’t look at this thing as a screen. Look at this thing as a window through which one looks into a virtual world. The research challenge in virtual worlds is [making] the picture in the window look real, move real, sound real, and even feel real.” And so for the last 50 [years], we’ve been trying to do that.
In our laboratory at the University of North Carolina at Chapel Hill, we started making head-mounted displays (HMD) in the 1970’a. Working with the technology that we had at the time, we used small television displays mounted vertically over the eyes and mirrors at 45 degrees that reflected the image into the eyes.
Some of the hard problems are image generation at necessary speeds, tracking where the eyes are, and developing the graphic capabilities to display high resolution colored images. The graphics processing has to be at a minimum of 30 frames per second (FPS) (Now the technology can manage to do 72 to 120 FPS).
The hardest technical problem with immersive environments is latency (the delay between the receipt of a stimulus by a sensory nerve and the response to it, in this case the screen).
How long does it take from the time you move your head until the image on the screen changes? The early systems had [tremendous] latencies. I’ve seen them as bad as 250 milliseconds, some of the best early ones were 120 milliseconds; that’s still bad.
A friend of mine was telling me how fighter pilots (in simulators) notice at 55 milliseconds that something is wrong; however, at 50 milliseconds they don’t notice. Now this is true specifically for flight simulators where the image is distant,so that you don’t have swift motion in front of your eyes. The same person told me 20 years later, “We’ve discovered now that if we’re doing aerial refueling, where there’s another plane and a boom closeby, we have to have latencies down to 25 milliseconds”. Similarly, in VR, it turns out that we need to get latencies below 20 milliseconds, preferably below 12 milliseconds.
So this was a flight simulator for military applications?
FB: Yes, this friend was with Lockheed Martin in St. Louis. They had six big simulators installed. If I recall correctly, they used these six simulators to train fighter pilots for air-to-air combat — three-on-three..
Now, flight simulators still create the best virtual reality experiences in the world. I had a chance to fly a Boeing 747 around London in one of the British Airways’s flight simulators — a $13-million machine in a three-story space, on a motion platform that threw the whole cabin around at necessary speeds. The best VR experience I ever had.
EMT’s and IED’s and VR:
Looking forward, what challenge can you offer to new developers and designers in the field of VR?
FB: An early milestone would be a system to train, four soldiers room-clearing (military application), or four EMT’s at the scene of an accident. They must be able to talk to each other, able to share real tools and patient dummies, , and virtually see the crowd around creating all the disturbance and difficulty that makes life hard on the working scene.
So we’ve got track eight eyes simultaneously, we’ve got to generate eight different images in real time, we’ve got to make it possible for them to see, hear, and feel each other, use the real tools, and see the virtual tools. We have to display the eight images separately to the eight eyes. We have to track hands and feet as well. So this is kind of an augmented reality problem, and in some ways, a mixed reality problem, and it is very challenging. The hardest part today is knowing where the eight eyes are at millisecond latencies and delivering the separate images..
What do you think will happen to augmented reality? Will virtual and augmented reality stay separate to each other?
FB: Well the real question is, What are the driving applications?
The difference between education and (military and first responder) training markets is that in training you are paying the trainee, and so the cost per hour of what you’re doing is important, because the big cost is the trainee salary — not the equipment or the applications. Education [already] has a cost per seat problem. And so military, industrial, that’s where I say it will take off.
“And so military, industrial, that’s where I say it will take off”
I talked with a fellow who trains firefighters. He said, “VR’s never going to be very useful for training firefighters…If you don’t feel the flames, you’re not in the situation, so you haven’t really trained. In firefighting, I want to know that the guy next to me is not going to quit halfway through the fire.”
The British Navy built a very impressive VR simulator for shipboard firefighting that gives users the haptic feedback of spraying down flames. It’s a simulator in which most of the visual imagery is virtual, but the smoke and flame is real.
Where should people entering the VR industry focus, what do they need to know?
FB: Well, first thing I would tell them is “Welcome aboard”. Developing VR and its applications is about as much fun as one can have in technology. Going through the successive generations of the technological development has been exciting. With each little step, we’re getting closer and closer to realizing Ivan Sutherland’s 1965 vision.
When was the collaboration element introduced to VR?
FB: As soon as we started wanting to use this to train teams.
And we still haven’t gotten there?
FB: Yeah. We can’t yet track eight eyes simultaneously and generate and display eight images separately and simultaneously, where they can see each other. No, we can’t do that yet.
“No, we can’t do that yet”
What are you excited for most in the immediate future for virtual reality?
FB: People putting it to work as opposed to play. The exciting thing about the flight simulator technology is now airline pilots will fly a new model of a passenger jet with passengers the first time they’re in it, because they will have trained intensively on the simulation of that model
When the geese took out all the engines on US Airways Flight 1549, Captain Sullenberger put it down at the right place in the Hudson for the rescue crews, and guess what? He had long been a pilot, but he had also been a pilot representative in charge of the simulator training for his airline. He said “No, I never did this emergency procedure. But I’ve done a lot of emergency procedures in simulators.” He kept his cool; that’s partly the fruit of the simulators.
“You can get excited about seeing lives being saved”
You can get excited about seeing lives being saved and the work that’s happening in medical VR; there’s not only remote viewing of operations for training, but also there’s robotics, the simulations, the training of surgeons with surgical simulators, where one can feel the tissue.
As a computer scientist what has most contributed to your many breakthroughs and successes?
FB: Being at the right place at the right time, which is the Lord’s provision. I was 13 when I read about [Howard] Aiken’s first American computer. I was sitting in the public library in Greenville, North Carolina, and reading Time or Newsweek. There the front cover was a great six-foot high, eight-foot wide monster computer. I knew then that was what I wanted to do. I pursued that line directly ever since and it’s been an immensely satisfying career. I was only a half-generation behind Howard Aiken, and so I knew most of the pioneers.
I was also at the right time and place to be part of the big operating system breakthrough — going from when a person operates a computer to when a control program operates a computer.
Who do you want us to interview next? Let us know!
Want to know all the secrets to developing VR/AR? Sign up to our email list!
Beyond the Headset is now active on Twitter! follow us for new articles: @beyondheadset
