In 2009, while working for WorldViz, I was assigned to a project for a customer to build the most amazing virtual reality system ever seen. The customer was a large US company, working on a secret project, and they wouldn't tell me what it was for. They had seen what existed already, and they wanted something even better than that. The customer actually wanted something that didn't exist yet, virtual reality user interfaces are still in their infancy, and they wanted the ability to work naturally with a system and do anything they could think of. They basically wanted the HoloDeck from Star Trek, which will be impossible to build for a long time.

The system had amazing hardware specifications. A huge warehouse sized room, tracked with a Motion Analysis mocap system. Four users would wear motion capture suits, and also wide field of view head mounted displays from Sensics. These are not your regular head mounted displays, they had a field of view of around 120 degrees, which is amazing compared to your average cheap HMD which is in the range of 30 degrees. This is combined with another user holding a 3D wand inside a 4 wall CAVE system from MechDyne. The end result is a system where the four mocap users can see virtual bodies representing themselves, they can pick up virtual tools, and they can perform a maintenance task. The person in the CAVE can fly around and observe the task, seeing everything in immersive stereo, where the walls disappear and everything appears to be around you.

Image of the CAVE with open walls during set up

The system was prototyped at the MechDyne facility in Iowa in 2009. I spent 10 days there performing the integration work, getting the MechDyne and Motion Analysis hardware talking to the WorldViz software. While the hardware is challenging, bringing it all together and creating an immersive and easy to use environment in virtual reality is even more complicated. 3D user interfaces are not very common, and need to be intuitive, and do not use keyboards and mice like traditional desktop machines. All you have is your body and your hands. Everything has to work just like it would in the real world. I worked on some of these research problems in my PhD thesis.

We spent the first few days getting all the pieces from each company integrated together. But I remember there was a moment when everything started to work. There were 4 people calibrated in the mocap suit, and a user in the CAVE. The customer had just arrived to see how the setup was progressing. I remember standing in the CAVE, and watching the virtual mocap users interacting with the 3D CAD model that was in the environment. It felt so amazingly real. It was so immersive. Everything felt like it was really there. You were no longer in a virtual environment, you were experiencing something so real that your body truly thought it was there. It was one of the most beautiful things I had ever seen in my life. I had created this artificial world out of nothing, and I was experiencing it and I felt like it was really there. I remember all these emotions came over me, I cannot explain why ... I was proud of myself for what I had created, but I also knew that I had experienced something that only a small number of people in the existence of humans had ever achieved ... the ability to create my own world and then to experience it, and I believed it was real myself.

At the time I remember the customer was busy inspecting the system, and I whispered to my colleagues that this was the most amazing thing I had ever seen in my life, and I'd seen a lot of cool demos. He told me to keep quiet, because he didn't want the customer to know that even we were surprised that it was as good as we thought it would be. He wanted the customer to think this was normal and just part of the job. Which it was. But it was an amazing experience for me. I had always wanted to build something like this, and today I had achieved that, and it was even more beautiful than I could have expected.

How I envisaged an AR building design system for my PhD proposal in 1999

Virtual reality is still in its infancy, and it is nothing like in the movies yet. However, someone has created a movie, World Builder, which shows what one day might be possible. When I started my PhD in 1999, and when I built this VR system in 2009, I had a dream in my mind of what the ultimate system would look like. The picture above shows a mockup I made with my PhotoShop skills of what I was thinking at the time. The author of the World Builder movie was able to capture this into a movie. And so I encourage you to go and watch it, it conveys a bit about what I was dreaming of back then, and it is also a beautiful movie showing the power of technology.

World Builder from BranitVFX on Vimeo.

Note: Posted in January 2017, but I originally wrote a draft of this in November 2013. With the arrival of Oculus and HTC to the VR market, experiences like I described from 2009 are now becoming possible at home for thousands instead of millions of dollars.