8
"The reward for work well done is the opportunity to do more work"
Jonas Salk, MD
This dissertation makes a number of significant and original contributions in the area of interactive 3D modelling for outdoor augmented reality environments. The contributions described in this dissertation can be summarised as the development of augmented reality working planes, construction at a distance techniques, user interface technology, and the software and hardware needed for implementation. These contributions have been demonstrated with real world examples and provide the capability to perform modelling of 3D geometry in outdoor environments. With these contributions there are a wide range of future possibilities for new research and applications that may be used in scientific and commercial settings.
Seven unique contributions to the area of computer science are described in the following sections. These contributions form a solution to the problems associated with interactive 3D modelling in outdoor augmented reality worlds. The introduction to this dissertation contained a set of six research questions and three research goals. The seven contributions answer each of these questions, providing a practical and demonstrated solution for each one. Each of the goals outlined were used as principles during the development of these contributions and guided their development.
The ability of a human to perceive depth information attenuates rapidly as the distance increases. Most existing techniques for virtual environments are designed to operate within arm’s reach and so different techniques are required for interacting with objects at large distances. Outdoor objects such as buildings are very large compared to the size of a human and are beyond their depth perception, even if a part of the building is within arm’s reach. By thinking of distance as sideways motion from a different view point, it becomes possible to avoid problems with inaccurate depth perception. The augmented reality working planes technique is developed based on existing concepts used in CAD modelling systems. By creating an AR working plane at a distance from the user, vertices may be projected against this surface from any distance with no reliance on the depth perception of the user. AR working planes may also be defined and used to manipulate existing objects, providing a complete set of modelling interactions. AR working planes have an accuracy which attenuates at a constant rate that is better than human depth perception, and is limited only by the resolution of the pixels on the HMD and the tracking devices in use. The concept of AR working planes encompasses a number of ways to create the planes, and may be made relative to the various coordinate systems the user is familiar with. By taking advantage of the alignment of landmarks that can accurately position a user, AR working planes may be created that are aligned with physical world objects. The flexibility of AR working planes may be used to perform a wide variety of modelling operations and forms a basis for the next contribution. A further advantage to the use of AR working planes is that only 2D input devices are required, which makes the implementation much easier using currently available mobile hardware.
A naïve way to perform outdoor modelling is to create each vertex by hand and then connect them together to form polygons and objects. This method is time consuming and does not take advantage of the many properties of outdoor objects that can be used to minimise the amount of data entry required. Humans are quite capable of understanding the design of objects around them and this knowledge is used to break down the modelling of a building into a set of simple and much higher-level steps. For example, instead of specifying vertices directly, the user may project walls aligned with the object and the system can calculate the vertices and polygons automatically based on the intersection points. This dissertation presented a collection of techniques named construction at a distance that provides the ability to perform various modelling operations. By iteratively combining these techniques, complex shapes may be formed. Using an AR preview the user can construct the model in steps and compare it against the physical world, iteratively refining the parts that need the most work according to the judgement of the user. The construction at a distance techniques have been demonstrated with examples performing the modelling of many complex outdoor shapes.
There are only a small number of user interfaces for mobile AR and many of these are quite simple and use existing 2D desktop metaphors. With AR focusing on the use of the body as an input device to view information, little research has been performed into performing higher-level interactions such as those required for 3D modelling. A further complication is that tracking technology for mobile computers suffers from problems not normally experienced indoors, and so this must be taken into consideration with any interface design. Command entry is required to select amongst the number of different AR working planes and construction at a distance techniques available. Furthermore, these techniques also require pointer-based interactions to create vertices and perform manipulation operations. This dissertation describes a user interface designed specifically for mobile AR applications, based on the user’s hands. By using the mapping of finger pinches to commands, the user may execute commands without using the cursor, leaving the hands available for pointing-based interactions. Keeping command and pointing actions separate allows both actions to be performed simultaneously if desired. To complete the AR user interface, a wide range of information is presented to the user so they understand the state of the application and the environment around them. By using external VR style views, the user is able to see information not normally visible from the immersive AR view. The design of this user interface was iteratively tested in small informal user studies to find design flaws and then make improvements.
There are a small number of input devices that are suitable for use with mobile computers because of limitations imposed by the outdoor environment. Most 3D tracking technology is not suitable for mobile computing but is also not required since AR working planes only requires a 2D input for a cursor. Existing desktop input devices such as trackballs and joysticks may be difficult to operate since there are numerous levels of indirection applied between the user and the task. Since operations are performed against AR working planes, the direct motion of the hands is used as an input device. Instead of manipulating a cursor indirectly, the user can simply reach out and point directly at the plane to perform operations. This dissertation presents a vision-based tracking system to support the user interface with intuitive hand-based control. Simple metallic sensors are used to detect pinching motions, and are mapped directly to the menu system. Fiducial markers mounted on the thumb are tracked using the HMD camera providing video overlay, and operates in a wide variety of lighting conditions. This hand tracker is a simple and yet very effective input device for mobile outdoor AR, requiring no major extra equipment not already included in the system.
This dissertation presents applications that have been developed using the other previously discussed contributions. The Tinmith-Metro modelling application demonstrates the use of AR working planes and construction at a distance techniques, controlled by the user interface and vision tracked gloves. This application implements all of the examples presented in this dissertation, with the goal being to produce a tool that may be used to create and capture the geometry of outdoor structures. The examples are used to demonstrate the capabilities of the techniques described in this dissertation.
Software toolkits for 3D virtual environments are not sufficiently developed to the point where applications may be developed by simply connecting together various components. In 2D desktop environments, simple toolkits were built and then used to perform research into increasingly higher-level tools. Current 2D toolkits are extensive and do not require the programmer to perform very simple operations since a best practice has been developed for many types of tasks. In contrast, 3D toolkits are much less mature and only simple tasks such as tracker abstraction have reached a best practice. The developer of 3D applications is currently required to implement many of the features that an application needs. Existing software was not adequate for the implementation of the contributions of this dissertation, and so a number of new solutions were developed to current problems. This dissertation contributes to the existing body of work by developing a software architecture with a number of novel features such as an object-oriented design based on data flow, a file system approach for an object store, and a streamlined and high performance implementation. This software architecture forms a basis for the applications developed for this thesis, and made the implementation much simpler than would have been possible by attempting to use a collection of existing tools.
Most of the equipment used for the development of AR systems is based on commercially available components, but these must be modified since the components are not normally designed for mobile and outdoor requirements. In other cases, components that can not be purchased must be designed and constructed. Since the desired task is mobile outdoor AR, all the equipment required must be worn by the user. Integrating a number of components into a system that can be worn outdoors is non-trivial, and must take into account restrictions such as weight, size, power consumption, mounting, and capabilities. This dissertation contributes a design for a mobile backpack computer that uses a flexible internal architecture to carry a number of devices while also being easy to change. Since hardware is constantly evolving, the ability to accommodate these changes is important when performing research with mobile AR systems.
With the availability of 3D modelling techniques for use in mobile AR environments, a wide range of possibilities for future work are available. I would initially like to explore possible applications where the 3D modelling techniques can be used to enhance existing work. As mentioned in Chapter 2, there are a number of techniques currently used to capture the physical world but they have some limitations, and my contributions address some of these. It would be interesting to explore how the modelling techniques may be integrated with existing capture techniques to provide real-time feedback using AR and allow interactive changes. Chapter 4 discussed how the modelling techniques may be used to create geometry for objects that do not exist yet. This capability would be useful for landscape designers and architects to plan the construction of new developments while on site. When I have discussed my research with various people, I have received much interest from people in areas such as architecture, construction, forestry, gaming, surveying, and visual arts who are interested in trying the technology for their particular applications.
There are a number of areas I would like to perform further research in to improve the contributions made in this dissertation. Hardware is continuously improving and this increases the features that can be implemented. It would be interesting to combine the construction at a distance techniques with existing image-based reconstruction algorithms, so that the system can automate simple tasks while still giving the user control over the process. As discussed previously, the use of speech recognition technology would be a very useful addition to the user interface in Chapter 5. Using the distributed software architecture described in Chapter 6, applications can be developed that support users on multiple systems (both indoors and outdoors) to collaborate on various modelling tasks. It would be interesting to explore collaboration tasks further to find out the types of interactions that are useful when working in these environments. With the availability of a reference platform, the development of user studies to test future improvements is now possible, and verifying which techniques perform the best in this environment would be useful – the current design is based on knowledge from the VR area that may not fully apply. For the technology to progress towards mainstream use, it will require all the components to be better integrated and miniaturised. I would like to explore the use of smaller computers such as hand helds and tablet PCs as replacements for the laptop, with the goal being to reduce the current backpack to something more manageable and that may one day fit into the user’s pocket.
When this dissertation was written, the goal was to create the necessary user interfaces and techniques required to perform 3D modelling. While this goal has been achieved, the modelling application now needs to be used to solve real world problems in order to be useful. To allow the ideas in this thesis to mature into applications that can be used in the real world, I have formulated a scenario which is both useful and can be used to thoroughly test out future developments. The scenario is of a disaster recovery system, to be used by rescue workers when areas of cities have been destroyed by an event such as an earthquake. A current problem with these cases is that workers may not know what the landscape looked like previously, and therefore they may not know where to dig and search. Using previously captured models overlaid onto the landscape, the previous environment is made obvious to the wearer. The AR equipped rescuers can then direct other traditional rescuers to places of interest, even in conditions of dust or darkness. Another important problem with rescue efforts is coordination amongst workers. Trying to communicate spatial information about the environment to another person on a radio is quite difficult, especially when there is confusion and panic. Using AR, the user is able to mark up the environment around them and perform tasks such as highlighting damaged walls and unsafe areas, modifying building models in 3D to reflect changes, and adding virtual post-it notes and warning signs. These changes can be shared with other users over a wireless network, and also communicated back to the operations managers at headquarters to increase their situational awareness. These operations managers are responsible for controlling the entire effort, and giving them the ability to see information gathered in real-time would help them better understand the situation and use resources effectively. Apart from just gathering information though, the operations managers also have the ability to modify the information and add markers to the environment to guide the mobile users. This could be used to indicate places to visit or avoid based on the combined knowledge of all the participating users. For this scenario, I envisage the various participants will be using numerous types of technology depending on what is appropriate for the task. For example, outdoor users may wear full AR systems or carry hand held computers. Indoor users may use desktop monitors, large wall displays, or projector tables with tangible interfaces. I think that making all of these different technologies work together to solve a difficult problem is the next big step, with many benefits to society as a result.
Research in the area of outdoor AR environments is still in its infancy. This dissertation has provided a number of contributions to the area but there is still much research to be performed before AR applications will become mainstream. This is an exciting time to be working in the field, with many currently untouched problems that remain to be solved and so many exciting possible applications. I hope that in the near future it may be possible to realise a mobile AR system that can be used both indoors and outdoors, fits into the pocket, works all the time without reliability problems, and provides a real sense of presence for the virtual objects overlaid onto the physical world.