Augmented reality

This article is in need of attention from an expert on the subject. WikiProject Computer science or the Computer science Portal may be able to help recruit one. (November 2008)

Augmented Reality example "Wikitude AR Travel Guide" on a G1 Google Smartphone (Android)

"AR Tower Defense" game on the Nokia N95 Smartphone (Symbian OS) use fiduciary markers

Augmented reality (AR) is a field of computer research which deals with the combination of real-world and computer-generated data (virtual reality), where computer graphics objects are blended into real footage in real time. The term is believed to have been coined in 1990 by Thomas Caudell, an employee of Boeing at the time^[1].

At present, most AR research is concerned with the use of live video imagery which is digitally processed and "augmented" by the addition of computer-generated graphics. Advanced research includes the use of motion-tracking data, fiducial markers recognition using machine vision, and the construction of controlled environments containing any number of sensors and actuators.

History

• 1849: Richard Wagner introduces the idea of immersive experiences using a darkened theatre and surrounding the audience in imagery and sound.
• 1938: Konrad Zuse invents the first digital computer known as the Z1.
• 1948: Norbert Wiener creates the science of cybernetics: transmitting messages between man and machine.
• 1962: Morton Heilig, a cinematographer, creates a motorcycle simulator called Sensorama with visuals, sound, vibration, and smell.
• 1966: Ivan Sutherland invents the head-mounted display suggesting it was a window into a virtual world.
• 1975: Myron Krueger creates Videoplace that allows users to interact with virtual objects for the first time.
• 1989: Jaron Lanier coins the phrase Virtual Reality and creates the first commercial business around virtual worlds.
• 1992: Tom Caudell coins the phrase Augmented Reality while at Boeing helping workers assemble cables into aircraft.
• 2003: GeoVector along with partners Vodafone, HP, Microsoft, Virtual Spectator and Animation Research Ltd showcase Actual Spectator AR app at America's Cup Sailing Races in Auckland, New Zealand.
• 2007: Sony released the Eye of Judgment for PlayStation 3 in Japan on October 25, 2007 and in America on April 24, 2008. Wikitude AR Travel Guide launches on Oct. 20, 2008 with the G1 Android phone and was downloaded about 50,000 in the first year of release.
• 2009: Sony scheduled to release the EyePet game for use with the PlayStation 3 camera.
• 2009: AR Toolkit is ported to Adobe Flash (FLARToolkit) by Saqoosha, bringing augmented reality to the personal computer.

Notable researchers

• Steven Feiner is the leading pioneer of augmented reality, and author of the first paper on the subject.
• Bruce H. Thomas is the current Director of the Wearable Computer Laboratory at the University of South Australia. He is currently a NICTA fellow, CTO A-Rage Pty Ltd, Member of HxI team, and visiting Scholar with the Human Interaction Technology Laboratory, University of Washington. He is the inventor of the first outdoor augmented reality game ARQuake. His current research interests include: wearable computers, user interfaces, augmented reality, virtual reality, computer supported cooperative work (CSCW), and tabletop display interfaces.
• Wayne Piekarski is the inventor of the Tinmith System.
• Oliver Bimber and Ramesh Raskar are the leading researchers in the field of spatial augmented reality (SAR)
• GeoVector Corporation began conducting research into augmented reality in early 1990s. GeoVector’s first patent in this domain, since issued as 5,815,411 [1], was filed September 10, 1993.
• AR+RFID Lab is another company coming up with AR teleconferencing and distance learning ^[2]

Main Computer Vision topics of Augmented Reality

• 3D reconstruction
• Bundle adjustment
• Exponential map
• Fiduciary markers
• Image registration
• Structure from motion
• Video tracking

Examples

Commonly known examples of AR are the yellow "first down" line seen in television broadcasts of American football games, and the colored trail showing location and direction of the puck in TV broadcasts of hockey games. The real-world elements are the football field and players, and the virtual element is the yellow line, which is drawn over the image by computers in real time. Similarly, rugby fields and cricket pitches are branded by their sponsors using Augmented Reality; giant logos are inserted onto the fields when viewed on television.

Another type of AR application uses projectors and screens to insert objects into the real environment, enhancing museum exhibitions for example. The difference to a simple TV screen for example, is that these objects are related to the environment of the screen or display, and that they often are interactive as well.

Many first-person shooter video games simulate the viewpoint of someone using AR systems. In these games the AR can be used to give visual directions to a location, mark the direction and distance of another person who is not in line of sight, give information about equipment such as remaining bullets in a gun, and display a myriad of other images based on whatever the game designers intend. This is also called the HUD

In some current applications like in cars or airplanes, this is usually a head-up display integrated into the windshield.

Current applications

This article is in a list format that may be better presented using prose. You can help by converting this article to prose, if appropriate. Editing help is available. (February 2008)

Advertising: Marketers may consider using AR to promote a new product via an interactive, web-based AR application.

Support with complex tasks: Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system. AR can include images of hidden objects, which can be particularly effective for medical diagnostics or surgery. Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound or open NMR devices. A doctor could observe the fetus inside the mother's womb. See also Mixed reality.

Navigation devices: AR can augment the effectiveness of navigation devices for a variety of applications. For example, building navigation can be enhanced for the purpose of maintaining industrial plants. Outdoor navigation can be augmented for military operations or disaster management. Head-up displays or personal display glasses in automobiles can be used to provide navigation hints and traffic information. These types of displays can be useful for airplane pilots, too. Head-up displays are currently used in fighter jets as one of the first AR applications. These include full interactivity, including eye pointing.

Military and emergency services: AR can be applied to military and emergency services as wearable systems to provide information such as instructions, maps, enemy locations, and fire cells.

Prospecting: In the fields of hydrology, ecology, and geology, AR can be used to display an interactive analysis of terrain characteristics. Users could use, and collaboratively modify and analyze, interactive three-dimensional maps.

Architecture: AR can be employed to virtually resurrect destroyed historic buildings as well as simulate planned construction projects.

Sightseeing: Models may be created to include labels or text related to the objects/places visited. With AR, users can rebuild ruins, buildings, or even landscapes as they previously existed. Combined with a wireless network, the amount of data displayed is limitless.

Simulation: Perhaps one of the most well-known applications for AR is for the creation of flight and driving simulators.

Collaboration: AR can help facilitate collaboration among distributed team members via conferences with real and virtual participants. Also see Mixed reality.

Entertainment and education: AR can be used in the fields of entertainment and education to create virtual objects in museums and exhibitions, theme park attractions (such as Cadbury World), and games (such as ARQuake or The Eye of Judgment)). Also see Mixed reality.

Future applications

• Expanding a PC screen into the real environment: program windows and icons appear as virtual devices in real space and are eye or gesture operated, by gazing or pointing. A single personal display (glasses) could concurrently simulate a hundred conventional PC screens or application windows all around a user
• Virtual devices of all kinds, e.g. replacement of traditional screens, control panels, and entirely new applications impossible in "real" hardware, like 3D objects interactively changing their shape and appearance based on the current task or need.
• Enhanced media applications, like pseudo holographic virtual screens, virtual surround cinema, virtual 'holodecks' (allowing computer-generated imagery to interact with live entertainers and audience)
• Virtual conferences in "holodeck" style
• Replacement of cellphone and car navigator screens: eye-dialing, insertion of information directly into the environment, e.g. guiding lines directly on the road, as well as enhancements like "X-ray"-views
• Virtual plants, wallpapers, panoramic views, artwork, decorations, illumination etc., enhancing everyday life. For example, a virtual window could be displayed on a regular wall showing a live feed of a camera placed on the exterior of the building, thus allowing the user to effectually toggle a wall's transparency
• With AR systems getting into mass market, we may see virtual window dressings, posters, traffic signs, Christmas decorations, advertisement towers and more. These may be fully interactive even at a distance, by eye pointing for example.
• Virtual gadgetry becomes possible. Any physical device currently produced to assist in data-oriented tasks (such as the clock, radio, PC, arrival/departure board at an airport, stock ticker, PDA, PMP, informational posters/fliers/billboards, in-car navigation systems, etc. could be replaced by virtual devices that cost nothing to produce aside from the cost of writing the software. Examples might be a virtual wall clock, a to-do list for the day docked by your bed for you to look at first thing in the morning, etc.
• Subscribable group-specific AR feeds. For example, a manager on a construction site could create and dock instructions including diagrams in specific locations on the site. The workers could refer to this feed of AR items as they work. Another example could be patrons at a public event subscribing to a feed of direction and information oriented AR items.

Specific applications

Mobile

• Gizmondo was an all-inclusive, one-platform computing mobile device, offering GPS, phone, contacts, gaming and the list goes on.^[3] The device was released to much acclaim and excitement. The most impressive feature was its use of AR—an emerging technology making its way from clunky headsets to more consumer-friendly portable devices. ^[4] In 2005, Tiger Telematics launched the Gizmondo to much acclaim. Unfortunately, due to poor fund management the company tapped out at 5,000 units in the UK. The potential for AR is emerging on many different platforms, including those above (iPhone, Nokia, Android).
• Characteroke is a portable AR display costume, whereby the head and neck are concealed behind an active flat panel display.
• MARISIL is a media phone user interface based on AR
• Wikitude is an application for the Android Phone and the iPhone which makes Wikipedia a location based service. The actual camera view is mixed with information from Wikipedia. ING offers an app with almost the same interface for the Android G1 phone locating the nearest ATM see Springwise for more info.

Headset

• Canon's MR Aquarium is an example of AR experience that utilizes a headset. Though headsets are bulky and limit users mobility, they offer a much more involved experience.

Desktop Computer

• LifeClipper is a wearable AR system
• BBC's Merlin MagicSymbol is a free download (U.K. only)from BBC's Merlin site giving access to exclusive Merlin content
• CyberCode is a visual tagging system where real-world objects are recognizable by a computer.
• Imgaugasse is a tractor engine's hydraulics block assembly project by VTT research center.

AR visors

• Cybermind VR/AR Head Mounted Displays ^[5]

Popular culture

Music

• Pop Punk band Blink 182 and rap artist Big Boi are using AR technology to broadcast a concert online ^[6]
• Pop group Duran Duran included interactive AR projections into their stage show during their 2000 Pop Trash concert tour.^[7]

Television

• The television series Dennō Coil depicts a near-future where children use AR glasses to enhance their environment with games and virtual pets.
• Ghost in the Shell 2: Innocence gives several examples of augmented reality in use.
• Several mecha television series, including Gundam, Gunbuster, Neon Genesis Evangelion, Voices of a Distant Star and Martian Successor Nadesico, depict 360° augmented reality cockpits that are used to display information.
• In Serial Experiments Lain, The Wired is overlaid onto the real world via electromagnetic radiation relaying information directly to people's brains, causing people to experience both The Wired and the real world.
• In the Star Trek universe, the Jem'Hadar used a sort of augmented display to view the real world and what was outside the ship, integrating with the star ship's main sensors to gain an outside view of the star ship.
• The television series Firefly depicts numerous AR applications, including a real-time medical scanner which allows a doctor to use his hands to manipulate a detailed and labeled projection of a patient's brain.
• In the 1993 ABC miniseries Wild Palms, a Scientology-like organization used holographic projectors to overlay virtual reality images over physical reality.

Literature

• The table top role-playing game, Shadowrun, introduced AR into its game world. Most of the characters in the game use viewing devices to interact with the AR world most of the time.
• Cybergeneration, a table top role-playing game by R. Talsorian, includes "virtuality", an augmented reality created through v-trodes, cheap, widely available devices people wear at their temples.
• The books Halting State by Charles Stross and Rainbows End by Vernor Vinge include augmented reality primarily in the form of virtual overlays over the real world. Halting State mentions Copspace, which is used by cops, and the use by gamers to overlay their characters onto themselves during a gaming convention. Rainbows End mentions outdoor overlays based on popular fictional universes from H. P. Lovecraft and Terry Pratchett among others.
• The term "Geohacking" has been coined by William Gibson in his book Spook Country, where artists use a combination of GPS and 3D graphics technology to embed rendered meshes in real world landscapes.
• In The Risen Empire, by Scott Westerfeld, most - if not all - people have their own "synesthesia". An AR menu unique to the user that is projected in front of them, but they can only see their own synesthesia menus. It is controlled by hand gestures, blink patterns, where the user is looking, clicks of the tongue, etc.

Advertising

At the 2008 LA Auto Show, Nissan unveiled the concept vehicle Cube and presented visitors with a brochure which, when held against a webcam,showed several versions of the vehicle interacting with the brochure. The brochure is also available from the company website http://www.nissanusa.com/cube/.

On 16 Dec 2008, at a Volvo Ocean Race 2008–2009 event, Volvo Car Malaysia demonstrated the use of this same technology with its media partners a 3D Volvo Open 70 racing yacht. This virtual 3D Volvo Open 70 racing yacht can now be built on their teaser website at: http://www.vcc.com.my/oceanrace/.

In January 2009 Toyota used Augmented Reality to provide an interactive demo of the new Toyota iQ. The program was created by Inition using their MagicSymbol system and can be downloaded from Toyota's website : http://www.toyota.co.uk/cgi-bin/toyota/bv/frame_start.jsp?id=iQ_reality

Conferences

• 1st International Workshop on Augmented Reality (IWAR'98), San Francisco, Nov. 1998.
• 2nd International Workshop on Augmented Reality (IWAR'99), San Francisco, Oct. 1999.
• 1st International Symposium on Mixed Reality (ISMR'99), Yokohama, Japan, March 1999.
• 2nd International Symposium on Mixed Reality (ISMR'01), Yokohama, Japan, March 2001.
• 1st International Symposium on Augmented Reality (ISAR 2000), Munich, Oct. 2000.
• 2nd International Symposium on Augmented Reality (ISAR 2001), New York, Oct. 2001.
• 1st International Symposium on Mixed and Augmented Reality (ISMAR 2002), Darmstadt, Oct. 2002.
• 2nd International Symposium on Mixed and Augmented Reality (ISMAR 2003), Tokyo, Oct. 2003.
• 3rd International Symposium on Mixed and Augmented Reality (ISMAR 2004), Arlington, VA, Nov. 2004.
• 4th International Symposium on Mixed and Augmented Reality (ISMAR 2005), Vienna, Oct. 2005.
• 5th International Symposium on Mixed and Augmented Reality (ISMAR 2006) Santa Barbara, Oct. 2006.
• 6th International Symposium on Mixed and Augmented Reality (ISMAR 2007) Nara, Japan, Nov. 2007.
• 7th International Symposium on Mixed and Augmented Reality (ISMAR 2008) Cambridge, United Kingdom, Sep. 2008.

See also

• Alternate reality game
• Augmented browsing
• Augmented virtuality
• Camera resectioning
• Cyborg
• Mixed reality
• Simulated Reality
• Virtual retinal display
• Virtuality Continuum
• Virtual reality

References

External links

• Star Trek Augmented Reality
• Blog and Projects on Augmented Reality, Germany
• Augmented Environments Lab, GVU Center, Georgia Institute of Technology
• Wearable Computer Lab, South Australia
• HITLab, Seattle
• HITLab NZ, Christchurch New Zealand
• TU Munich
• Studierstube, Graz University of Technology, Vienna
• Columbia University Computer Graphics and User Interfaces Lab
• Projet Lagadic IRISA-INRIA Rennes
• HowStuffWorks: How Augmented Reality Will Work
• Resources Page: Jim Vallinos AR
• Networked Augmented Reality (AR) Future: Matt Groves
• Mixed Reality: Augmented Reality, Augmented Virtuality, Virtual Reality. Kolsouzoglou Anthony's Research Site
• Game Daily - Mobile Augmented Reality
• DART: The Designers Augmented Reality Toolkit
• Nissan Cube Website with Video Exploring the Use of the Technology and Mailing List for Receiving Augmented Reality Brochure
• Volvo Ocean Race 2008–2009 teaser website (by Volvo Malaysia) showcasing a 3D Volvo Open 70 yacht.
• Augmented Reality and RFID art projects - AR+RFID Lab, Royal Academy of Fine Arts, Music and Dance, The Hague, Netherlands
• Mobile Augmented Reality games projects and research
• GE Smart Grid Augmented Reality