You can analyse a tennis ball’s flight, recognise strangers and play with a stegosaurus. Charles Arthur reports on augmented reality, coming soon to a smartphone near you
As the players pause between ends in a match at Wimbledon, the TV screen suddenly overlays the court with a pattern of yellow and black dots – showing where the receiver has been returning the first and second serves. As they walk back out, the overlay vanishes and they’re back to play.
As another wicket falls in the Ashes, a replay shows the flight of the ball, and how it was going to clip the off-stump before it was stopped by the batsman’s leg. And in the US, TV viewers watching an American football match see a yellow line running across the field – the “first down line” that the attacking team must reach to retain possession. Except that it’s invisible to the players on the field: it’s added in the TV studios. And this weekend’s Open Championship golf will show the greens overlaid with contours, revealing the territory each putt must negotiate.
All are examples – already so familiar as to feel quotidian – of “augmented reality” (AR), a burgeoning field that mixes computer power with real life to add extra information to a scene or event. The sports examples are only the beginning, relying as they do on static locations. The next generation of augmented reality is designed for people on the move – and it’s already being implemented.
For example, spectators visiting the All-England club this year with an Android-powered phone could download an AR application called Wimbledon Seer, which, when they held the phone up and pointed it at the courts, would display match data, where the refreshment stands were, or whether a cafe had an exceptionally long line.
Unlike virtual reality, or immersive reality (think Second Life), AR takes what is already there in the real world and uses computer sensing to add more information – whether in touch (“haptic”), visual or aural formats.
Pilot scheme
It has already been used in niche applications by well-funded organisations: Boeing, for example, uses AR so that engineers can do the complex wiring on its aircraft. Since a trial in 1996 – which involved PCs worn on a waistband and special goggles – its engineers have seen the wiring diagram overlaid on the place where they are looking, so they don’t have to keep referring back to paper wiring diagrams (where it would be easy to lose your place).
But even that’s not the original form of AR – which was arguably the tapes that you could buy or borrow at museums: slot them into your cassette player (which shows how old the idea is) and as you walked to each exhibit, the tape would provide a more detailed explanation of what you were seeing. It may be the first time art has fostered a technology breakthrough. Nowadays, AR is used in museums in a more dramatic manner – such as Canon, which has a version for showing off dinosaur exhibits: viewed through a special camera, a three-dimensional stegosaurus appears to be right in front of the visitor.
Jen-Hsun Huang, president and chief executive of the graphics card maker Nvidia, is certain that augmented reality is going to become part of our daily life – and soon. “You’ll see it in your car when you’re backing up: you’ll get a readout showing how close objects are. Golfers will be able to put on glasses and it will show them the contours of the green. Already Sony has been using it in videogames – you put a camera over a board, and you put the game cards on the board, and you see a rendering of the monsters from the cards on your computer – and they’re fighting each other. Lego has done one where the model appears to pop out of the box.” He’s sure that the growth in processing power and location-sensing will mean AR will become commonplace in a few years.
Great leap forward
Certainly, AR out in the field needs smartphones with a number of elements built in. First, video or camera input in a high enough resolution; location sensing; direction sensing; and then, the onboard computing power to analyse the visual information and decide what and where to overlay. It’s only in the past year or so that smartphones with all those elements have begun to be affordable, and include elements such as the compass built in to the recently released iPhone 3GS and Android-based G1 mobile.
And there are already a number of startup companies trying to make the most out of this burgeoning area. One is Layar, from SPRXmobile, a Dutch company: it overlays local restaurant, hotel and property data on to the scene that it “sees” through the camera.
Meanwhile a Swedish company, The Astonishing Tribe, has gone a step further, with a facial recognition system called Augmented ID. It tells you who people are, based on identifying their picture via a technology called Polar Rose, which analyses faces and then searches for photos on Flickr that match it – and pulls out the name from the tags.
Another, called Nearest Tube, for the iPhone 3GS, uses its GPS and video capability to give real-time directions – overlaid on to the scene, viewed through the iPhone – to the nearest underground station. An Austrian-based company, Mobilizy, has developed an Android application that, given a camera view and a location, overlays information about it from Wikipedia and photos from Panoramio. So far there are 800,000 points around the world where it works.
Use your imagination
And it turns out that the programming isn’t the hardest part. Chetan Damani, a director of Acrossair, which developed the Nearest Tube application, says: “The app itself wasn’t that complex. Apple released the 3.0 SDK a few months prior in beta version, so we started conceptualising the app in advance. I would say in total we spent 20-30 man days on the project (design, strategy and build).”
He adds that AR isn’t a homogenous field. “Firstly, you have AR, which involves overlaying data in to the current surroundings, like our Nearest Tube application on the iPhone. For that you need the geodata (longitude and latitude) and a capable device. The second type of AR is when you create a virtual object and layer that on to a real-world view, like the BMW Z4 AR app accessed on a PC. For this you need a visual tag the camera can recognise to create the virtual object; in this type of AR app you do not need the geodata.”
It is still early days, though. “It’s really picking up now because of the devices. AR provides a much more intuitive interface to viewing mapping data, and the one thing that the internet era has taught us is that the interface drives interest. The hardest element about AR is getting hold of accurate data – you need to have detailed longitude and latitude data, and although this is available for landmarks and for certain stores, it’s not easy to obtain.”
But with GPS getting ever more precise, and mobile phones getting ever more accurate, it may not be that long before the spectators at sports events are lifting their phones – or perhaps even special glasses – to their eyes to “watch” the event in more detail, and in ways we presently have to stay at home for. Augmenting reality could make experiencing reality much more rewarding.
