The State of Augmented Reality

by Peter Altamirano, Jam3

Advancements in augmented reality have been steadily turning heads over the past few years. What started as a niche and futuristic technology has hit the mainstream to wide appeal, and creative approaches in deploying it have led to many magical implementations of AR in recent times.

3 Jam3 Illustration [Converted]-01.png

Now, it’s a rapidly swelling market – and while the rate of growth in the segment is certainly noteworthy, what’s even more exciting is that we’re just at the beginning of the journey. There is so much exciting innovation on the horizon.

To appreciate how far augmented reality has come, let’s take a look at its history. The earliest origins of AR date back to 1862, when scientist and inventor John Henry Pepper presented an awe-inspiring illusion during a performance of Charles Dickens’ The Haunted Man. The effect, which became known as ‘Pepper’s Ghost’, used a combination of lighting and refraction to project an apparition on to the stage, seemingly out of thin air. Though Pepper’s Ghost preceded digital technology by over a century, the same basic technique is still used today for creating AR experiences such as Tupac’s appearance at Coachella festival in 2012. 

The first example of AR as we know it today – augmenting a physical environment with digital objects or overlays – appeared in 1992 when Louis B. Rosenberg, a researcher at the US Air Force Armstrong Labs, built an experimental headset which provided virtual visual assistance for a physical task. Rosenberg’s research proved proof of the conceptual viability of AR technology, and acted as a predecessor to commercial AR headsets that are on the market today, such as the Microsoft HoloLens or the Magic Leap One. 

Headsets such as these are one of two distinct ways that we can experience augmented reality. There are many different headsets on the market currently, and as usual the big players want in: Facebook has confirmed that they’re building AR glasses, and Apple is also rumored to have something in the works. But while it’s true that we’ll begin to see ever more compact and ergonomic AR headsets for commercial use, major advancements are likely still five or more years away. On the bright side, nothing stimulates innovation like a competitive market, which could lead to incredible developments such as liquid crystal-based AR contact lenses. 

The other way that we can experience augmented reality is through screen devices like smartphones and tablets. This branch of AR is generally more accessible due to the ubiquity of screen devices, and because many people are already familiar with how it works due to the commercial successes of things like Pokémon Go or face filters on social media platforms. 

There’s a core distinction to be made between AR as experienced through a screen as opposed to a headset. When using a mobile device or screen, which mediate AR through a camera feed, their entire experience of the intersection between digital and real-world elements occurs within the enclosed frame of the screen. On the other hand, when wearing a headset, a user’s field of view is unchanged from their usual POV, allowing the addition of virtual content to provide a far more immersive and authentic AR experience. 

As with the output, the process of creating AR experiences is also a little different between headset and mobile. Building for headset usually means making an app using a game engine such as Unity or Unreal combined with a supported plugin released by manufacturers, or using the headsets native SDK. For mobile application, both Google and Apple have released their AR SDKs: ARCore for Android and ARKit for iOS. Both include support for game engines and the utility to make a fully functional AR experience, but if desired more specialized features can be added using Vuforia, or other AR cloud start-ups.

While these methods are standard for native AR – experiences that require an app download to support them – things aren’t quite the same for website applications. Web AR experiences are generally coded using JavaScript alongside WebXR API and AR libraries such as AR.js, 8thWall TensorflowJS. The benefit of building a website AR experience over a native one is that it makes it more accessible to consumers by bypassing the requirement of downloading an app to try it. The trade-off is that browser-based AR doesn’t offer the same level of performance nor the number of features that can be applied to native AR. 

Despite the minor drawbacks, web AR has taken a huge leap forward in recent times, and now both formats are viable in their specific and distinct uses. Longer storytelling experiences such as East of the Rockies suit native AR better because of its depth and functionality, whereas simpler, low-poly experiences that only require plane- or image-tracking like Portal Hunt or this Spiderman game are perfect for web AR. 

On top of there being different ways to build and interact with AR, there are also different types of experience that can be created. One of these is known as world tracking. This category uses a ‘marker’ or logo present in 3D space to act as a kind of grounding point for the experience, where digital content is then placed in relation to it. While this is standard for many AR creations, advancements technological advancements have begun to allow for a marker-less approach, or what is known as SLAM (Simultaneous Localization And Mapping). A SLAM build doesn’t require a marker since it detects nearby surfaces in a physical environment and uses them as reference points for the digital augmentation. 

Another common format for AR experiences is human body tracking. The most widely used example of this is face tracking, because it provides remarkable accuracy and has a huge variety of applications. Face swapping, makeup application, and expression detection are just some of the near endless opportunities for face tracking AR – but while face tracking is pretty advanced, other human body parts (besides some breakthroughs in hand tracking) remain much harder to track.

Even though the technology has yet to catch up with our imaginations, there are big things in store for AR as innovation increases at an ever-accelerating rate. Exciting leaps forward, such as AR cloud, now stand to radically change how we interact with AR. From multiplayer, to perfect localization and occlusion capabilities, AR Cloud brings us a step closer to true mixed reality and the ability to create much more immersive AR experiences. 

No less impressive are the hardware upgrades also in development. Oculus’ Chief Scientist, Michael Abrash’s predictions for AR and VR are well worth watching. Fascinating progress is also being made using audio instead of image for AR experiences, in particular with the latest AR audio sunglasses from Bose. The next few years will be full of incredible milestones as we witness AR redefine the user shopping experience, change our conceptions of entertainment and transport, and introduce ground-breaking applications to the education and medicine sectors.

So, while it’s clear that AR has come a long way since Pepper’s Ghost, the story is really only just beginning. Some of the best creative and technical minds on the planet are working to bring into reality experiences that past generations could never have even dreamed of, experiences that improve and enhance human capability down to the very way we interact with the reality around us. As Arthur C. Clarke once said: “any sufficiently advanced technology is indistinguishable from magic” – if that’s true, it looks like we’re well on the way to becoming professional magicians.

Peter Altamirano, Jam3

Peter Altamirano, Jam3

About the Author: Peter Altamirano is Technical Director at Jam3. He guides tech projects and teams, from frontend and backend to AR and installations. He loves exploring new technologies, bringing new ideas, prototyping, improving production processes and development. He is a creative technologist discovering new possibilities and making things happen. He has previously worked with brands like Google, Facebook, Adidas, eBay, Disney, and IBM.