As AR slowly becomes part of our everyday lives through an impressive number of mobile applications including the much-awaited Google Augmented Reality Glasses, the benefits of this concept can be extended beyond the functional and sometimes gimmicky uses, in the movie entertainment industry.
Augmented Reality Movies So Far
Augmented reality technologies have been portrayed in films for decades, some of the most popular being the Terminator series, Minority Report, Avatar and Tron, but these are not the kind of augmented reality movies we are referring in this article.
What we mean by “augmented reality movie” is the film that allows the viewer to experience and interact with the movie (i.e. set, action, characters) in real-time, altering their perceptions, sensations and conceptions about reality.
Such an immersive movie experience has been attempted in several mobile apps in the last few years with increasing successful results but still far from perfection.
NFF AR Movie Experience. Developed for the Netherlands Film Festival in 2011, this AR mobile experience uses the Layar app to create a “3D surround film set of a ‘movie’ happening in augmented reality” in the city centre of Utrecht, according to the project’s website. This set is filled with ‘speech bubbles’ floating in thin air above certain locations, containing the dialogue from the movie scenes filmed there.
The app’s use of augmented reality is rather limited to these digital dialogue boxes; however it allows users to follow different storylines throughout the city while taking photos of their friends or passers-by with the bubbles above their heads. In this way, the unsuspecting actors become part of the story which could also be used to create a ‘narrative navigation’ by guiding people from one venue to the other during the film festival.
Augmented Reality Cinema. A 2011 concept video showing what this app will be able to do is all that exists, nevertheless it provides a few interesting scenarios and clues about how an augmented reality movie may become reality (the pun is intentional). With almost no direct link to AR, except around 0:57 when a real person ‘enters’ the movie scene, all other scenes have been added in post-production and have nothing to do with the real-time characteristic of AR.
The video gives some ideas about how an augmented reality movie can be split into scenes, depending on the location where they were filmed. These scenes ‘activate’ on the viewers mobile device only when the camera is pointing in the right direction within that location.
Another interesting feature is the participation (even indirectly) of a person within a movie scene (around 0:57). Although the viewer is not directly involved in the movie, by seeing a real person breaking the boundary between their reality and the film’s reality, everything feels more immersive and the entire user experience is significantly improved.
The Witness. Claimed to be the first movie in the outernet, the Witness was created for the launch of the 13th Universal Movie Studio inGermany using Junaio for their augmented reality application. It is one of the closest to our definition of augmented reality movie through its interactivity, cinematic experience and viewers’ apparent contribution.
Nonetheless, the film provides only a linear narrative (despite allowing the user to make choices that will influence the film’s ending) that will have the same number of limited predefined scene sequences and endings, no matter what the viewer is doing during their treasure hunt/ detective work.
The above three examples give only a short overview of the breakthroughs and technological limitations that demonstrate that augmented reality cinematography is not as simple and straightforward as some other AR applications. If many issues will most likely be solved in the near future, there are still some core features of augmented reality and cinematography which are difficult to tackle in an AR movie.
Interaction in AR Movies
Augmented reality refers not only to a merger of digital data and real information but also to an interaction between both. One of the core indicators of any AR application’s success is its degree of interactivity. The same thing applies to an augmented reality movie. If the viewer cannot interact with any element of the film, their experience is almost equal to seeing it on a normal TV, which makes the entire AR concept obsolete.
This leads to another extremely difficult problem: how can a viewer truly interact (without any pre-set scenarios) with all elements of a movie? Even in the most advanced and massively interactive video games there are still a limited number of actions a character can take in a particular situation. The answer to the above question is that they cannot, at least for the time-being. For this to be possible, some sort of artificial intelligence would be needed that can create an infinite number of scenarios based on a user’s interactions which are then ‘served’ in real-time.
However, interactive elements can be created to respond to a limited set of actions. For example, a speeding virtual car may react to a person’s position in the real world by avoiding them. Having multiple interactive elements in one movie scene could potentially create an immersive AR experience for the viewer.
As any motion picture, the AR movie has to be filmed before a viewer can experience it. This automatically voids the real-time quality of augmented reality, or does it? Although a film scene cannot be experienced in real-time (especially if it involves real actors), the film could potentially trick our mind by reacting to the surrounding environment and creating as many real-time digital elements as possible.
For example, if a person is on a street during a rainy evening watching through their Google glasses a scene from their favourite AR movie, most elements of the movie could potentially react to this situation and the viewer could experience passing cars splashing water, some umbrellas being blown by the wind and even hear and see thunders and lightning. The same scene could look and be experienced completely different if the weather was sunny and nice, even if the film action or dialogue between the characters was the same.
As shown by all three examples, the viewer has to be in a certain spot within a location in order to ‘activate’ and experience the AR movie. Even within the same spot, it is very difficult to keep the smartphone camera at the same angle and height as the original camera. Consequently, the current AR movie apps only trigger the film scene when the smartphone’s camera is pointing at the right element without being influenced by the camera movement. This makes everything seem less real, less accurate and less immersive.
A genuine AR movie should allow the viewer to watch the same scene from different angles, positions and heights, transforming everything into a 360o 3D film experience that happens around them. Unfortunately this technology is only possible at a very small scale within a video production studio and it is currently being used for on set live previews. This technique uses a series of markers strategically positioned in a film studio plus a lot of computer power and people to add the digital information (e.g. scenery, spaceships) while filming live actors on a green background. There is a big question mark on how all of these can be replicated outside the studio in order to produce an AR film experience.
Beyond the technological incapacity of creating an augmented reality movie, producers are also limited by a lack of understanding of how an AR film should look like and what its final goal should be. Understanding its elements and the augmented reality concept is not enough to create a final product. A great amount of research, creativity and technological capabilities are needed to create a film that will take the viewer out of their home comfort and immerse them in the augmented cinematic world. What other limitations stop filmmakers from creating augmented reality movies? When do you think we will be able to experience an AR film? Why would you be more interested in experiencing an AR movie than in watching a movie in a cinema (if this is the case)? Please leave your comments in the section below.