Augmented Reality Movies and Their Challenging Future

The Witness - AR MovieWhat do you do when watching passively a film on a screen does no longer excite your cinephile senses despite the latest 3D HD video and sound technologies? You experience  an augmented reality movie.

As AR slowly becomes part of our everyday lives through an impressive number of mobile applications including the much-awaited Google Augmented Reality Glasses, the benefits of this concept can be extended beyond the functional and sometimes gimmicky uses, in the movie entertainment industry.

Augmented Reality Movies So Far

Avatar Augmented Reality Display

Avatar AR Display

Augmented reality technologies have been portrayed in films for decades, some of the most popular being the Terminator series, Minority Report, Avatar and Tron, but these are not the kind of augmented reality movies we are referring in this article.

What we mean by “augmented reality movie” is the film that allows the viewer to experience and interact with the movie (i.e. set, action, characters) in real-time, altering their perceptions, sensations and conceptions about reality. 

Such an immersive movie experience has been attempted in several mobile apps in the last few years with increasing successful results but still far from perfection.

NFF AR Movie Experience. Developed for the Netherlands Film Festival in 2011, this AR mobile experience uses the Layar app to create a “3D surround film set of a ‘movie’ happening in augmented reality” in the city centre of Utrecht, according to the project’s website. This set is filled with ‘speech bubbles’ floating in thin air above certain locations, containing the dialogue from the movie scenes filmed there.

The app’s use of augmented reality is rather limited to these digital dialogue boxes; however it allows users to follow different storylines throughout the city while taking photos of their friends or passers-by with the bubbles above their heads. In this way, the unsuspecting actors become part of the story which could also be used to create a ‘narrative navigation’ by guiding people from one venue to the other during the film festival.

Augmented Reality Cinema. A 2011 concept video showing what this app will be able to do is all that exists, nevertheless it provides a few interesting scenarios and clues about how an augmented reality movie may become reality (the pun is intentional). With almost no direct link to AR, except around 0:57 when a real person ‘enters’ the movie scene, all other scenes have been added in post-production and have nothing to do with the real-time characteristic of AR.

The video gives some ideas about how an augmented reality movie can be split into scenes, depending on the location where they were filmed. These scenes ‘activate’ on the viewers mobile device only when the camera is pointing in the right direction within that location.

Another interesting feature is the participation (even indirectly) of a person within a movie scene (around 0:57). Although the viewer is not directly involved in the movie, by seeing a real person breaking the boundary between their reality and the film’s reality, everything feels more immersive and the entire user experience is significantly improved.

The Witness. Claimed to be the first movie in the outernet, the Witness was created for the launch of the 13th Universal Movie Studio inGermany using Junaio for their augmented reality application. It is one of the closest to our definition of augmented reality movie through its interactivity, cinematic experience and viewers’ apparent contribution.

Nonetheless, the film provides only a linear narrative (despite allowing the user to make choices that will influence the film’s ending) that will have the same number of limited predefined scene sequences and endings, no matter what the viewer is doing during their treasure hunt/ detective work.

The above three examples give only a short overview of the breakthroughs and technological limitations that demonstrate that augmented reality cinematography is not as simple and straightforward as some other AR applications. If many issues will most likely be solved in the near future, there are still some core features of augmented reality and cinematography which are difficult to tackle in an AR movie.

Interaction in AR Movies

Zugara Canon Ballz

Interactive AR Game from Zugara

Augmented reality refers not only to a merger of digital data and real information but also to an interaction between both. One of the core indicators of any AR application’s success is its degree of interactivity. The same thing applies to an augmented reality movie. If the viewer cannot interact with any element of the film, their experience is almost equal to seeing it on a normal TV, which makes the entire AR concept obsolete.

This leads to another extremely difficult problem: how can a viewer truly interact (without any pre-set scenarios) with all elements of a movie? Even in the most advanced and massively interactive video games there are still a limited number of actions a character can take in a particular situation. The answer to the above question is that they cannot, at least for the time-being. For this to be possible, some sort of artificial intelligence would be needed that can create an infinite number of scenarios based on a user’s interactions which are then ‘served’ in real-time.

However, interactive elements can be created to respond to a limited set of actions. For example, a speeding virtual car may react to a person’s position in the real world by avoiding them. Having multiple interactive elements in one movie scene could potentially create an immersive AR experience for the viewer.

Real-time Feeling

Lynx Fallen Angel AR Experience

Lynx ‘Real-Time’ AR Experience

As any motion picture, the AR movie has to be filmed before a viewer can experience it. This automatically voids the real-time quality of augmented reality, or does it? Although a film scene cannot be experienced in real-time (especially if it involves real actors), the film could potentially trick our mind by reacting to the surrounding environment and creating as many real-time digital elements as possible.

For example, if a person is on a street during a rainy evening watching through their Google glasses a scene from their favourite AR movie, most elements of the movie could potentially react to this situation and the viewer could experience passing cars splashing water, some umbrellas being blown by the wind and even hear and see thunders and lightning. The same scene could look and be experienced completely different if the weather was sunny and nice, even if the film action or dialogue between the characters was the same.

Viewing Angle

As shown by all three examples, the viewer has to be in a certain spot within a location in order to ‘activate’ and experience the AR movie. Even within the same spot, it is very difficult to keep the smartphone camera at the same angle and height as the original camera. Consequently, the current AR movie apps only trigger the film scene when the smartphone’s camera is pointing at the right element without being influenced by the camera movement. This makes everything seem less real, less accurate and less immersive.

Previzion AR system

Previzion AR System for On Set Live Preview

A genuine AR movie should allow the viewer to watch the same scene from different angles, positions and heights, transforming everything into a 360o 3D film experience that happens around them. Unfortunately this technology is only possible at a very small scale within a video production studio and it is currently being used for on set live previews. This technique uses a series of markers strategically positioned in a film studio plus a lot of computer power and people to add the digital information (e.g. scenery, spaceships) while filming live actors on a green background. There is a big question mark on how all of these can be replicated outside the studio in order to produce an AR film experience.

Beyond the technological incapacity of creating an augmented reality movie, producers are also limited by a lack of understanding of how an AR film should look like and what its final goal should be. Understanding its elements and the augmented reality concept is not enough to create a final product. A great amount of research, creativity and technological capabilities are needed to create a film that will take the viewer out of their home comfort and immerse them in the augmented cinematic world. What other limitations stop filmmakers from creating augmented reality movies? When do you think we will be able to experience an AR film? Why would you be more interested in experiencing an AR movie than in watching a movie in a cinema (if this is the case)? Please leave your comments in the section below.

 

7 thoughts on “Augmented Reality Movies and Their Challenging Future

    • Hi Ashboo, you should be able to find some more information by looking at the links from the above post.

      However I am not very sure what you are looking for. My post is about how movies can be experienced by the viewer as a complex and interactive AR experience. If this is what you are looking for I am afraid there isn’t much out there but a Google search should give you a fair amount of results.

      If you are interested in how augmented reality is used in the process of making films, it is a completely different subject and I am sure you will find loads of information about this on the Internet.

      • Thank you for your reply Daniel~! Yeah it’s true I want to find information about AR was used in film how it is work and how to make the film nicer~In google or other search engine I can’t find a lot of information about it …Yours is the best one!

        Sorry just saw it today..About my report I will finish it today so that come to search something..Anyway thanks for your help~!

        I am delighted to receive your reply ^ ^.

        • Hi Ashboo ! i am working on my thesis these days about AR and cinematography, For now on i just find commercial uses and those attempts listed above in the article. I would be greatfull if you share with me any content that you ve find like articles or something else usefull.

          can’t wait to have good news from you or somebody else in the blog !

  1. Pingback: Augmented Tomorrow - Augmented reality AR archive.

  2. I couldn’t have really asked for a much better blog. You are always at hand to provide excellent information, going straight away to the point for easy understanding of your subscribers. You’re really a terrific pro in this arena. Many thanks.

Leave a Reply