Context

Originally, the selected project “Shift: the inner journey” was a virtual reality documentary on shifting: a mental meditation technique that appeared on the TikTok social network. Through various discussion and brainstorming sessions, the project was renamed “Beyond Midnight” and transformed into an immersive installation with a narrative mise en abyme, while retaining the subject of the mind’s journey through technology and spirituality.

The residency took place in Kaohsiung. We had around ten days to make a prototype for the opening of the festival, then another ten days to make a final prototype for the arrival of the professionals, artists and producers. We worked on the design of the interactivity, the sensations of the experience, the installation, the collection of content using photogrammetry and volumetric capture, and interviews with technological, cultural and spiritual actors.

A very thin time frame for such a planning. Time definitively had an impact on production design decision: we decided to focus on content, interactivity and installation to the detriment of the narrative. This last point is now the keystone of the next steps of the project.

Narrative

The narrative objective of the project is to create an immersive experience that showcases digital creation. Through the pursuit of a character, we discover what goes on behind the scenes of a digital project that unfolds as we explore. The character we are looking for is a young person who has managed to use the shifting technique to travel through his inner world, and whose mental potential is so brilliant that they breaks through virtual boundaries to enter the imaginary world of the project. By staging the creators of the experience in our own image, we are addressing the user, giving them the mission of finding this disruptive agent with infinite potential.

The link in between

All the layers have the purpose to build an installation that is both a digital sculpture to appreciate from a distance and an interactive immersion to experience inside. The next step is to develop further the link in between all medium, make it explicit that we are perceiving the same digital world through different lens.

For instance the projection will project user interface gizmos that tracks the position of the headset and controllers, as if we were seeing through an x-ray screen. Data such as velocities, elevation, localization will also be illustrated as global tracking interface. The projection will also reuse the current position of the video game avatar to show the current digital landscape.

The video game avatar and the virtual reality user will also be interacting with each other through network synchronization.

Interviews

Through interviews with players in the field of technology, spirituality and culture, we explore the theme of the digital as a new spiritual space and how the world of the mind can be a mental technology.

Wang Airu

A Taiwanese documentary cinematographer renowned for her work, including the critically acclaimed series “Island of Mountains”. While filming “A Journey with Invisible Friends” a documentary exploring spirituality, she unexpectedly discovered her own spiritual abilities. Currently, she is in the process of filming a documentary that features her as the protagonist, documenting her extraordinary
encounters with otherworldly entities.

Huang Douni

An Information Security Engineer at the Ministry of Digital Affairs, he is recognized for initiating the FAB DAO, which translates art memes into tangible assets. This creative endeavor involves transforming them into NFT collectibles and auctioning them, aiming to bridge the industry into the Web 3 era.

Shang Chiao Li

Writer/director of film and writer of comic, she is dedicated to cross cultural and discipline boundaries through story telling. Li, Shang Chiao is the screenwriter of animated film and manga “Mayfly Island”. The works were shortlisted for Excellent Scripts Award, Golden Horse FTP, and was supported by the Ministry of Culture, Taiwan. In 2020, she was awarded Youth Creation Award held by the Ministry of Culture, and is expected to create a collection of historical fictional short stories “The Twilight Zone”. The film “March: The Southern South ” will be included as one of the short stories.

Working at Pier 2

KFF provided a large studio and hardware to work in situ. We were able to use the space to reproduce the festival installation: project video on the wall and test the VR experience in the same area dimension space.

Volumetric capture

We used the Kinect Azure depth sensor to record volumetric capture of the interviews.

Virtual exploration

In this video, Dorian is presenting the video game experience of the installation.

Building cyberspace

In this video, Dorian is presenting how he works with digital content in Unreal engine.

Immersive interaction

Inside the experience, you unveil the environment by observing it. Thousands of particles are projected from your gaze and settle on the surfaces as they are revealed. These points react to our presence, initially attracted and then stretched in the direction of our movement.

Photogrammetry

We use photogrammetry to capture real places and moments.

Set Design

We used vintage monitors and old computers.

Props

We created a set of prints, inspired by old fashioned movie posters and video game jackets.