yamammy: A short VR documentary.

 

yamammy is a virtual reality documentary that explores how displaced refugees navigate a world in which they possess disparate overlapping ethnic, religious, and regional identities.

In this project, a former refugee from Sierra Leone Yamammy describes her feelings about some major life transitions – moving from Sierra Leone to Guinea to New York to a small town in Idaho – and how her sense of home and identity has evolved. With this doc, we wanted to create a fragmentary digital portrait of Yamammy using images and stories she describes from her memory.

In class, we presented a rough cut of the 360 video version of the documentary (still a work in progress, so publishing date TBD).

For the ITP Winter Show, we built the VR version of the documentary, which allows audience members to some of the images and stories that Yamammy describes. See the teaser:

We wanted to give the user some autonomy but not too much autonomy. There is some movement but we wanted the user to focus on the audio. Ultimately we wanted Yamammy’s voice to dictate the pace and the rhythm of entire experience.

The concept.

When Ruta and I first discussed this project, we both agreed that we wanted the story itself to determine the technology that we would use. We were both interested in exploring the idea of locative memory: memories that are bound to location or spatial dimensions.

It wasn’t until I had an initial conversation with Yamammy that the documentary began to take shape. We realized that many of the experiences she described were disjointed. Often she would remember tiny specific details while glossing over other parts of her life. Memory itself functions in this way: it’s fragmentary, it’s unreliable, and we tend to remember the stories that match patterns.

Photogrammetry as an aesthetic seems to capture the feeling we wanted our documentary to evoke. Tiny details come into focus while other details are distorted, twisted. It’s an imperfect process, just like the human recall.

The audio.

I initially did a pre-interview with Yamammy to gauge how she might feel about telling some of her stories. It went well and so Ruta and I moved forward with the full interview, which we conducted over Skype. From the two hours of audio, we then combed through the interview and edited it down to a handful of distinctive stories.

The visuals.

We started finding footage online based on Yamammy’s stories. We wanted to capture some of the landscapes she described in an abstract way and so drone footage became a useful creative tool for us.

ffmpeg

We then generated individual frames from the footage and pulled those images into Photoscan, where we created points, then point clouds, then a mesh. After exporting the model from Photoscan, we cleaned it up and smoothed it out in Meshmixer, exporting it as an .obj and a .jpg texture.

untitled-7

We then brought the objects into Unity, where we arranged them into various “scenes” for each audio story we had edited.

light

For the 360 video, we used a Unity plugin called VR Panorama, which allows you to animate a camera that flies through a scene. We edited those videos together with the audio in Adobe Premiere to create the 360 video we presented in class.

For the VR experience, we worked with the Samsung GearVR. We went back into Unity and animated the camera to inch slowly through each scene. We added the audio story to each scene. We also wrote a C++ script that would trigger the next scene when the previous scene had ended.

The feedback.

We showcased the VR experience in December 2016 at the ITP Winter Show. It functioned as a user testing of sorts, since Ruta and I still felt as if the documentary was in a rough place. We received some really good feedback about the experience which included:

  • Tightening up the stories to reduce overall time length.
  • Thinking about replacing some of the visuals – the visuals that worked best were the landscapes.
  • Emphasize the collage effect/aesthetic.
  • Rethink using the Samsung GearVR, as the user experience is not the best (the trackpad is too sensitive, the headset doesn’t fit everyone’s face).
  • Adding music to emphasize certain emotions.

We’re planning to continue working on the VR film and hopefully will submit it to some film festivals this spring.

Project update: Statelessness and identity.

Remembrance of things past is not necessarily the remembrance of things as they were.
– Marcel Proust

Farewells can be shattering, but returns are surely worse. Solid flesh can never live up to the bright shadow cast by its absence. Time and distance blur the edges; then suddenly the beloved has arrived, and its noon with its merciless light, and every spot and pore and wrinkle and bristle stands clear.
― Margaret Atwood

the concept

statelessness and identity is a short 360 documentary using photogrammetry that seeks to understand how displaced individuals navigate a world in which they possess overlapping ethnic, religious, or national identities while lacking a legal identity.

the narrative

Last week I interviewed my friend Yamammy, who was a refugee from Sierra Leone who was resettled in the U.S. in 2001 during the civil war. We talked about the process of getting here, how her idea of home and family has shifted over the years, and what it was like adjusting to a new identity and culture. Ruta and I are scheduling time to re-interview Yamammy and record the audio for the documentary.

Some of the questions we plan to ask her during the interview: How long have you been here? Where do you come from? Did you move with your family? Tell me about the moment you found out that you would be coming to the U.S. When you first moved here how did you feel? To what degree did you connect with the culture here? Tell me about a time you felt connected. What makes you feel at home here? Tell me about a time you felt disconnected. Do you keep in contact with family or friends from home anymore? What was the most difficult part of the process? What do you wish you could do differently if you had to do it all again?screen-shot-2016-11-14-at-6-21-26-pm

the aesthetic

Initially we had a lot of different ideas for what we wanted the documentary to look like. The primary goal was that the documentary be something that was accessible in the browser, whether on FB or youtube. Initially we talked about patching together different 360 images from Google Street View with layered photogrammetric models. After we met with Ziv and Julia, however, they recommended we check out Shining360, a project that uses individual frames from The Shining to create a 3D video using photogrammetry.

I like how tactile the experience is. I was immediately struck by the aesthetic of the video – it’s fragmentary and unpolished, just like human memory. We immediately decided that we want to use this same method in order to re-create the landscapes Yamammy describes in her stories from various video sources.

This week, we’ve planned to meet with Rebecca Lieberman to review the process by which you generate frames from a video and then I’m going to teach Ruta how to use photogrammetry to generate a scene.