yamammy: A short VR documentary.

 

yamammy is a virtual reality documentary that explores how displaced refugees navigate a world in which they possess disparate overlapping ethnic, religious, and regional identities.

In this project, a former refugee from Sierra Leone Yamammy describes her feelings about some major life transitions – moving from Sierra Leone to Guinea to New York to a small town in Idaho – and how her sense of home and identity has evolved. With this doc, we wanted to create a fragmentary digital portrait of Yamammy using images and stories she describes from her memory.

In class, we presented a rough cut of the 360 video version of the documentary (still a work in progress, so publishing date TBD).

For the ITP Winter Show, we built the VR version of the documentary, which allows audience members to some of the images and stories that Yamammy describes. See the teaser:

We wanted to give the user some autonomy but not too much autonomy. There is some movement but we wanted the user to focus on the audio. Ultimately we wanted Yamammy’s voice to dictate the pace and the rhythm of entire experience.

The concept.

When Ruta and I first discussed this project, we both agreed that we wanted the story itself to determine the technology that we would use. We were both interested in exploring the idea of locative memory: memories that are bound to location or spatial dimensions.

It wasn’t until I had an initial conversation with Yamammy that the documentary began to take shape. We realized that many of the experiences she described were disjointed. Often she would remember tiny specific details while glossing over other parts of her life. Memory itself functions in this way: it’s fragmentary, it’s unreliable, and we tend to remember the stories that match patterns.

Photogrammetry as an aesthetic seems to capture the feeling we wanted our documentary to evoke. Tiny details come into focus while other details are distorted, twisted. It’s an imperfect process, just like the human recall.

The audio.

I initially did a pre-interview with Yamammy to gauge how she might feel about telling some of her stories. It went well and so Ruta and I moved forward with the full interview, which we conducted over Skype. From the two hours of audio, we then combed through the interview and edited it down to a handful of distinctive stories.

The visuals.

We started finding footage online based on Yamammy’s stories. We wanted to capture some of the landscapes she described in an abstract way and so drone footage became a useful creative tool for us.

ffmpeg

We then generated individual frames from the footage and pulled those images into Photoscan, where we created points, then point clouds, then a mesh. After exporting the model from Photoscan, we cleaned it up and smoothed it out in Meshmixer, exporting it as an .obj and a .jpg texture.

untitled-7

We then brought the objects into Unity, where we arranged them into various “scenes” for each audio story we had edited.

light

For the 360 video, we used a Unity plugin called VR Panorama, which allows you to animate a camera that flies through a scene. We edited those videos together with the audio in Adobe Premiere to create the 360 video we presented in class.

For the VR experience, we worked with the Samsung GearVR. We went back into Unity and animated the camera to inch slowly through each scene. We added the audio story to each scene. We also wrote a C++ script that would trigger the next scene when the previous scene had ended.

The feedback.

We showcased the VR experience in December 2016 at the ITP Winter Show. It functioned as a user testing of sorts, since Ruta and I still felt as if the documentary was in a rough place. We received some really good feedback about the experience which included:

  • Tightening up the stories to reduce overall time length.
  • Thinking about replacing some of the visuals – the visuals that worked best were the landscapes.
  • Emphasize the collage effect/aesthetic.
  • Rethink using the Samsung GearVR, as the user experience is not the best (the trackpad is too sensitive, the headset doesn’t fit everyone’s face).
  • Adding music to emphasize certain emotions.

We’re planning to continue working on the VR film and hopefully will submit it to some film festivals this spring.

Project update: Statelessness and identity.

Remembrance of things past is not necessarily the remembrance of things as they were.
– Marcel Proust

Farewells can be shattering, but returns are surely worse. Solid flesh can never live up to the bright shadow cast by its absence. Time and distance blur the edges; then suddenly the beloved has arrived, and its noon with its merciless light, and every spot and pore and wrinkle and bristle stands clear.
― Margaret Atwood

the concept

statelessness and identity is a short 360 documentary using photogrammetry that seeks to understand how displaced individuals navigate a world in which they possess overlapping ethnic, religious, or national identities while lacking a legal identity.

the narrative

Last week I interviewed my friend Yamammy, who was a refugee from Sierra Leone who was resettled in the U.S. in 2001 during the civil war. We talked about the process of getting here, how her idea of home and family has shifted over the years, and what it was like adjusting to a new identity and culture. Ruta and I are scheduling time to re-interview Yamammy and record the audio for the documentary.

Some of the questions we plan to ask her during the interview: How long have you been here? Where do you come from? Did you move with your family? Tell me about the moment you found out that you would be coming to the U.S. When you first moved here how did you feel? To what degree did you connect with the culture here? Tell me about a time you felt connected. What makes you feel at home here? Tell me about a time you felt disconnected. Do you keep in contact with family or friends from home anymore? What was the most difficult part of the process? What do you wish you could do differently if you had to do it all again?screen-shot-2016-11-14-at-6-21-26-pm

the aesthetic

Initially we had a lot of different ideas for what we wanted the documentary to look like. The primary goal was that the documentary be something that was accessible in the browser, whether on FB or youtube. Initially we talked about patching together different 360 images from Google Street View with layered photogrammetric models. After we met with Ziv and Julia, however, they recommended we check out Shining360, a project that uses individual frames from The Shining to create a 3D video using photogrammetry.

I like how tactile the experience is. I was immediately struck by the aesthetic of the video – it’s fragmentary and unpolished, just like human memory. We immediately decided that we want to use this same method in order to re-create the landscapes Yamammy describes in her stories from various video sources.

This week, we’ve planned to meet with Rebecca Lieberman to review the process by which you generate frames from a video and then I’m going to teach Ruta how to use photogrammetry to generate a scene.

Statelessness and identity: A virtual reality experience.

Background research

When Leal’s father was born in Lebanon, her grandfather never registered his birth. Leal herself was born in Lebanon but is undocumented because her father wasn’t able to register the birth of any of his children. Lebanon is one of 27 countries that practices discriminatory nationality laws, according to a 2014 UNHCR report. The nationality law of Lebanon allows only Lebanese fathers to confer their nationality to their children in all circumstances; women cannot confer citizenship to their daughters.

Leal is considered “stateless” according to the UN definition. The term ‘stateless person’ refers to someone who is not considered a national by any country under the operation of its law. There are a number of reason individuals become stateless – perhaps they were displaced after a conflict or belong to an ethnic group that was never recognized by a nation as citizens of the country.

“To be stateless is like you don’t exist, you simply don’t exist,” Leal says. “You live in a parallel world with no proof of your identity.”

Today there are at least 10 million people worldwide who are stateless. Because such individuals are denied a legal identity, they aren’t afforded the same basic human rights we enjoy. Often they are denied access to housing, education, marriage certificates, health care, and job opportunities during their lives and confer the stateless status to their children. Most of these individuals lose their nationality by no fault of their own.

Invisible is the word most commonly used to describe what it is like to be without a nationality,” says Mr. Grandi, Commissioner of the United Nations High Commissioner for Refugees. “For stateless children and youth, being ‘invisible’ can mean missing out on educational opportunities, being marginalized in the playground, being ignored by healthcare providers, being overlooked when it comes to employment opportunities, and being silenced if they question the status quo.”

Why it matters

Statelessness is a product of a world that is increasingly defined by political boundaries and identities. It’s also a product of loose migration laws in regions like Europe, where migrants leave home and then find that they cannot return. Individuals who are stateless often have a difficult time seeking asylum in other countries even though the loss of legal identity isn’t their fault.

Another theme that emerged from our research was the role that discrimination plays in producing stateless populations. As mentioned above, many countries discriminate against women in nationality laws. There are also a number of groups denied citizenship due to ethnic or religious discrimination. For instance, over 1 million of the Muslim Rohingya people in Myanmar do not have citizenship due to religious discrimination against Muslims. In the Dominican Republic, new laws have stripped up to 200,00 Haitian immigrants and their descendants of Dominican citizenship – even going so far as to deport thousands of people.

Ruta and I interested in understanding how such stateless individuals navigate a world in which they might possess overlapping ethnic/religious/national identities but lack a legal identity. The feeling most often described by these individuals is that of invisibility. With this project, we’re aiming to give voice to an individual (or group of individuals) who are unclaimed by a state government.

Concept

The project is a virtual reality film that explores narratives of statelessness and displacement. The idea is to find a compelling personal story and pair it with stunning visuals that match the content of the story. We want the narrative itself to drive the tenor and the mood of the film.

In terms of audience participation, Ruta and I felt more drawn to a heavily curated experience in which the audience hears the audio of a story and explores places that appear in the narrative. We think the best user experience will be one in which the narrative is somewhat more controlled rather than exploratory. We want to maintain an emotional, personal tone to the piece.

Audio will obviously play a huge role in the realization of this project, so we want to make sure the recording we use is itself a character in the film.

Next steps

Right now Ruta and I are reaching out to people we know who are experts in the field of human rights law, refugees, and migration. We’re hoping to make connections with people who are advocating for stateless individuals and find the right story or individual to drive this piece.

I’ve reached out to the following friends/experts:

  • Devon C. – Works for the UNHCR to resettle refugees in the Middle East and has been an advocate for refugees in the current Syrian refugee crisis (see her recent Foreign Policy piece).
  • Thelma Y. – Worked as an activist in Myanmar
  • Estee W. – A student at UPenn law studying discriminatory employment laws in Arab countries. Studied women’s employment laws in Jordan on a Fulbright scholarship.

 

 

 

Unidentified halo: A wearable that thwarts facial detection.

title2

Unidentified halo is a wearable hat that responds to widespread surveillance culture and a lack of biometric privacy in public spaces. The hat is intended to shield the wearer from facial detection on surveillance cameras by creating a halo of infrared light around the face.

As recently as last week, new information has emerged suggesting that as many as half of all Americans are already included in a massive database of faces. Government reports have long confirmed that millions of images of citizens are collected and stored in federal face recognition databases. Police departments across the country use facial recognition technology for predictive policing. One major problem with these systems is that some facial recognition algorithms have been shown to misidentify black people at unusually high rates. There is also the problem of misidentifying a criminal – and how such mistakes can have disastrous consequences.

Shir David and I worked together on this project. We saw this piece as not only a fashion statement, but also an anti-surveillance tool that could be worn by anyone on the street who is concerned about protecting their privacy.

screen-shot-2016-10-18-at-2-30-51-pm

Since the human eye can’t see infrared light, the hat doesn’t draw any attention to the wearer. In the image above, the surveillance camera “sees” a halo of light around my face, preventing Google’s Cloud Vision platform from detecting a face. When we ran the images through Google’s API, it not only detected Shir’s face but even offered suggestions of her emotion based on facial indicators. My face, on the other hand, went undetected.

The project began as a subversive “kit” of wearable items that would allow the wearer to prevent their biometric data from being collected. Shir and I were both frustrated with both the ubiquity and the invisibility of the mechanisms of biopower, from surveillance cameras on streets to fingerprint scanners at the airport. We discussed the idea further with several engineers at NYU and they suggested that if we were interested in pursuing the idea further, we should construct a hat that shines infrared light on the user’s face.

We all agreed that the hat shouldn’t require technical know-how and the battery could be easily recharged. To do this, we soldered together 22 IR LEDs that are powered by a rechargeable 500mAH lithium battery and monitored by a potentiometer, and then adhered the circuit to a baseball cap. The LEDs are wired along the bill of the hat and the battery is tucked into the rim.

img_6683

img_6692


895a9939

Humans can’t see the infrared light unless they are looking through the feed of a surveillance camera, so the wearer won’t draw attention to herself when she wears it on the street. In terms of users, we imagine that this wearable will be worn by someone who wants a way to protect his biometric identity from being tracked while he’s in public without causing a stir.

 

895a9956

895a9988In future versions of the project, we think we would move the LEDs further down the bill of the hat so that it’s closer to the face. We also would ensure that the lithium battery is safely wrapped in a plastic enclosure so that there’s no way it could be accidentally punctured. And, of course, we would sew everything together to improve the appearance of the hat.

We also need to address why the infrared light appears on some IP surveillance cameras but not others – and what kinds of cameras are in use on subway platforms or street corners, for example. Of course, this project fails to address the ubiquity of iPhone cameras, which don’t pick up infrared light and have extremely advanced facial recognition algorithms. These questions will inform the next iteration of the wearable.

Progress on the biometric kit: Prototype and field research.

This week Shir and I did some field research, speaking with several engineers, scientists, and software developers about the viability of some of the ideas we had for our anti-surveillance biometric kit.

We first spoke with Nasir Memon, a professor at the NYU Tandon School of Engineering who specializes in biometrics. He had some ideas for the kit, including some kind of wearable (a hat?) that would hold infrared LEDs that would shield the face from facial recognition while remaining imperceptible to the human eye. Upon his suggestion, we then spoke with three NYU engineering students about the viability of this idea and got some real feedback (some of which was positive, some of which presented more challenges).

We talked to Eric Rosenthal, a scientist and professor at ITP, about some of the work he’s done with IR lights and biometric identity verification while at Disney. Shir also spoke to Lior Ben-Kereth, a partner at the facial recognition company acquired by Facebook.

We decided to move forward with the infrared LED wearable idea, but first we needed to ensure that a range of different kinds of cameras do indeed pick up the infrared light. We connected a cluster of IR LEDs and pointed them at our iPhone camera, FaceTime, Snapchat, and a range of IP surveillance cameras – including three different kinds that are in use at ITP.

You can see the result of our test below:

Biometric data kit: Update on project development.

This week, Shir and I have been discussing the biometric anti-surveillance kit that we will be building for our midterm project.

What is biometric data? 

Biometrics are the measurable, distinct characteristics that are used to verify the identity of individuals, including groups that are under surveillance. Biometric data includes fingerprints, DNA, face recognition, retina scans, palm veins, hand geometry, iris recognition, voice, and gait.

Problem area

Biometric data is extremely sensitive. If your data is compromised, it’s not replaceable (unlike password tokens). The widespread collection of personal, biometric data raises questions about the sharing of such data between government agencies or private companies. Many of us use the Apple Touch ID on a daily basis and yet we don’t think about the fact that Apple now has access to a snapshot of our fingerprint. In addition, biometric data is most often collected by the state about populations that are already vulnerable, including criminals, travelers, and immigrants.

Proposed project

We intend to put together a biometric resistance kit, a toolkit of wearable objects aimed at masking and altering user’s personal biometric identity. The aim of the project is to prototype non-intrusive objects that can be worn by anybody to protect their biometric identity in public spaces.

hacking-story-frame-works-001Contents of the kit

We had several ideas of what the kit could contain.

hacking-story-frame-works-007 hacking-story-frame-works-008 hacking-story-frame-works-009 hacking-story-frame-works-010 hacking-story-frame-works-011

Relevant projects

We researched what had been done in the past and found several other artists and engineers experimenting with anti-surveillance materials.

hacking-story-frame-works-013

Identity is a project by Mian-Wei that uses a band-aid made of silicon and fibers to trick the Apple Touch ID into thinking it’s a real fingerprint. The solution is simple and effective, something we would like to achieve with our project.

hacking-story-frame-works-014Biononymous Guide is a series of DIY guides for masking your biometric identity (specifically, DNA and fitness trackers). We loved the format of the website – the mix of printed materials, physical objects, and how-to videos match the kind of kit we’re hoping to build.

hacking-story-frame-works-015

Adam Harvey is an artist whose anti-surveillance work includes Stealth Wear, a line of Islamic-inspired clothing to shield against drone attacks & thermal cameras, and CV Dazzle, a makeup guide that beats facial recognition algorithms. We loved how he tried to work with styles that people would actually want to wear. This will likely be a major concern in the development of our project.

Field research and interviews

We’ve spoken with a few experts in the field of biometrics, surveillance, and facial recognition algorithms and are planning to continue these conversations.

First, we spoke to Nasir Memon, a computer science professor and biometrics expert at NYU’s Tandon School of Engineering. He had some ideas for the kit, including a hat with infrared lights that would beat facial recognition algorithms. We also spoke to NYU engineering students who are researching surveillance, machine learning, and biometrics. They gave us additional technical guidance about what in our kit would be most viable. We also have set up meetings with artist Adam Harvey and NYU professor Eric Rosenthal to discuss our project idea.

User personas

Shir and I have had many conversations about who this kit would affect. We realized that for the time being, there is no reasonable way to combat the fingerprinting that occurs for immigrants, foreign visitors, criminals, or people who are required to scan their fingerprints for work. Instead, we realized that this kit would be best suited for people who are worried about their privacy and surveillance when they’re in public spaces. Here are several example users we created:

hacking-story-frame-works-002 hacking-story-frame-works-003 hacking-story-frame-works-004

Midterm project proposal: Biometrics.

Problem framework.

For my midterm project, I’d like to address the ethics and implications of widespread biometric data collection.

Biometric identifiers are defined as measurable, distinctive characteristics that are used to label or describe individuals. They’re commonly used by governments and private organizations to verify the identity of an individual or group of individuals, including groups that are under surveillance. Physiological characteristics include fingerprints, DNA, face recognition, retina scans, palm veins, hand geometry, and iris recognition. Behavioral identifiers measure behavioral patterns like voice and gait.

Here’s the breakdown of identification accuracy based on biometric input:

screen-shot-2016-09-27-at-5-33-40-pm

The earliest record of fingerprinting cataloguing dates back to 1891. Biometrics arguably originated with “identificatory systems of criminal activity” as part of a larger system to categorize and label criminal populations. “The biometric system is the absolute political weapon of our era” and a form of “soft control,” writes Nitzan Lebovic. Under the post-9/11 expansion of the Patriot Act, biometric systems have expanded from the state to the private market and blurred the lines between public and private control.

While biometric data is seen as being more accurate and therefore more reliable as a way to identify an individual, it is also not replaceable. If your private password was somehow compromised, for instance, you could simply change your password. On the other hand, you can’t replace your fingerprint or change other physical characteristics.

Italian theorist Giorgio Agamben experienced the implications of “bio-political tattooing” firsthand in 2004 when he was told that in order to obtain a U.S. visa to teach a course at New York University he would have to submit himself to fingerprinting procedures. In a piece published in Le Monde, Agamben explains why he refused to comply, arguing that the electronic filing of finger and retina prints required by the U.S. government are ways in which the state registers and identifies naked life. According to Agamben, biometric data collection operates as a form of disciplinary power.

Audience.

This issue affects potentially everyone so our audience is very broad. Biometrics data is most often collected about populations that are already vulnerable, including criminals, the poor, and immigrants. Corporations put a monetary value on biometric data, and yet individuals don’t think about data collection as an intrusion.

The goal of this project is to foster an awareness of the implications and ethics of biometric data collection.

Ideas for the project and user journey.

Concept #1: A physical installation that gives the user personalized information based on a biometric input.

Concept #2: A speculative VR experience with advertisements tailored to user’s biometric data.

Concept #3: A kit of wearable objects aimed at masking and altering user’s personal biometric identity.


http://motherboard.vice.com/read/i-replaced-my-fingerprints-with-prosthetics-to-avoid-surveillance

Concept #4: Collect (non-identity-compromising) biometric data from various participants and sell data on eBay in order to gauge the monetary value of the data.


http://thecreatorsproject.vice.com/blog/this-artist-turned-herself-into-a-corporation-to-sell-her-data

 

Redesign of a narrative experience: Mrs Dalloway in 360.

screen-shot-2016-09-20-at-5-32-44-pm

She would not say of any one in the world that they were this or were that. She felt very young; at the same time unspeakably aged. She sliced like a knife through everything; at the same time was outside, looking on. She had a perpetual sense, as she watched the taxi cabs, of being out, far out to the sea and alone; she always had the feeling that it was very, very dangerous to live even one day.
– Virginia Woolf, Mrs. Dalloway

For this week’s assignment, we were asked to redesign a narrative experience according to the agile human-centric design principles we discussed last week.

For my source material, I drew from the themes and text of Virginia Woolf’s Mrs Dalloway, a 1925 novel written in a stream of consciousness literary style that sketches the portrait of life of one woman, Clarissa Dalloway, during the course of one single day.

In the first chapter of the book, Clarissa walks the streets around London running errands in preparation for a party she is throwing that night. When I reread the book, I was struck by the ways in which the novel sharpens our attention to details of time and space, especially the specificity of London during Clarissa’s walks. Time is a significant theme in the novel, with clocks ringing the hour and signs of aging and death made hypervisible in the text. So much of the narration in the novel occurs inside the head of the protagonist, with special attention paid to her surroundings.

With this project, I wanted to explore creating a film that employs this stream of consciousness narrative style while physically putting you in the shoes of the protagonist. I chose to reimagine Mrs Dalloway as an immersive VR/360 experience in order to explore this narrative style not only in text, but also in film.

The idea behind the project was to film myself walking in New York using 360 video, paired with a voice over narration of the opening chapter of the novel. I made slight changes to the text in order to accommodate the sharp departure in setting (from 1925 London to 2016 New York). Much of the narration in the novel is observational — Clarissa sees a woman in a taxi cab, she arrives at the park, she looks in shop windows — and I wanted to replicate those moments in the film as much as possible.

Check out the initial prototype of my idea.

YouTube:

 

My audience for this project could be anyone, really. Because it’s a 360 video, the user has full control over what he or she is looking at during the film. Just like London, New York is replete with observational details; I wanted the audience to experience that same sensory overload in my project.

Vimeo:

Other ways of telling: Andrei Tarkovsky.

Listen: you are not yourself, you are crowds of others, you are as leaky a vessel as was ever made, you have spent vast amounts of your life as someone else, as people who died long ago, as people who never lived, as strangers you never met. The usual I we are given has…none of the porousness of our every waking moment, the loose threads, the strange dreams, the forgettings and misrememberings, the portions of a life lived through others’ stories, the incoherence and inconsistency, the pantheon of dei ex machina and the companionability of ghosts. There are other ways of telling.

― Rebecca Solnit, The Faraway Nearby

One of the goals of VR documentaries, suggested by the filmmakers behind Collisions and Clouds over Sidra, is to give the audience a sense of ‘presence.’ Immersive experiences in VR allow the participant to feel that he or she is physically in the same location as the camera. The technical constraints of VR filmmaking demand slow, deliberate camera movements. Many VR films have a structured narrative while still giving the audience the agency to decide what they will look at during the film. To paraphrase one reviewer of Clouds over Sidra, what moviemaker in the past would include 20 seconds of ceiling shots, looking up at the top of the tent?

I include these observations about VR film because I’d like to highlight the ways in which Russian filmmaker Andrei Tarkovsky uses similar techniques in many of his films to produce an immersive, experiential cinema experience.

During his career, Tarkovsky directed only seven feature films, including Solaris, Stalker, Andrei Rublev, and Mirror during the period 1962-1986. He was a pioneer in the field of cinema, experimenting with new narrative techniques and theories. Many of Tarkovsky’s films are characterized by extremely long takes, slow camera pans, and very few cuts. He developed a theory of cinema called “sculpting in time,” in which he explored how film can twist and alter the audience’s experience of time. Unedited movie footage and lengthly sequences were used to heighten that feeling of time passing.

I watched Solaris for the first time in college and I remembered being shocked by a long, drawn-out driving sequence within the first ten minutes of the film. The scene is nearly five minutes long.

Tarkovsky writes: “If the regular length of a shot is increased, one becomes bored, but if you keep on making it longer, it piques your interest, and if you make it even longer, a new quality emerges, a special intensity of attention.”

Tarkovsky also employed common motifs of running water, clouds, and reflections in his films. Many understood his preoccupation with reflective surfaces to mirror his own interest in self-reflection and introspection. Of Tarkovsky, Ingmar Bergman said: “Tarkovsky for me is the greatest (director), the one who invented a new language, true to the nature of film, as it captures life as a reflection, life as a dream.”

On a personal level, I’ve learned from Tarkovsky that compelling films do not need to have a strict narrative, nor follow time limits or other cinematic constraints. Tarkovsky’s films were so powerful because they pushed the audience into a state of heightened attention.

Crafters of immersive/experiential films often need to make similar decisions about timing, camera movement, and narrative in order to tell the most compelling story possible. The VR film Collisions, for instance, mixed beautiful, wide landscapes with meandering narration in very lengthy shots. With my work in this class, I’m interested in exploring the kinds of experiential, non-linear narrative that Tarkovsky’s films often embody.

Here’s the presentation I gave in class: