Impressions from MozFest 2017

Last week I attended MozFest 2017 as a Ford-Mozilla Open Web Fellow. This year there was a huge emphasis placed on building a diverse and inclusive community of technologists engaging critically with the internet – a crucial decision that really added value and weight to the conversations we were having.

The Thursday night before the conference, the other fellows and I set up a booth to hand out the zines we had each created. I designed a zine called “The Ordinary Affects of Facebook,” which explored the personal narratives and anecdotes I had collected from friends about their affective experience on the platform. The content for the zine came from research I conducted for my ITP master’s thesis on Facebook.

I also handed out another zine entitled “Altered States: AR and the Politics of Urban Social Memory” which summarized some of the concepts I spoke about in a workshop Zoe Bachman and I taught last week at Pioneer Works.

 

Some highlights and general themes:

Digital Security
 

 

I attended a hands-on session with the organization CHAYN, which aims to build tools to empower women online. In groups we brainstormed ways in which we could use metaphors, illustrations, and GIFs to illustrate the basic principles of digital security: VPNs, malware, cookies/trackers, etc. I found it to be a useful exercise to take some of these complex ideas and break them down into simple, concrete metaphors. For instance, we were thinking about cookies/trackers in terms of health/nutrition and VPNs in terms of stamps in a passport.

 

 

In a keynote, Matt Mitchell of CryptoHarlem spoke about strategies for establishing digital security norms at NGOs. He argues that we should be treating security the same way we treat sexual harassment or discrimination in the workplace: through a code of conduct. Start by creating a (1) Security policy (2) Checklist for emergencies (3) Data retention policy (4) ‘Weakest link’ policy (5) Making digital norms a regular practice, starting with the onboarding process.

Algorithmic bias & policing

The ACLU is investigating law enforcement’s paid use of Geofeedia, a now-defunct social analytics software that purported to identify social media users who were radical Islamists. The ACLU obtained thousands of emails and training documents through a FOIA request to better understand how Boston’s police department was using the software. The issue here is that data isn’t collected uniformly and often just reflects institutional bias.

Keynote speaker Julia Angwin spoke about the investigative work she’s done at ProPublica on policing and algorithms. She emphasized the ways in which our society metes out forgiveness and punishment selectively. Algorithms that predict risk of recidivism or car insurance algorithms that are based on zip code penalize one set of people, while forgiveness for another set of people is baked into the algorithms. More systems of oversight and accountability are needed.

Net Neutrality

Global Voices introduced its report analyzing and critiquing Facebook’s ‘Free Basics’ program, which is Facebook’s attempt to bring more people online in areas where there is low internet penetration. Global Voices’ research found that the program gives users access to a suite of pre-loaded apps that are largely Western-centric and prioritize Facebook products. The research also found that contrary to Facebook’s claims that it was not collecting data on Free Basics Users, users were constantly sharing data with Facebook. The program violates the basic tenets of net neutrality, with certain websites being prioritized over others.

In a panel discussion, lawyers and policymakers from Brazil, the EU, the U.S., and India discussed the particular contours and challenges of net neutrality in their respective countries and strategies for keeping the internet open. From the U.S. perspective, Gigi Sohn reminded us that consumers now predominantly view broadband internet as a pipe – and we should therefore treat it as a utility. In the EU, there are questions questions about where the open internet ends and our personal networks start. The introduction of the Marco Civil internet bill of rights in Brazil was groundbreaking for net neutrality, but it is not necessarily enforceable. In India, the net neutrality debate has been derailed by parties who argue that free access to the internet should be prioritized over competition between telecom companies.

Fake News

Verge reporter Sarah Jeong and data scientist and activist Emily Gorcenski had a conversation in which they discussed the distinct affordances of social platforms and how clickbait profiteers were able to hijack these platforms to disseminate fake news stories. The problem of ‘fake news’ is at its heart a structural problem, as we are producing content that is meant to propagate. “We’ve figured out how to weaponize people’s confirmation bias,” says Gorcenski.

Making Privacy Personal

In the past several weeks I’ve had several one-on-one conversations with friends and colleagues about digital privacy rights. Some were concerned about the latest news that WAP2 protocol has a huge glaring security vulnerability. Others were curious about what information Facebook collects about them and how that information might have been used in the 2016 U.S. election.

What became clear to me is that there is a real lack of literacy when it comes to talking about online privacy. People are concerned but often they’re too embarrassed or ashamed to admit that they don’t understand what they should be concerned about – usually people just ignore the problem unless they’re directly attacked.

There’s a community of organizers who host CryptoParties – free, community-based workshops that teach the basics of cybersecurity to a lay audience. You don’t need to know anything about PGP or Tor in order to attend and participate. I want to organize a small home-based cryptoparty first with my friends and then later more broadly.

The goals of my workshop:

  • Educate internet users about their digital privacy rights
  • Help consumers think critically about the online platforms and tools they use
  • Empower individuals to assess their own situation through ‘threat modeling’
  • Foster a safe, judgement-free space in which people can express their specific concerns and fears about their privacy
  • Teach the group about specific tools they can download and behaviors they should model to mitigate risks:
    1. Passwords (and password managers e.g. LastPass)
    2. Trackers and cookies (Privacy Badger, ad blockers, disabling cookies)
    3. The big social platforms: Facebook and Google
    4. HTTPS, TLS, and asymmetric encryption (HTTPS Everywhere)
    5. End-to-end encryption (Signal, WhatsApp, iMessage)
    6. Encrypting email (PGP)
    7. VPN (Private Internet Access vs. your own server)
    8. Tor Browser

I will be developing the curriculum via a GitHub repository located here.

User testing: Feedback from the Quick & Dirty show.

Photo by Tiriree Kananuruk

Yesterday I had the opportunity to user test my thesis project as it exists in its current state at the Quick & Dirty show. Since I’ve been doing some disparate experiments, I decided to show two of the pieces in an attempt to get feedback on what works, what feels compelling, and how the projects might be better synthesized.

First, I showed a web application I built that uses IBM Watson’s Personality Insights API (i.e. psychometrics) to make assumptions about who you are as a person. The user logs into Facebook in the application and then a dashboard appears that shows them their predicted psychological makeup and purchasing habits. I tried to take a satirical/speculative approach, suggesting what psychometrics could look like in the future.

Photo by Tiriree Kananuruk

Second, I showed the work I had done on generating 3D facial models from 2D images. The idea is that after a user logs into Facebook, the application will automatically produce a 3D model of their face just from their Facebook photos. I showed the ‘Manufactured Self’ website with the 3D model of my face. Earlier in the day, I had 3D printed a face, so for the show I projected the isomap facial image on top of the 3D model to lend the 3D experiment more tactility.

People responded really well to the visual aspect of the project and expressed a desire to see more of a connection between this visual and the psychometric web app.

Photo by Tiriree Kananuruk

Overall the feedback was so useful. I felt as if the common theme was a desire for a stronger framing of the project. How do I want the audience to feel as an end result? What kind of approach or tone should I be taking? Here’s a summary of some of the comments and my initial responses.

  • What do you want the audience to feel? Creeped out, shocked, confused, grossed out. 
  • There needs to be space within the project for users to RESPOND or express their feelings about this. How can I add a way for users to respond to the project?
  • Is it an aura reading or a satire on marketing of the future? Should it be more speculative/dystopian? Which approach will achieve what you want?
  • How is the 3D model connected to the personality test? Make that connection clearer & stronger.
  • This kind of data matters more in the aggregate. It would be cool if you could see how you compare against other people. Maybe when you add your data it enters like a big visualization and you can see who you are closest to.
  • It needs to be clearer that this is meant to be CRITICAL of such processes. How do you make this more absurd? Change the language, push it further.

Right now I need to figure out how to strike the right tone with this project so as to make it as compelling and powerful as it can be.

Locating Facebook metadata in physical landscapes.

My Facebook metadata as landscape.

This semester, I’ve focused my attention on creative ways of interpreting and visualizing my personal Facebook data.

I’m interested in exploring the concept of “digital dualism” – the habit of viewing the online and offline as largely distinct (source). We are actively constructing our identities whether behind a screen or in person. As Nathan Jurgenson writes, “Any zero-sum “on” and “offline” digital dualism betrays the reality of devices and bodies working together, always intersecting and overlapping, to construct, maintain, and destroy intimacy, pleasure, and other social bonds.”

The exact location where I made a Facebook update.

With this project, I wanted to try re-inserting the digital world into the physical world. I decided to locate specific actions I took on Facebook within a physical geography and landscape.

It’s very easy to download your Facebook metadata from the website – all you have to do is follow these directions. In my data archive, I found information about every major administrative change I’ve made to my Facebook account since I created the account in 2006, including changes to my password, deactivating my account, changing my profile picture, etc. This information was interesting to me because from Facebook’s perspective, these activities were in all likelihood the most important decisions I had ever made as a Facebook user.

I rearranged that data into a simple JSON file:

I decided to explore the IP Address metadata associated with each action. I wanted to know more about the physical location where I had made these decisions concerning my Facebook account, since I obviously didn’t remember where I was or what I was doing when I had made these changes.

I wrote a Python script (see code here) that performs several different actions for each item in the JSON file:

(1) Takes the IP address and finds the corresponding geolocation, including latitude & longitude & city/state;

(2) Feeds the latitude/longitude into Google Maps’ Street View and downloads 10 images that each rotate 5 degrees;

(3) Adds a caption to each image specifying the Facebook activity, the exact date/time, and the city/state; and

(4) Merges the 10 images into a gif.

The result was two dozen weird undulating gifs of Google Street View locations, which you can check out on the project website.

After doing all that work, however, I didn’t feel satisfied with the output. If the goal was to find a way to re-insert my digital data trail into a physical space, I felt that the goal hadn’t yet been realized in this form. I decided to take the project into a different, more spatially-minded direction.

I wrote another Python script that programmatically takes the IP address and searches for the latitude/longitude on Google Maps, clicks the 3D setting, records a short video of the three-dimensional landscape, and then exports the frames of that video into images.

Programmatically screen recording Google Maps’ 3D landscape.

Using the photogrammetry software Photoscan, I created a 3D mesh and texture from the video frames. Then, I made a quick design of the Facebook app on an iPhone with the specific Facebook activity associated with that location & IP address. Finally, I pulled the landscape .obj into Unity with the iPhone image and produced some strange, fantastical 3D landscapes:

Pulling the 3D mesh into Unity and inserting the Facebook metadata into the landscape.

Thesis progress, Week 7.

The past few weeks have allowed me to think deeply about what I want to get out of my thesis project and what form this project will take. I wrote last week of the idea of the “manufactured self” – a self that has been constructed socially by external sources of power.

I stumbled on Alexandru Dragulescu’s thesis paper Data Portraits: Aesthetics and Algorithms, which outlines his creative practice for data portraiture. He describes “the concept of data portraits as a means for evoking our data bodies” and showcases his “data portraiture techniques that are re-purposed in the context of one’s social network.”

With my project, I will attempt to create a portrait of each participant based only on his or her Facebook data. I want to use facial recognition models (C++ and Python), 3D modeling (Three.js, Blender), the Facebook Graph API, and IBM Watson’s Natural Language Processing and Personality Insights APIs.

 

Visit the website: http://rebecca-ricks.com/manufactured-self/becca.html

After much experimentation, I have an overall idea of what the user flow will look like. There will be an online web application + a physical component. Here’s the flow:

(1) User logs into web application (with Facebook Oauth)

(2) Real-time analysis of personality + generate 3D facial model

(3) The 3D object is manipulated/distorted based on the personality insights (?)

(4) At the show, users will be able to take home a physical artifact of their data portrait (thermal print of the 3D model? An .obj? A list of personality insights?)

This week, I used C++ and Python to get this library up and running, which allows you to create a 3D model of a face from a 2D image. I spent a significant amount of time trying to install the library, generate the landmark points, and run the analysis on my own images. Here’s what that process looked like:

Generating the landmark points based on a photo of my face.
Generating the isomorphic texture that will be applied to the 3D model.
The 3D mesh model that the texture is applied to.
The final output displayed in the browser using three.js.

I also got access to a few of IBM Watson’s APIs via the Python SDK. Specifically, I’m looking at the Personality Insights API, which analyses a body of text (your Facebook likes, your Facebook posts, etc). I ran the analysis on my own Facebook data, and added the information to the website I built from the JSON file that was generated.

You can see an example of what that analysis looked like on my own Facebook data:

http://rebecca-ricks.com/manufactured-self/becca.html

I also decided to test my 2D to 3D model on an earlier image I had created of my composite face based on every Facebook photo I’ve been tagged in.

http://rebecca-ricks.com/manufactured-self/facemash.html

Thesis progress, Week 6.

Last week I presented my midterm presentation and received some great feedback and suggestions. I resonated most with what Sam said about the monetization, commodification, and production of the self that occurs on Facebook. How can I incorporate that more fully into my thesis project?

I’m still iterating on a few different ideas, but eager to find the final form that my project will take, whether it’s one fully-developed web application or several different experimental applications.

I found some visual inspiration that has fueled the project I’m working on this week.

Source: https://labs.rs/en/

Share Lab has been investigating ‘The Facebook Algorithmic Factory’ with the intention “to map and visualize a complex and invisible process hidden behind a black box.” The result is an exploration of four main segments of the process: Data Collection (“Immaterial Labour and Data harvesting“), Storage and Algorithmic processing (“Human Data Banks and Algorithmic Labour“), and Targeting (“Qualified lives on discount“).

I was struck by not only the depth of research into Facebook’s policies and practices but also the beautiful (static) data visualizations produced as a way to clarify the research.

Source: https://labs.rs/en/
Source: https://labs.rs/en/

These data visualizations are simple but powerful. It left me thinking: How do I make this complex web personal ? How do I communicate the ways in which this process immediately affects every Facebook user? Can I make use of the Facebook API to build a graphic that takes the user’s personal information (likes, friends, advertisements) and displays them in an interactive web-based application?

I want to make use of a lot of the good research done by Share Lab as well as my own research to build an interactive web application that helps users see how their personal data is collected, stored, and used in order to manufacture a self, or a “consumer profile.” I was struck by what Nancy said about Facebook manufacturing a self and I think this would be a good conceptual starting point.

Right now I’m starting to build the web application using the Facebook Graph API, Facebook CLI, and a D3 clustering algorithm. I’m starting by building a web application that collects information about user_likes clustered according to category.

Film trailers as told through ‘visually similar’ images

This week, we reviewed useful tools ffmpeg and imagemagick to manipulate images and videos found online. I decided to start working with the trailer to Akira Kurosawa’s 1985 film Ran (Japanese for “chaos”). Ran is a period tragedy that combines the Shakespearian tragedy King Lear with legends of the daimyō Mōri Motonari.

The trailer is filled with beautiful, carefully framed shots. I wanted to see if there was a way to automatically detect and chop up the trailer into its individual shots/scenes. It turns out there is no simple solution to that problem so I hobbled together my own bash script to do so.

Once I had chopped up the trailer, I decided to export one image from each scene for analysis. I did so by writing a script that saves the first frame from each video.

I then used selenium to programmatically upload those images into a reverse image search that was powered by an image classifier that had been trained on Wikipedia data. The image classifier had been trained by Douweo Singa and the site can be accessed here. It’s described this way: “A set of images representing the categories in Wikidata is indexed using Google’s Inception model. For each image we extract the top layer’s vector. The uploaded image is treated the same – displayed are the images that are closest.” You can read more detailed notes about training the data in Douweo’s blog post.

I ended up with hundreds of ‘visually similar’ images, organized according to the shots in the trailer. I combined them into a side-by-side comparison video, where you can see some of the images that were deemed ‘visually similar’ by the training set. Check out the full video for Kurosawa’s Ran:

I then decided to repeat the entire process for the trailer to Dario Argento’s classic horror film Suspiria:

Thesis: Progress, Week 4

This week, I decided to experiment with a few different ideas and technologies in order to further develop my thesis project. Here are some of the projects/experiments I worked on:

Experiment #1: Chrome Extension (for Facebook).

A chrome extension that swaps all the pictures in your Facebook feed with the logos of the advertisers that currently have your contact information.

See the code repo here.

I started by downloading my entire Facebook archive (do it yourself). I found a list of all the advertisers who have my contact information from Facebook – more than 200 entities in total. This information shocked me, especially because a number of them were data collection companies and politicians running for office.

I took that list of advertisers and decided to scrape Google Images to download all their logos.

Then I shifted gears and built a Google Chrome extension that swaps all the images on Facebook for any images of your choosing. I wrote code that picks a random image from the folder of advertisers every time the page reloads.

Personally, I found that the advertisements for various Senators and politicians to be the most intrusive and unwanted.

Experiment #2: Facemash (for Facebook and LinkedIn).

A Python script that scrapes tagged images from Facebook and LinkedIn, and then identifies & overlays the faces using OpenCV.

See the code repo here.

I wrote several different Python scripts. One scrapes all the Facebook images you (or your friend) are tagged in. Once you have those images, you can run another script to identify the face and then overlay the faces on top of each other.

I used a few different Python packages and models, including OpenCV for facial recognition/warping and dlib for overlapping the images.  Read detailed instructions here (many thanks to Leon for his helpful workshop).

Here are some examples for me and my sisters:

I wrote another Python script that scrapes all the profile pictures from your LinkedIn connections. I was able to scrape the first 40 connections and then ran those images through the facemash script. This is what my average LinkedIn connection looks like:

Experiment #3: Aristotle Search (for Twitter).

I’ve been thinking a lot lately about how we search for and filter information online, and the ways in which Twitter and Google, for instance, make decisions for you about what’s most relevant. What if you wanted the ability to filter Twitter results according to a different set of criteria?

Inspired by Ted Hunt’s Socratic Search, I built a sister search engine called Aristotle Search that filters Twitter results according to Aristotle’s criteria for persuasive argument: logos (appeal to logic), pathos (appeal to emotion), and ethos (appeal to ethics).

The search engine is meant to be an exercise in speculative design that allows us to think about how a redesign of social platforms would change how we approach and engage with them. What if you approached Facebook with the intention to strengthen your relationship with family or reconnect with high school friends? What if you approached Google with a desire to challenge your own assumptions or seek clarity? (see: Socratic Search)

Rendering the familiar unfamiliar.


Clement Valla, The Universal Texture, 2012

I spoke with Taina Bucher and Surya Mattu today, who gave me excellent advice and direction on my thesis project.

Conversation #1: Be clear about your audience. 

In our conversation, Taina drew a distinction between the work that she has done and the work done by other researchers seeking to gauge digital literacy and algorithmic awareness (“do people know that they’re not seeing everything?”). They want to make the algorithm visible to its users. Taina, on the other hand, is more interested in user beliefs or expectations of how the algorithm system should perform. Her research aims to understand not only how people believe the Facebook algorithm functions, but also their normative conceptions about what algorithms should do.

Of course, it’s difficult to measure feelings or beliefs, so I’ve found myself asking: How can I observe user beliefs about how a platform should or does perform? How can I create an interaction that creates or observes those experiences? Many researchers take a qualitative approach – talk to study participants, interview them – but what will be my creative rupture?

Sterling Crispin, Data Masks, 2014

Conversation #2: Make it real, not hypothetical.

The conversation with Surya was really productive – he immediately understood what I was trying to achieve with this project and gave me 3-4 references of projects that had tried to achieve a similar effect, including the Chrome Extension alter, which allows you to scroll through a screenshot of someone else’s News Feed. We talked about his work on the Black Box Chrome Extension, which he said was his attempt to try to poke at the few aspects of the Facebook algorithm that are public but not immediately visible.

One piece of advice Surya gave me was to focus on the real data rather than the hypothetical. Sometimes the simplest intervention yields the most effective response. For instance, I told him about the research Taina Bucher has done to collect anecdotes from people who had had strange run-ins with the Facebook algorithm. He felt that she took a very powerful approach by focusing on the stories and images of real people, rather than theorize. He recommended that I start experimenting and see what resonates most with users.

Alter, Sena Partel, 2016

 

After talking to Taina and Surya today, I feel like I’m ready to move forward and experiment with Facebook, including downloading my own Facebook archive, building a Chrome Extension, building a 3D model of my face from tagged pictures, scraping Reddit discussions about how the Facebook algorithm works, and using personal data in an unexpected way.

I often think about Jenny Odell’s observation about her own work, that rather than take an new, unfamiliar technology and make something boring with it, she takes a familiar technology and renders it unfamiliar. I want to do the same with my thesis project.