Category: ProgrammingAZ

Crystal gazing: A Twitter bot that uses computer vision to describe images.

Posted on by baricks

This semester I’ve been interrogating the concept of “algorithmic gaze” vis-a-vis available computer vision and machine learning tools. Specifically, I’m interested in how such algorithms describe and categorize images of people.

For this week’s assignment, we were to build a Twitter bot using JavaScript (and other tools we found useful – Node, RiTA, Clarifai, etc) that generates text on a regular schedule. I’ve already build a couple Twitter bots in the past using Python, including BYU Honor Code, UT Cities, and Song of Trump, but I had never built one in Javascript. For this project, I immediately knew I wanted to experiment with building a Twitter bot that uses image recognition to describe what it sees in an image.

To do so, I used the Clarifai API’s robust machine learning library to access an already-trained neural net that generates a list of keywords based on an image input. Any Twitter user can tweet an image at my Twitter bot and receive a reply that includes a description of what’s in the photo, along with a magick prediction for their future (hence the name, crystal gazing).

teapot-002

After pulling an array of keywords from the image using Clarifai, I then used Tracery to construct a grammar that included a waiting message, a collection of insights into the image using those keywords, and a pithy life prediction.

celebtech-05-cosby

I actually haven’t deployed the bot to a server just yet because I’m still ironing out some issues in the code – namely, asynchronous callbacks that are screwing with the order of how functions need to be fired – but you can still see how the bot works by checking it out on Twitter or Github. It’s still a work in progress, however.

You can see the Twitter bot here and find the full code here in my github repo. I also built a version of the bot for the browser, which you can play around with here.

Finnegans Wake-ify: An experiment in generative text.

Posted on by baricks

For this week’s assignment in Programming A to Z, I built a simple web application that allows the user to input a paragraph of text and a computer program outputs a text that sounds like the stream-of-consciouness writing style characteristic of James Joyce’s Finnegans Wake.

Test out the generator here.

Example output (from an email I’d written):

You scraggy riverrun kidding. I completely agree with you – Eve reason shore war: recirculation interesting to me war: swerve contradictions. commodius shore bay Fantasy, for commodius commodius song “Runaway” catches you off guard vicus it war: recirculation bay Howth Castle to Eve rest Environs commodius album. recirculation every “Runaway,” Howth Castle there scraggy 10 iterations Environs passen-core am isthmus Sir Tristram I scraggy he’s captured isthmus picture Environs Europe kind Environs self-aggrandizement to side penisolate artists become victim. The real question at stake Europe war: to wielderfight degree he war: self aware. vicus everything depends on penisolate doesn’t it? We can’t tolerate blind egoism. It has to be in some way self-referential or winking or ironic.

The web app was built in ReactJS + Webpack + Node.js and you can find the full code repo here.

To do some of the text manipulation, I used nlp_compromise, a node package that allows you to use Natural Language Processing (NLP) tools in Javascript. I was disappointed, however by how limited the tool was. Next week I plan to explore using the RiTA software toolkit with NLP.