Being that Runway ML is a platform that I have no prior knowledge of, I am somewhat lost to the full spectrum of what it has to offer. From my understanding, Runway ML is a relatively new platform where models can be applied to exisiting forms of art to create different forms of synthetic media and push the boundaries of storytelling (shoutout to ITP alum Cristóbal Valenzuela). Although I may currently be a novice, I am looking forward to the experimentation process soon to be embarked on and concepts to be learned that will help with stories I hope to tell. Let’s take a look at some models that may help in my ML journey.

DeepDream

The DeepDream model is one that immediately grabbed my attention while navigating the Runway interface. In particular, the phrase “hallucinate patterns within “ is what sparked my intrigue, prompting me to think as to what that may look like. According to the site, “DeepDream is a computer vision program created by Google engineer Alexander Mordvintsev which uses a convolutional neural network to find and enhance patterns in images via algorithmic pareidolin.” Looking through the gallery, I saw a work of art, presumably from the Renaissance, that can be used as inspiration for a concept I would like to experiment with. A piece of art from the Renaissance would be a great concept for this model because most paintings from that era inlcude nature and individualism, which can be adequate for that hallucinogenic effect.

im2txt

To be transparent, this is probably the most basic of the three models that I chose but I believe could be the most interesting to dissect. The im2txt model is exactly what the abbreviated title suggests, generating sentence descriptions of images. Described as “a deep convolutional neural network”, the image encoder is widely used for various image tasks and object recognition. Previously mentioning that this may be the most interesting model to decipher, I would like to explore the capabilities and limitations of this model through the use of black and white photos, primarily photos that depict social commentary. This concept will allow for a fun experimenation as it will test limitations and imagine possible extensions that can be added to the model.

First Order Motion Model

Last but not least, the first order motion model is a model that can in short, bring images to life. This image animation model uses a self-supervised formulation that decouples appearances of objects within an image and motion information, from a pre-defined set of videos. A potential concept for this model are self-portraits that show facial expressions that convey different emotions. Similarly to the im2txt, I would like to see how extensible this model is. How does animation change from varying portraits of the same object? Does it convey tone, expression and emotion? These are all questions that I can’t wait to be answered!