A filmmaking robot

This robot makes short films based on its visual experience. Its eyes travel about the city on buses while the body sits in a gallery. The eyes collect snippets of video, and transmit them to the body when their buses come within range of a Cafenet wireless internet node. The robot body splits the video into individual frames and analyses each one, obtaining twenty numbers reflecting the arrangement of colour, shape and detail within the frame. These numbers are treated as coordinates in a twenty dimensional space, in which distance is somewhat related to visual difference. For twelve hours a day the robot traces a zigzagging path through this space. This path passes through a series of images, which become a video sequence. Visitors to the gallery can see this video, called variously the robot’s “dream” or “stream of consciousness”. At the end of the day the robot looks over its days work and joins the best parts together as a finished film. The robot uses neural networks and heuristic rules to choose waypoints for its daily dream, but the finished film is mainly selected for the smoothness of its movement through the space. The robot will remember everything it sees until it has five million images in its mind, after which it will replace its least favourite images with new ones. In addition to getting images from the eyes, the robot creates false memories by combining and manipulating well-liked and overused images. These notes are incomplete.

You can see samples of his work on the page. It also gives a rundown if its aesthetic training, which gives some clue as to how it is making judgments.

The most meaningful part of these pieces is definitely the credits: “By a Filmaking Robot”. The choice of the indefinite article by the bot’s designers is somewhat unfortunate.

I’ll give full commentary after I am done teaching.

edit: I mirrored the films on my webspace, since the host above is rather slow. The september video is probably the most representative.

July 28
August 16
September 26

Link via Newscientist