How to Break Your Movies

hotline-bling-1038x576
datamoshed still from ‘Hotline Bling’

This class will be divided into two parts: a lecture & discussion about a variety of media practices that come under the broad heading of “glitching” followed by a demonstration of a few glitching techniques. If you’d like to follow along for the demonstration portion you’ll need to download a few things:

Some media files to experiment with

VLC  an open source video player

A hex editor like Hex Fiend (mac) or HxD (Windows).

Audacity an open source audio editing software

Goals

Learn more about how video technology work

Explore new critical perspectives on video technology

Explore some new creative tools, modes of media practice

Glitch

Watch excerpt from Anyplace  // Artist: Ryan Patrick Maguire

Artist James Connolly lists few elements of a working, provisional definition of glitch:

  • introduction of impersonal chance operations, or randomness into the creative process
  • a way of understanding and exploiting the materiality of analog & digital media
  • breaking the expected / intended use (subversion of the consumerist logic of these media)

A very brief non-history

Japanese Pottery / Wabi Sabi

931_01_t

Duchamp’s – 3 Standard Stoppages

standard-stoppages

Len Lye – Free Radicals

giphy9

John Cage – Prepared Piano Works

101004_r20053_p886

Nam June Paik‘s Magnet TV

https3a2f2fblogs-images-forbes-com2fjonathonkeats2ffiles2f20132f012fpaik_magnettv_lo

The Sandin Image Processor and Phil Morton‘s Colorful Colorado

Rosa Menkman‘s DCT:SYPHONING. The 1000000th interval (The 64th interval)

giphy6

GLI.TC/HTakeshi MurataHito Steyerl and more!

Why?

Storytelling

Experimental techniques always find their way into narrative and commercial filmmaking and can be effective storytelling tools.

Kanye West – Welcome To Heartbreak ft. Kid Cudi

giphy

Irma Vep (dir. Olivier Assayas)giphy2

Nike Commercial

giphy4

Ghost in the Shell

giphy3

Understand the technology

As an editor, cinematographer or director, a deep understanding of how video compression works will only help the decisions you make on set and in post. If your storage drives fail on set and you need to switch to a more compressed format, which will you choose?

block_partition

Critical awareness

We work in a medium that is defined by technologies which are often developed by multi-national corporations and governments. Digging into, breaking, re-appropriating these technologies gives us new ways of understanding the ways gender, class, and race are operative within the technologies we use, and allow us to think more critically about our relationship to the economic interests that structure the environment of media production.

Screen Shot 2018-04-11 at 9.45.09 PM
The MPEG LA office

For example, Kodak stocks and later the JPEG algorithm were engineered around white skin tones. Godard famously refused to use Kodak stock. Polaroid was sued for making cameras that would help the apartheid South African government surveil dark skin people. Consumers complained that HP webcams could not recognize faces with dark skin.  As we enter the age of machine learning and artificially intelligent cameras, researchers and activists like Joy Buolamwini show how algorithms can encode racial bias.

 

 

 

 

 

 

What is a codec?

codec: coder-decoder algorithm

A digital image is really just a sequence of 1’s and 0’s representing the brightness value of each pixel (or of the brightness of individual RGB channels in a color image). A codec is a program that can tell the computer how to interpret this data. How many 1’s and 0’s should be read as the data for one pixel? Is this image color or black and white? What are the dimensions of the image? Usually codecs also compress an image, making tradeoffs between data size and image quality, deciding which data is unnecessary and can be thrown away.

Screen Shot 2018-04-11 at 6.50.57 PM
A codec is a program that tells a computer how to interpret numbers as images

The photo site on an image sensor responds to light by sending out an electrical signal in proportion to the number of photons that hit it’s surface. This signal is represented in digital form – bigger number represents a brighter pixel. In an 8-bit black and white image, 0 would represent black and 256 would represent white, with the values in between representing shades of gray.

This information is of course represented in binary form  as 1’s and 0’s. An image’s “bit depth” refers to the amount of data available to encode the brightness value of a given pixel. A lower bit depth will result in fewer gradations of color and brightness

bit_depth_picture

Codecs like H.264 will use intra-frame compression to scan individual frames and try to average together similar information in small areas. One such technique is chroma subsampling. The human eye is less sensitive to small changes in color information than it is to changes in brightness. So many codecs will simply average together color information over different groups of pixels.

 

7781570_orig

 

Inter-frame compression compares differences among sequences of frames, and tries to get rid of redundant information.

I-Frames or “keyframes” are completely independent frames.

P-frames are predicted frames. P-frames holds only the changes in the image from the previous frame.

iframe

When you removing I-Frames the predictive frames look for a reference frame that isn’t there, so fill in information from earlier frames. You get this kind of effect.

giphy8

When you duplicating P-Frames, each new frame begins to recursively reference the changes from the previous frame, creating these streaking effects

giphy7

Manipulating I-frames and P-frames is beyond the scope of this class, but you can explore software like Avidemux, as explained in this tutorial, or more advanced command-line tools like FFMPEG and Tomato.

Techniques

Exercise #1 Hex-ing your files

A hex editor is a program that lets you view and edit the low level binary information that constitutes a digital file. The name ‘hex’ comes from ‘hexadecimal’: a standard representation for numbers that has 16 as its base 16, just as the familiar decimal representation has 10 as its base. Using a hex editor you can directly alter the data of the video file, byte by byte.

For this technique you can use this web based hex editor, or download a hex editor for windows or mac. You can also experiment with opening the digital file in a normal text editor like NotePad or TextEdit.

Download this dolphin.bmp file and try copy/pasting chunks of text, replacing certain sequences, or deleting portions all together. Remember to backup the original before you destroy it!

Download this cat.avi and try messing with it in the hex editor.

pro-tip: try to avoid modifying the beginning and end of a file – they usually contain header information, which if corrupted will make the image unreadable.

Exercise #2 using Audacity

Audacity® is free, open source, cross-platform audio software for multi-track recording and editing. You can very easily open import video files into Audacity and interpret them as sound information, then applying effects to them to mess with the underlying information.

After Audacity is installed and running, navigate to the File > Import > Raw Data menu

Select the video file you’d like to import (e.g. cat.avi)

Match your import settings to these:

Screen Shot 2018-04-02 at 3.16.08 PM

Next, select a portion of the video in the middle (to avoid corrupting the headers). Select and effect from the Effects dropdown menu. It’s best to start with small changes. Doing too much to the file can just break it entirely.

To export your file first select the entire track by navigating to Select > All.

Then navigate to File > Export > Export Audio

Make sure your export settings match the import settings (Raw header-less, A-law). Also make sure to give your file a .avi extension. It should open up in VLC after exporting.

 

Experiment with some other effects and see what kind of results they give you.

Some advanced tools: using the commandline

Tomato is a command line tool written in python that allows you to work precisely with I-frames and P-frames.

$ python tomato.py -i cat.avi -m bloom -c 80 -n 30 output.avi

$ python tomato.py -i cat.avi -m shuffle output.avi

FFMPEG is a very powerful free, open-source command line tool for handling video. If you work in post production its a great tool to have in your back pocket if you ever need to work with obscure codecs, do very specific tasks to large amounts of data. For example, FFMPEG can output the difference between two data streams:

$ ffmpeg -i cat_01.mp4 -i cat_02.mp4 -filter_complex “blend=c0_mode=difference” -c:v libx264 -crf 18 -c:a copy diff.mp4

 

Reading / Resources

GLITCH_READERROR_20111-v3BWs.pdf

Glitch && Human/Computer Interaction

Thoughts on Glitch[ART]v2.0

The Glitch Moment(um) – Rosa Menkman

Experimenting With Noise in Markerless Motion Capture

glitchet: resources

software

byebyte

tomato

avidemux

ffmpeg

Audacity

VLC

Hex Fiend (mac) or HxD (Windows).

hardware

lzx industries

 

contact:

Tom Rosenberg

rosenberg {dot} thomas [at] gmail.com

tomrosenberg.net

 

 

Advertisements