Keaton Reeder

Since my last post, Michael Jumper and I have been collaborating on our experimental typography project. The process has been to link a midi sampler between an audio program called Ableton, and a “VJ” (video jockey) program. This is done by linking pads on the sampler to sound files in the audio program, and linking the same pads to the video program so that when a pad is pushed, it triggers both audio and video files.

 

(above) Ableton, audio program with sounds loaded into sampler.

close up of audio files assigned to pads in Ableton.

screenshot of GrandVJ, showing a video file linked to a midi note.

It’s been difficult to document both the audio and video being triggered at once because I haven’t been able to find a program that records what’s happening on the screen along with the audio — I’ve tried iShowU, but after tinkering around with the audio settings I was still not able to get the audio to record. Due to this, below is a video capture without sound:

the video shows video being triggered, along with audio (although the audio was not able to be captured). the visuals are being controlled by pressing a midi pad, and by turning a knob that has been assigned to parameters of an effect called “kaleidoscope” in GrandVJ. At this phase, the visuals and sounds were strictly formal, just to get a grasp of what I was doing with the technology, so any messaging or content was not really there.

This was the point at which Jumper and I decided to collaborate, because he was working with taking letter-forms cut out form wood and comparing the sounds of dropping them on a metal surface to the visual form of the sound-wave they made.

examples of Jumper’s wooden letterforms.

these letterforms were dropped on a metal surface and the hit was recorded:

the dropping sounds were visualized, and then we assigned the sounds of the letters dropping to the pads on the midi board along with images of the sound file (above) to be triggered at the same time, so for example when you press on a pad, the sound of the wooden Z dropping would play, and the image of the sound file would be simultaneously triggered on the screen.

After this phase of experimentation, it was brought up in critique that since we need to make a product for the next stage of the class, that we should create an interface that lays over the midi-board to both cover it up, and use the real estate to give the viewer information on how to interact with the interface.

a foam-core  mockup of a box that covers the midi sampler:

this diagram on the box shows what happens to the soundfile when the knob is tweaked; as you turn the knob, the soundfile slows down auditorially, and visually the letterform on the screen is triggered which stretches the typeface from condensed to extended, visually representing real time input of the user.

From this point we were encouraged to take more of a “skinning” approach to the cover for the sampler, rather than a clunky box. Moving forward, we are going to create an overlay for the interface, with typography and diagrammatic information printed on it to inform the user of how to interact with the “product”. We are leaning towards having this be an experiential artifact that would be interacted with in an exploratory setting rather than a product for sale.

My first experiment will be to create a pool of short audio clips and sounds. I will present these different sounds to a wide variety of people and ask them to write down which letterforms, words, or typefaces they associate with the sounds. they can either visualize the sound or describe it in words.

I will creating a track-list of sounds and name them by number. On a sheet of paper I’ll have each sound # listed, paired with an open space to draw/write a response.

I’m assuming I will come across a variety of responses for each sound depending on the person, but I’m hoping that some patterns will start to emerge. From these patterns I will decide how to use that information further.

I think that some control variables will be important here, one of which I can think of would be the established area for responses. Particularly attributes like the size of the letterforms they choose to write will be important data as to the visualization of the audio file. To do this, I may break up the space into increments of 10 or so, imagining that a letterform measuring one unit might represent a quiet or weak sound as opposed to larger forms might representing bolder ones.

I debated whether or not I should create another control variable by sin a set pool of letterforms with a variety of weights, sizes, and typeface styles, but I’m thinking it may  have the potential of biasing the data. If the results come back too all over the place, then maybe setting a control pool of responses will be appropriate.

What I hope to get from this experiment is a general comparison of typographic shapes and expressions that represent certain sounds. This could be particularly useful to myself and fellow designers in times when we are looking for ways to represent an experience or type of music.

The areas that I am interested in experimenting this semester in typography are the visual & auditory aspects of communication. I’m drawn to the idea of combining  newer technologies with typography, hoping that they will produce interesting results. I’m interested in incorporating motion and spatial qualities into my experimentation, hopefully to bring context to the messaging. As far as visuals go, pairing visual effects such as blur, visual noise, and fade might also add meaningful layers to the communication.

One tool I want to work with is my MIDI sampler. It is a set of 16 pads that have the ability to trigger programmed sounds when pressed. I want to explore incorporating triggering visuals into the sampler if this is possible. I’m thinking there could be a way of using recorded phonemes, words or phrases that would be triggered by the user pressing the pads. Knobs on the sampler allow for added programmed effects like reverb and delay.

One experiment I’m thinking about is recording the sounds of letter-forms and programming them into the sampler. I do not know if it is possible yet, but I’d also program specific visuals in the form of movie files or animations to the pads as well, so theoretically when pressing a certain pad it would play a sound file and also trigger a movie at the same time. One thing that samplers allow is repetition and duplication, which I think is at the heart of typography’s function. I think it would be interesting if these visuals and sound files could be layered in some way, and the sequence of which they are pressed will differ each time depending on what the user presses. I’d be interested in projecting these pairings into space, if it can add any meaning that would be great but for right now in a testing phase it would probably just be for atmosphere and effect before developing any further depth.

One similar experiment with potentially different results would be to use the marriage of flash code, audio and typographic visuals. It’s possible to use key commands with programing to prompt specific actions, and I think this could produce some interesting results as well.

Another technology that I am interested in experimenting with is voice recognition software. It has the ability to detect auditory patterns and form them into communicable words on the screen. Although it has for a long time been a technology prominent in the science fiction realm, this software is relatively new and we are even becoming familiar with it by now. When it first was introduced in the early 1990’s the success rate was low, and the software would often conjure up errors, having a success rate of about 15% but in recent years this technology has become much more efficient, and so I’m interested in experimenting with this modern technology and pairing it with typography since they both share communicative potential.