Combining MIDI and Motion: Unlocking the power of music technology in film with Max Halstead
Max Halstead has worked in the film industry for 18 years, across multiple departments and as a crew member on close to 500 productions. His work spans advertising, music and film for brands such as Netflix, Amazon, Nike, Prada, John Lewis, Coca-Cola, and Pepsi, and artists such as Gorillaz and John Hopkins.
Now working as a cinematographer and motion control operator, Max tells us about his career and how his passion for music and technology helped him solve many technical problems for film in innovative ways...
Where It All Started
Before I worked in film, I was (and still am) into music production and DJing. I've always been fascinated with old synths and hardware, and all the ways it's possible to integrate old drum machines and synthesisers with new software and hardware.
When I started out in the film industry, I was lucky enough to get a job at a small stop motion animation studio called Clapham Road Studios in South London. Since animation is a slow process, which often stretches budgets thin, it gave me the opportunity to help out across various departments and learn how they worked.
I started off working as an electrician, then a gaffer. Later on, I learned motion control and found there was a lot of similarity with the integration of software and hardware that I’d experienced from working with music technology.
Motion control is, simply, the control of movement, whether it’s moving cameras, lights, set pieces, puppets, etc. Motion is created and controlled using anything from motors or motorised sliders with belts or worm screws, to cranes that carry cameras. Events like practical effects need to be accurately controlled, too, and motion control plays a key role here.
Discovering MIDI’s Motion Control Applications
Early on, I’d realised there was a potential application for MIDI in a motion control setting. I’d been looking into DMX (a digital language that controls lighting) and figuring out how the two might overlap.
Both MIDI and DMX run on timecode, making them interoperable, and I realised that if you were to use music software as the system of control, you could send MIDI information to a lighting desk to trigger lighting cues.
And if it was possible to take a MIDI device and make each note a simple switch, then it would be possible to have precise control over anything electrical by simply switching things on and off with a MIDI note, either recorded or in real time.
Building a Custom MIDI-to-Relay Device
With some research, I built a box that did exactly that. It housed 4 groups of 16 channels. With the MIDI input to these channels, each note could switch a 10-amp relay, which could be used to switch, for example, a group of 12v solenoids, another group of mains voltage lights or motors, and also to tell a motion control system to run its program.
Anything electrical could be switched on or off with millisecond precision, giving control accurate enough for high-speed photography. It also provided controlled timing of anything to the beat of a track or to the rhythm of any instrument within a track.
Testing Out MIDI Triggering on a Google Job
A job came up for Google, where I was asked to design a mechanical system for controlling musical instruments, a bit like a robot orchestra.
By this point, I'd built a bit of a reputation for problem-solving, specifically on jobs which required the triggering and programming of complex or elaborate manoeuvres, and building custom rigs to handle the movement.
I knew that I could use recorded or real-time MIDI messages to switch any circuit (and trigger an effect) for the duration the note is held down.
There’s a huge range of MIDI instruments available, too, which you can program separately. So, it’s possible to use drum triggers to switch on lights in real time, or a keyboard to record or playback a MIDI sequence of actuator movements. The range of applications is vast.
For the Google job, the idea was to use solenoids (electromagnets with a small moving arm similar to the striker in an old, analogue fire alarm) to move beaters that would play a xylophone and strike the drums, cueing the music that would introduce the spot.
Although the job was ultimately cancelled, I ended up with this fantastic device that I'd go on to use on countless jobs over the years, including my own.
Triggering Chain Reactions for Pepsi
I used the MIDI-to-relay technology on a Pepsi advert to overcome the immense jeopardy of a one-take action shot, where 500 ping pong balls atop 500 mousetraps would be triggered in a chain reaction.
The traps took 5 to 6 hours to set up. There would be no time for a second take. Predictable, highly accurate timing to synchronise practical effects with these split-second events would be crucial.
For this, I sent MIDI from music software to a lighting desk to trigger multiple lighting cues that tracked the first ping pong ball’s movement.
The benefit of using MIDI was that the timeline for MIDI notes is in milliseconds and seconds, which makes it useful for frame rates above 25 fps — ideal for high-speed photography.
The Benefits of MIDI in Post Production
For their advert, Argos wanted balloons of varying sizes to pop spontaneously and reveal their products. Each note in the music track would correspond to a popped balloon.
This involved triggering linear actuators with scalpel blades attached to them in order to pop the balloons independently as the camera passed by. We did different passes with the camera, and they were composited together in post-production.
MIDI made it incredibly easy to edit together, using different frame rates on the camera and timecode to keep the rig and the balloon pops all in sync.
Programming Motion Control to the Beat of Live Music on Set
In 2016, I worked on a job for Coca-Cola called 'Happiness is Movement'. They wanted it to feature an automaton; an intricate piece of engineering which moves automatically using gears, levers and motors.
The automaton was enormous. It was engineered to show a rapid sequence of milestones in a couple’s life story in reverse, while the characters within it danced to music throughout.
My job was to program the motors that drove each automata and make sure they all danced in sync with the music.
It was the extensive programming of this complex machine to a live click track that made me realise there was possibly more to unlock from the integration of MIDI and motion control.
Full Music and Motion Control with Cartoon Network
An opportunity came along where I could finally test the limits of what was possible from this technology.
Cartoon Network were looking for a cool ident. An eye-catching, 15 to 20-second image piece was the perfect format for showcasing a concept that was only possible with the operating power of the MIDI-to-relay box I’d built.
The concept I pitched was for a mechanical kinetic sculpture, called The Machine.
I was inspired by David Batchelor’s Magic Hour, artwork I’d seen in 2013 at the Hayward Art Gallery Light Exhibition, and designed a sculpture comprising stacked cubes of acrylic which contained lights and moving mechanical elements.
There was a mechanical hand that emerged from a box and switched a light from green to red; a golf putter with a mechanical gold ball delivery system; a record player with a tone arm that would bounce using a solenoid; a cuckoo clock; a conveyor belt delivering random Cartoon Network-inspired objects; and many other mechanical devices in the design. At the very top was the Cartoon Network logo with a helicopter orbiting it. There was a lot going on.
When I pitched this job, I decided I was going to write the music for it. That way, with the MIDI information triggering the whole device, it would be completely controllable.
Every rhythmic and percussive element in this track would be represented physically by something electronic in The Machine.
Because I had the music, I worked out that by putting the track into Logic music software, taking up each MIDI track (like the hi-hats), I could then take that sequence of MIDI notes and connect it to a mechanism in the machine.
The workflow involved programming using the MIDI-to-relay device from the beginning, offsetting everything to account for the delays in things physically moving.
Using motion control to operate the mechanical elements and the camera rig, I could then program them to keep time with the music. From there, it could then be triggered alongside the lights and solenoids, all via MIDI.
It was a full circle moment and the culmination of years of technical problem-solving to finally use my MIDI-to-relay technology to control all motion: camera tech, lights, rigging and all other elements of the set. It was one of the highlights of my career.
Far-reaching Possibilities of Music Technology for Film
Throughout my career, I’ve always looked at ways music technology can open doors in the film world.
I did a project for the BBC with compositor and graphic designer Will Studd, where we used Ableton to trigger clips using VJ software called Resolume, which is a powerful tool for live visuals and allows for projection mapping and pixel mapping, as well as a huge range of effects.
At Clapham Road, I’ve experimented combining Resolume’s capabilities with Dragonframe, the industry standard animation software. While timelapsing Resolume to display graphics on a bank of LED pixel tubes, it became apparent there was a way to get the two pieces of software to communicate.
Talking with the developers of both bits of software, an OSC output option was added to Dragonframe, and suddenly it was possible to get the timelines in both bits of software to link together, unlocking projection mapping and pixel mapping within stopframe animation.
When two things that were never meant to work together do, it opens my eyes to what’s possible for the industry. Some of the most rewarding parts of my job have been discovering these systems.
If it wasn't for my love of music and its technology, I might never have found out.