AI in movement and dance performance

Most conversations about generative AI in the arts are about images and text. This is a primer on the part that gets far less attention: what happens when artificial intelligence is used inside movement and dance performance — not to illustrate it afterwards, but to make sound and shape the piece while it is happening.

Written by the Movement, Music & Machines lab — a mathematician and musician, a dancer, and a music producer turned technology founder, working out of Stockholm.

What AI in dance actually means

"AI in dance" is used for at least five different things, and they have very little in common technically:

1. Movement-driven sound. A camera or sensor tracks a dancer, and the resulting data drives a synthesiser or a generative audio model. The dancer becomes the musician.
2. Real-time generative music. A model composes or continues music during the performance instead of playing back a finished track.
3. Choreographic assistance. Models trained on motion data propose phrases, variations or transitions that a choreographer can accept, reject or distort.
4. Reactive visuals and lighting. The same movement data drives projection, light and stage state, so the visual layer follows the body rather than a cue sheet.
5. Documentation and analysis. Pose estimation applied to rehearsal footage, used for notation, feedback or archiving.

Our work sits mostly in the first two categories, and pushes on the fourth. That is a deliberate choice: it is the area where the artistic hierarchy actually changes.

Movement as the prompt

Generative AI is usually prompted with text. A prompt, though, is just a conditioning signal — and a body produces an enormous amount of signal: joint positions, velocity, acceleration, weight shift, direction, effort, stillness.

Treating movement as the prompt means mapping those signals onto the controls of a sound system: pitch, timbre, density, harmony, intensity, silence. Done well, it does not feel like operating software. It feels like playing an instrument that happens to be the whole body.

This is also where the interesting artistic problem lives. A literal mapping — arm up, pitch up — is legible but quickly boring. A mapping that is too abstract stops feeling causal, and the audience loses the connection between what they see and what they hear. The craft is in the middle.

The instrument we are building

The project our lab keeps returning to is a new kind of instrument that uses the body as the controller and a generative AI model as the sound generator. It was demonstrated at our January 2024 workshop in Stockholm, and the recording is on our YouTube channel.

The questions we keep testing in front of an audience are simple to state and hard to answer: can movement control sound convincingly? Is generative AI fast enough to create music in real time? And can the hierarchy between music, dance and lighting be challenged with the help of AI?

Why music and dance lag behind visual art

Generative AI is alarmingly underdeveloped in music and dance, missing out on vast opportunities. While the visual arts have seen a surge in AI-generated content, music and dance have been left behind. There are reasons for the gap:

• They are time-based. An image can be judged in an instant; a phrase of music or movement only exists over time, which makes both generation and evaluation harder.
• The data is thin. There is no equivalent of the web's image archive for annotated dance, and motion capture datasets are small, narrow and expensive.
• Dance is embodied. Weight, breath, risk and physical presence are not captured by joint coordinates, so a model sees a stick figure where an audience sees a person.
• Latency is unforgiving. A late image is fine. A late note is a mistake, audible to everyone in the room.

The hard problems

If you are trying to use AI in a live performance, these are the constraints that will shape your piece more than any aesthetic choice:

• Latency. Anything above a few tens of milliseconds between movement and sound breaks the feeling of causation.
• Controllability. A model that produces beautiful output you cannot steer is a recording, not an instrument.
• Repeatability. Performers need to be able to rehearse. Pure randomness is not a collaborator.
• Legibility. The audience has to be able to perceive that the dancer is causing the sound, or the whole premise collapses.
• Authorship. When a model generates the music, who is the composer? This is a practical question about credit and rights, not only a philosophical one.

How to start experimenting

You do not need a research budget to try this. A workable first setup is a camera, a pose estimation library, a mapping layer and a sound engine:

1. Capture movement. Real-time pose estimation from a single webcam is good enough to get joint positions; wearable sensors give you cleaner acceleration data.
2. Derive features. Do not send raw coordinates. Compute things a dancer recognises — speed, expansion, weight shift, stillness — and use those.
3. Map deliberately. Send those features over MIDI or OSC into whichever sound environment you already know.
4. Generate sound. Start with a synthesiser you can predict, then swap in a generative model once the mapping feels musical.
5. Test with a dancer, early. Every assumption an engineer makes about movement gets corrected in the first ten minutes of rehearsal.

This is exactly the format of our workshops and hackathons: mixed groups of dancers, musicians and software engineers, building something rough enough to be tested in the room the same evening. You do not need to be a programmer to take part.

Questions and answers

See it in person

We publish everything we can: recordings of past workshops are on YouTube, past and upcoming sessions are listed on our events page, and the people behind the lab are Nils Kakoseos, Mira Buus and Mark Schellhas.

If you want an invite to the next workshop, hackathon or live demo, join the waiting list. If you want to collaborate, email hello@movementmusicmachines.com.