Abstract:

Nonlinear music is a contemporary concept whose development has been accelerated by the computer game industry (Collins, 2008), yet its roots lie in experimental practices of open form and process, such as those of John Cage and the Fluxus movement. Often emerging from human–machine interaction or data-driven processes, nonlinear music disrupts the conception of the musical work as a fixed, linearly unfolding form, reframing it as a relational and emergent process. While conceptually straightforward, its pedagogical implementation remains challenging.

This paper introduces a novel approach to teaching nonlinear music by integrating form, shape, viewpoint, spatial concepts, and compositional techniques. It is realized in a hybrid form with paper, a web camera, and browser-based software. Students engage in a cross-modal workshop to explore and experience the relationship between shape, space and music via variable, contingent, and interactive structures. This methodology draws inspiration from the form pedagogy established at Pratt Institute by Rowena Reed Kostellow and further developed through Karen Stone’s teaching (both discussed in Hannah, 2002), as well as from Akner-Koler’s Form & Formlessness (2007).

The workshops are structured around 1) a three-dimensional physical planar shape, 2) a musical composition, and 3) video recognition. Initially, students are provided with a rectangular piece of cardboard and instructed to carefully score three lines with a knife and ruler, one parallel, and the other two slightly diagonally across the shorter dimension, to enable sharp, precise folds. This creates a manipulable three-dimensional geometric form comprising four asymmetric rectangles. Each rectangle is then painted with a unique colour to facilitate video tracking.

Secondly, the musical material is introduced. The composition is organized in a two-dimensional representation - reminiscent of the unfolded cardboard and analogous to that of a digital audio workstation or musical score. It comprises multiple parallel stems arranged sequentially along the horizontal time axis. Each cardboard rectangle is algorithmically mapped to a corresponding segment of this visual music structure. The musical material is pre-composed to support real-time parametric control, including variable playback positions and the dynamic superimposition of multiple rectangular segments.

Thirdly, video recognition technology is employed to track each visible color-coded rectangle using a hand-held web camera. This provides participants with an identical perspective to their own vision, and allows for moving in and around the folded structure. The data streams derived from the video recognition system in MediaPipe control the playback within WebAudio in real-time through a WAXML script (Lindetorp, 2024).

In the presentation, we will show filmed documentation from two workshops and a live demonstration of the concept. The method offers new possibilities for teaching nonlinear music, particularly by highlighting the three-dimensional aspects that have largely been overlooked. Beyond its technical realization, the approach provides a platform for experiential and cross-modal learning, where students develop a systemic understanding of composition, space, and interaction. By translating abstract musical concepts into tangible spatial experience, the method supports creative exploration, reflective practice, and a deeper comprehension of relational processes.

Keywords:

Nonlinear music, Interactive composition, Spatial form, Cross-modal pedagogy, Human–machine interaction.