Scanning Spaces: Paradigms for Spatial Sonification and Synthesis
Ryan Michael McGee
PhD Disseration Defense
June 9th, 2015
www.spatialmodulation.com
Abstract
In 1962 Karlheinz Stockhausen's "Concept of Unity in Electronic Music" introduced a connection between the parameters of intensity, duration, pitch, and timbre using an accelerating pulse train. In 1973 John Chowning discovered that complex audio spectra could be synthesized by increasing vibrato rates past 20Hz. In both cases the notion of acceleration to produce timbre was critical to discovery. Although both composers also utilized sound spatialization in their works, spatial parameters were not unified with their synthesis techniques. This dissertation examines software studies and multimedia works involving the use of spatial and visual data to produce complex sound spectra. The culmination of these experiments, Spatial Modulation Synthesis, is introduced as a novel, mathematical control paradigm for audio-visual synthesis, providing unified control of spatialization, timbre, and visual form using high-speed sound trajectories.
The unique, visual sonification and spatialization rendering paradigms of this dissertation necessitated the development of an original audio-sample-rate graphics rendering implementation, which, unlike typical multimedia frameworks, provides an exchange of audio-visual data without downsampling or interpolation.
Visual shape more important than color, color enhances shape
Problem Statement
Stockhausen and Chowning both created a continuum between acceleration and timbre and both made extensive use of sound spatialization,
but technical limitations left spatial parameters separate from timbral control paradigms.
However, physical principles such as Doppler shift and distance-based gain-attenuation provide an intrinsic, mathematical connection between moving sound sources and FM/AM.
The symmetrical Doppler shift approximation formula ubiquitous in the field of digital audio is physically inaccurate at high-speeds over approximately 100 m/s (225 mph).
While this approximation is sufficient for existing Doppler-based effects that do not require such high-speed source motion, it prevents a fully realized connection between spatialization and high-index FM timbres.
Current multimedia software frameworks provide inadequate control rates for spatialization trajectories which limits the ability of spatial modulation to achieve high frequencies of modulation.
Sample rate spatialization choreography is necessary to achieve audio-rate frequency and amplitude modulation from simulated spatial motion.
Likewise, graphics computation at audio-sample rate is necessary to accurately visualize spatial modulation sound trajectories at full resolution.
Spatial Modulation Synthesis overcomes previous technical obstacles for accurate simulation of high-speed sound sources that can unify spatial, timbral, and visual composition.
Timeline
2009: W.A.N.T.S
2010 - 2012: Sound Element Spatializer
2011: Skate 1.0
2011: SenSynth
2011 - 2013: Image Sonification Studies
2011: No Heritage
2012 - 2014: Seismic Sonification Studies
2014: Kinetic
2014 - 2015: Spatial Modulation Synthesis and Kinetic V2
W.A.N.T.S. (2009)
High-speed Fibonacci spirals of sound
First study producing timbral changes from spatialization (AM and panning, no Doppler)
Tedious, non-real time control and rendering via MATLAB
Spatial Amplitude Modulation via Panning
Faster Spirals Leading to Granulation
Sound Element Spatializer (2010-12)
Sound Element Spatializer (2010-12)
Real-time spatialization of an arbitrary number of simultaneous live or recorded sound sources over an arbitrary loudspeaker arrangement
Dynamic selection between multiple panning algorithms with distance cue including Doppler shift, gain attenuation, and air absorption
High-speed movement of sound sources without "zipper" noise artifacts
Flexible, precise control of sound trajectories using the OSC protocol
Robust and easy to integrate with DAW software or any audio application
Cross-platform standalone application in C++ (easy to use, no Max programming required)
Provide a link between visual and sonic worlds. Visual artists may provide trajectories for sound and musicians may provide sound for trajectories. Should encourage collaboration and exploration for both visual and sound artists
Work Exploring Acceleration of Spatial Trajectories and Spatial Data
4 Conference Proceedings Publications
3 Original Compositions with Public Performances
3 Software Contributions to Long-term Art Installations
2 Mobile Sonification/Synthesis Apps
Unified Audio-Visual Rendering Paradigm
Sample rate trajectory control contributions to open source AlloSytem C++ suite
Spatial Modulation Synthesis
Mathematical unity of space, timbre, and visual form based on physical first
principles of Doppler shift and distance-based gain attenuation using spatial controls of velocity and bounds
Physically Accurate Doppler rendering implementation
Coupling with dynamic anti-aliasing filter allows SM to achieve high FM indicies by moving up to the SOS
Trajectory-timbre relationships produce an intrinsic and novel visual-timbre taxonomy
2 Public Exhibitions in the AlloSphere and 1 conference publication
Original software plug-in implementation that exists as both effect and audio-visual instrument
Future Work
Additional UI Features for Plug-in (listener position, orbit center, index ADSR)
Additional complex motion trajectories (source and listener)
Additional Distance Cues
Smaller Bounds, Higher Sampling Rates
Beyond SOS
Audience as an Audio-visual Canvas
Thank You!
Committee: Dr. JoAnn Kuchera-Morin, Dr. Curtis Roads, George Legrady
Professors/Mentors: Matthew Wright, Andres Cabrera, Stephen Travis Pope, Clarence Barlow, Marcos Novak
MAT Community: Reza Ali, Yuan-Yi Fan, Lance Putnam, Graham Wakefield, Karl Yerkes, Pablo Colapinto, Dennis Adderton, Myles Sciotto
Collaborators: Daniel Ashbrook, Johannes Girardoni, D.V. Rogers, Damon Seeley, Jatila van der Veen, Sean White