Music of Encounters - Pinakothek Der Moderne

For a period of five months between June 5 and November 9, 2025, the 25-meter-high rotunda at the Pinakothek der Moderne in Munich transformed into a real-time multi-channel polyphonic musical instrument that fills the space based on the presence of people and time of day. In collaboration with composer Rupert Huber and Ars Electronica Solutions, Music of Encounters brings people, music, and architecture into a constantly evolving dialogue. 28 speakers, placed on the ceiling and surrounding the rotunda, offer a panoramic experience of the music as it travels around in space.

The system I developed for this installation is divided into two separate parts/applications running on Linux and communicating through OSC. One part is dedicated to tracking people's presence inside the rotunda space, and the other is dedicated to the multi-channel audio playback system.

The tracking system works with a single PoE OAK-1 camera sensor located almost 30 meters high at the center of the rotunda ceiling. The distance from the ground coupled with the FOV of the sensor allowed for coverage of the whole area with a single sensor, which simplified the development process a bit. The whole rotunda space was divided into 72 separate polygonal tracking zones, and the presence of people in any of the zones would trigger the audio playback system through OSC. The division of the tracking space into polygon zones where detections could be associated was made with the Supervision Python library. For the actual detection, after evaluating a few different models of various flavours (e.g. YOLO architectures, RTMDet, YoloWorld, and OmDet-Turbo from open vocab), I went with RF-DETR for the speed and accuracy of results in this specific context plus the Apache licensing option.

The multi-channel audio playback system was written in two parts. One part is the configurator app, written in PyQt6, where I could set polygon zones directly on the camera feed or an image with the ability to assign to each zone various sample pools that would be active depending on the time of day. The output channel(s) of each of those samples, along with other parameters such as max amplitude and panning factors, could also be set directly from the configurator app. The audio samples, more than 300 in total, were short instrumental phrases covering a wide range of instruments ranging from viola to piano to the experimental Irvine instrument played by Rupert Huber himself or other musicians. As people walk through the 72 zones of the rotunda, the different audio layers overlap and mix, creating a unique soundscape at each moment in time.

The second part of the playback system is the real-time module that loads the exported configuration from the configurator and takes care of reacting to messages arriving from the tracking component. It is written in Python with Pyo, a great DSP Python module written in C that provides a lot of cool and useful tools for building complex playback and effect audio pipelines. It also has great support for multi-channel audio input/output through Jack, which provided us with low latency audio patching and helped in order to communicate with the Dante-enabled PCI audio interface that fed the 28 output channels.

More details about the project and the concept can be found here.

Video Credits: Ars Electronica Solutions