Editorial⎜About immersive audio, a look back at DOK Exchange XR 2024

In 1989, speaking of virtual reality, Jonathan Carey wrote that “spectacular power cannot be reduced to an optical model but is inseparable from a larger organization of perceptual consumption…The full coincidence of sound with image, of voice with figure, not only was a crucial new way of organizing space, time and narrative, but it instituted a more commanding authority over the observer, enforcing a new kind of attention.”

In the last decades, the visual often dominates the conversations around virtual reality, but other senses play a big role in emplacing the experiencer. John Berger might have pushed the ways of seeing, but unconsciously we also we see with our ears.

Cover: Spatial Sound panel, moderated by Dr. Markus Zaunschirm, with Anan Fries, Shervin Saremi and Oliver Kadel 📸 Susann Bargas Gomez

In XR experiences, we are trying to get as close to putting people in a specific place or situation as possible. This means either directly recording a scenario/location, or layers the recordings to rebuild a soundscape that reflects “what we hear”. With the emergence of spatial sound recording and other technologies, we now have the ability to capture reality’s sound more closely and to compose soundscapes that are richer, deeper and more complex. In virtual reality and immersive installations, this more advanced audio experience is what can truly make someone feel connected.

Audio is no longer an afterthought, but something to be built into all stages of development and production, and this year’s DOK Exchange XR conference put spatial audio front and center. Here’s a chat with coordinator Weronika Lewandowska about the inspiration, and a reflection on some of the discussions.

Weronika Lewandowska – It was many things that inspired us. First, for me, it’s the “grain of voice” concept of Roland Barthe, referring to the physical and sensual quality of voice beyond its somatic meaning. It highlights how the voice, how sound as a product of our body carries its presence through its texture and vibration. The discourse around XR is often dominated by the visual aspect. But of course, there is sound, which is for me very crucial in this act of creation. We can really hear the physical aspect of reality.

Sound also is like the first virtual reality. It opens our imagination. It gives us hints about danger around us. Sound connects what is physical and what is virtual. And I think now we are trying to find how to connect virtual experience with the world around us, with the physical experience of physical space. There’s a shift in exhibitions so there is not only XR, not only headsets, but more installations, more big, immersive projections. So we are trying to find how we bring together the physical and virtual experience, and sound has this way of connecting those two things.

That’s the philosophical part. But our team had a lot of conversations about ‘what is the future of immersive storytelling?’ How this will develop and what is important, what have many people missed? At DOK Leipzig we would like to support creators, producers looking for new ways to tell stories and bring the ambitious projects to this technology realm. We are interested in how we can develop audio storytelling, and that’s why we decided to show this aspect, immersive spatial audio creation, production, distribution.

Distribution is also very important, because sound is easier to distribute with the technology we have right now. So I think it’s the right moment to have this reflection about not only XR and sound, but also what we can learn from analog, performative forms, physical forms, which develop their own way of storytelling.

The Role Of Sound

K.C. – For a long time, in VR, people were using sound to direct where the viewer would look or how the person would experience the piece. In VR or XR work, are people starting to see it in a different way?

W.L. – Of course, sound plays a huge role in how to direct your attention. In a VR headset, some people only look in one direction, they don’t really understand that it’s a spatial experience. So sound can open you to experience that spatially, and to experience space around you – it can really bring your awareness about your body, that you are not inside your own dream, but you are there, present. XR without sound would be very flat for me. It really is what gives you this embodiment, this embodied soundscape or landscape. And it can be very important for telling stories, for bringing this human aspect with the voice.

W.L. – Audio artists use space like how this spatial audio experience in real life is created, how our perception is created, how we interact with space. It’s also about proxemic collaboration, social interactions. There is a lot of aspects hidden in this spatial audio. In our podcast with [conference speakers] Aili Niimura and Ana Monte and Oliver Kadel, we talk about how in the metaverse, we can shape better collaboration between people just by creating this spatial audio experience, so we really feel this proximity, how far we from other people.

W.L. – There are many things to do with sound in XR , things we didn’t think about two, three years ago. And now spatial audio is a huge topic for developers and creators. Before, we could only experience XR by ourselves. Now we can share this experience with others. And when we have more people inside a virtual room, then this audio communication becomes a really important aspect.

K.C. – When we talk about audio and soundtrack, there’s two ways you can go. Recording what you have, and then recording, but recreating the mirror of what you have. So audio has more of a flexibility to recreate reality in a way that’s less judged, and also, as you said, brings you closer to it with the emotions.

W.L. – Yes, I think it’s this grain of the voice concept. You feel more present, and it brings you the emotional aspect of other people’s experiences, real experiences. Voice is like the act of reality, it’s really bringing you some kind of truth about life. There are some XR pieces where creators bring real audio because it’s easier to record spatial audio wherever you are. And sometimes audio is implemented from real experiences and the rest is recreated.

Unpacking Immersive Audio – From Psychology To Tools

K.C. – So, the conference… two full days talking about sound and sound design in the context of an XR industry…

W.L. – We want to start with what exactly immersive audio for XR is, and then go into the psychology of sound and tools, how to create it. We have a panel discussion about how immersive audio is implemented in different forms of art, so we will compare different disciplines of creation. We discuss how to work with sound engineers, sound designers. If you are a creator, what this brings to your work; if you, from the beginning, are focused on this audio layer of your experience. New technologies bring even more on how to shape this spatial sound experience, and you can experience it at your home, on your walk. There are a lot of possibilities for that.

W.L. – We are also presenting a performance art aspect. We have some performers who are using, in real time, this game engine to create a virtual world, and it’s projected behind performers who are performing on stage and at the same time in virtual reality. We also have some XR case studies, knowledge from domes and how to shape audio for physical space, how to mix real audio with immersive-created audio in the physical world, so many different perspectives showing us what will influence the direction of XR media and storytelling development.

Dok Exchange Conference – What Can Spatial Sound Offer

The DOX Exchange conference delivered on Lewandowska and her team’s desire to illustrate how important sound and audio are to quality and successful immersive experiences. Both opening speakers, Oliver Kadel, sound editor, designer and engineer, and Dr. Markus Zaunschirm, an expert in spatial and immersive audio, referenced Hyunkook Lee’s concept model of immersive experience. To deliver the best possible immersive experience you need to find the sweet spot between three things: physical presence (I am part of the environment), social or self-presence (my actions have influence) and involvement (I am solving the task).

Kadel equated psychoacoustics with a german potato salad metaphor: very few ingredients, all in the seasoning. In XR, the ingredients being input stimuli plus signal reception (the ears, whole body shapes what you receive) plus brain analysis – everything affects and influences how you perceive sound.

There’s an interaural time difference (sound arrives to one side faster than the other one), interaural intensity difference, sound in front of you, sound behind you, in additional to visual cues. In real life, we are “head tracking” all the time, which also affects how sound comes to us. There’s a greater sense of presence, awareness of objects, distance and location. Our natural 3D listening detects loudness, reverberation, etc… which tells us about the distance of sound.

Binaural and spatial sound can deliver a sense of space and presence, enhance the object positions and distance, enhance the sensation of disbelief, and offer a more realistic way to communicate, yielding more impactful experiences.

But to help audio mimic real life, you have to model many things to get it right, as Zaunschirm explained. “You know how a room should be behaving…The acoustics of what you’re seeing and hearing need to match to give you a more holistic experience. It’s not about static content, it’s dynamic. In real life, everything is always moving. The acoustics are always changing, the distance is always changing.”

Sound can be used to recreate that reality or to create a dream world. “Sometimes we try to recreate a world that we just record, whether in video or a game engine, that we want to be familiar and real,” offered musician and audio-engineer Shervin Saremi. “Sometimes we want to create a world that doesn’t necessarily function like the physical world as we know it.”

Spatial audio is the key to delivering both emotions and information. For Ana Monte, spatial audio expert and co-founder and sound designer at DELTA Soundworks, spatial sound gives a sense of presence; reduces cognitive load; enhances tension and fear, without interrupting the immersion with cues or instructions; creates emotional closeness (or isolation); and overall improves the user experience.

“Spatial audio is the concept where the space where the sound occurs plays a role in the aesthetics of what we’re doing. Think of the space, and how you can use the space to tell the story.” It also gives you more bang for your buck in terms of the experience outcome. But you have to start with the best possible sound in collection and production.

No Longer The Stepkid, Audio Is In The Conversation From The Start

To create an effective experience, audio has to be part of the development from the beginning; it’s an integral part of the project, transdisciplinary artist Anan Fries stressed. “It’s perceptible in the work…If I say I want to create an immersive experience, then the sound is really the bridge for me personally. The sound is really what ties the virtual space to the physical space.”

Luckily, audio has become part of the conversation. You no longer have to ask “what are we going to do about audio?” for XR projects, Kadel notes.

The future is here. Spatial audio is a standard. You don’t need to convince people to do it. But there’s a lot to improve.

He sees challenges such as fragmentation between formats, constantly changing technologies, feasibility both in terms of budget and set up times, who has access to the technologies (especially expensive hardware) and the knowledge. There are still barriers to accessibility and understanding, not just for makers but also for technologists, curators and educators.

Exhibition remains a major challenge for immersive audio experiences, and curators need to grapple with how to use or reconstruct the spaces they have to deliver the proper audio environment for audiences. “None of these spaces were designed with sound in mind,” noted Luke Kemp, Head of Creative Programme for Barbican Immersive. They’re working through how to make sound experiences work in the Barbican and testing different non-gallery spaces. They’ve created transition zones with speaker sound systems, specific acoustic zones and sound blocking systems, so there’s a blended sonic experience that pulls people from one moment to the next of the exhibition.

And what of the ethics of sound immersion? Fries posed the question “Why do we want to augment immersion so much?” Kadel noted that “life is immersive, and the way we navigate the world is spatially.” But whenever we recreate that digitally, it’s not ”good enough”. The technology is not quite there for capturing it, and our expectations are so high, so we have a tendency to add more, striving for hyper-real, to make something bigger than life. Fries questions the ethics of pushing for immersion to make people “feel” the dramatic experiences of real life.

You can catch the recordings of the XR conference on the DOK Leipzig website.