ANCESTORS is a unique participatory journey six generations into the future, exploring how today’s choices shape tomorrow’s world. In a large-group experience, the piece uses AI and guides you by your smartphone to connect with others around you. Directed by Steye Hallema (THE IMAGINARY FRIEND) for the Smartphone Orchestra, it premiered at IDFA 2024 and was recently featured at Venice Immersive.
S.H. – As the creative director of the Smartphone Orchestra, I’m always looking for group experiences that makes sense. I was asked to brainstorm by the Finnish video installation artist, Hanna Haaslahti, who was working on capturing faces and face merging. I had this “wait a second” idea that you could make a face merge of two virtual people too and create a grandchild. We worked on this virtual family idea a while until we realized we should make our own versions based on our strengths. My intuition told me this is a really good Smartphone Orchestra idea, because if the whole audience is an ancestor of someone in the future, then there is a very natural dramaturgical reason to connect the whole group.
There was a certain point when I realized that if this is about the future, then in some way it needs to be activistic. But although I’m quite idealistic, I’m not really an activist.
I look for urgency in the work I make. I needed to find a voice in making a good point, which I eventually found. If it’s about the future, it cannot be about anything else other than taking responsibility about the future. So that’s the main message, but the underlying message is that we can only get there if we do it together.
If you create a bit of order and a nice setup, then random strangers can understand each other on common ground. I think that’s real message, and that took quite a while to figure out. I call it “cute activism”. Not being too strong and creating stuff that makes us belong together, even right-wing minded people… because, you know, we have to work it out together.
It’s about the future of the world, and you talk about flooding, climate change, but what I come away with is that feeling of facing the future together. But I think that’s absolutely activism. There’s so much in there, but it’s subtle and woven through. Some people would go too heavy and make it “About Climate Change!” Where is that balance is for you?
S.H. – I make things that are about the player. As the storyteller that I am, I need to be super subtle. What I referenced earlier, about finding my activistic voice? The first script that I wrote, and when we did the first test, a lot of feedback was that it was too leading, because I “told” too much and didn’t give people enough room. But it’s not about me, so I tried to get myself out as much as possible.
People who say “I’m using VR, so there’s going to be empathy” is completely besides the craft, which we actually all should learn. If you want to create empathy, it doesn’t matter what kind of medium, there’s craft in telling the story and creating it.
At the end, when everyone is standing in concentric circles, and you’re asking “Would you thank, forgive or condemn the generation of your grandparents?”, and then your parents, your children, etc…I have nothing to say bad about my mom and my grandmother, but people who are in their 20s are not going to thank their parents. And it’s that moment where you realize, oh yeah, I don’t have anything to really blame my parents for, but everybody’s going to blame my generation.
S.H. – That’s exactly the realization I’m trying to have you make.
You’re always thinking of the Smartphone Orchestra and what would be appropriate for it. Do you have the idea and the output, and then it’s just figuring out how that happens together? How does this work technologically?
S.H. – The real technological framework is synchronizing the phones, having created a framework that sends messages to the people at the right time. The Smartphone Orchestra is already almost 10 years old. It started as a musical idea: if you can synchronize phones and there’s a speaker in them, we can send musical parts to them that are different but fitting, to create spatial sounds. That’s how it started, and we explored that for a couple of years, until we realized the real promise of it is involving the audience, creating experiences because we can talk with everyone personally.
The real work is really not so much the technology, although what we built is really clever, but the dramaturgy, which is really 60 percent of it. What we’ve learned to do well is how we prompt, how we seduce you to come along. You might have noticed there’s a lot of silliness in it, but it’s really the silliness that makes everyone open up. And then there’s the peer pressure, the herd mentality that if everyone does it, most people will also do what the group is doing. So there are psychological, dramaturgical ways we figure it out to make sure it goes well.
I’m often a bit skeptic about XR works that try to create connections, because why would we need technology to create connections if we can just connect? Obviously, technology could facilitate something so you can connect in a different way, which can open up a new view. But often there’s too much work in the technology and not enough in the dramaturgy and the psychology of it.
Did you use any sociological or psychology studies? Because it seems you have things mapped out in a way that is really mindful of when people may lose interest, or when people want to be pushed…
S.H. – Of the meter and a half pile of books next to my bed, it’s probably 3/4 psychology books. I read a lot of them – behavioral psychology, group psychology – to understand. Because I guess as a creative director, my niche is to involve the player. And that’s what I really like doing, because I think this is where XR can make a difference. One of the challenges, if you want to involve the player, is how do you make sure that it fits everyone? So you have to have strategies that can work in certain ways.
So technologically, you have this framework that you built. You take a selfie at the beginning, and then the system feeds the prompts to the people. Is there any interaction after that, is your phone reading you?
S.H. – You take a selfie at the beginning, and we use that selfie through AI to create a face merge. Sometimes that works really well. Sometimes the AI is just bias as hell, and it creates white people out of black people. It’s just a horrible tool. It doesn’t give you what you need or want.
But after that, it’s just what happens in the room.
S.H. – With the thank, forgive or condemn point, I really wanted to know what people are putting in. Anecdotally, it’s what I suspected – that people actually often thank their grandparents, forgive and condemn becomes significantly bigger for parents, and then when we get to the children, a lot of people feel like they should condemn or forgive us. So I really want to know if that’s how that works, but nothing now is feeding back.
We build all the technology ourselves. We have Ableton, which is a digital audio workstation that is built in such a way that you can easily create a sequence so we control it from [a station], and then that runs into Max for Live, which then connects to the server and goes to the phone. There are all these little back doors that we build ourselves and can have control over.
If you’re going to build connective experiences, you also need to build in the knowledge of how to deal with glitches, interruptions and things. We also have a lot of non-technical strategies for these cases.
Do the audiences react differently when they’re coming from the XR space or from somewhere else?
S.H. – I would say the difference is not so much that XR people are different, it’s more about introvert or extrovert. We did it at South by Southwest. America has quite an extrovert culture; people are easy chatters. Then we came back, and did it in the east of Holland, where the saying goes: ‘speaking is silver, but not speaking is gold.’ And we’re there worried, thinking, “They are supposed to say something.” But in the end, they loved it.
The Achilles heel of ANCESTORS is language. For people that are really fluent with English. It’s easy. But later in the week [at Venice], we had more groups that were almost only Italian and a lot of people really had trouble with it. It’s more fragile, although they still love it, and it holds up.
We’re working on that – we have a Dutch version now, and soon a French version. And now we’ve built a system that we can have many more languages. Because I would really like to see this work outside of the art world. It’s really for everyone. I want the UN or some sort of NGO to put it out there.
It’s great for a mixer, a getting-to-know-you, opening thing for any kind of conference or meeting, because you lose your inhibitions and relating to someone at a different level than you normally might.
S.H. – We’re now in conversation with Cornell University, which has quite a big teambuilding facility, and they’re thinking about licensing it. That is really interesting. I’m proving to the whole XR community that there’s actually business models that work. Already so many people can do it at the same time, and then if you can license it in these kind of contexts, we could probably do without subsidy at a certain point.
Until now, we need to operate it ourselves. But then we would train staff and build a more robust setup where we could take Ableton out and it would just be a dashboard on the website. It would all be automated.
And what’s next for ANCESTORS, and also the next works?
S.H. – It’s going to the Geneva International Film Festival, and then to the Portland Art Museum. We will have played 90 shows in one year. The success is really cool, but it’s a lot of work, and we’re a small team. That’s what I’m mainly busy with now, making sure that it will really work.
We just got a grant for the Smartphone Orchestra, the underlying platform. Until now, I’ve been the main artist. If I had an idea, I would apply for money, and if we got it, we’d build it. Most of the pieces are hard coded to work in a certain way. But after 10 years, we realized we’re kind of reusing stuff all the time. So now we’re trying to create the system in such a way that it will be a CMS, and easier for other people to create Smartphone Orchestra pieces curated by and guided by us – not only the technology but the dramaturgy as well. I’m really looking forward to putting that out, and at certain point have other people telling stories. That’s the heart of the Smartphone Orchestra. It’s about the people, and then I should not be the only one telling the story.
Leave a Reply
You must be logged in to post a comment.