Artificial intelligence has made its nest on the island of Venice Immersive 2023. The rapid expansion of AI-related technologies – and the artistic works that incorporate them – couldn’t avoid one of the XR events of the year. Meet Marc Da Costa and Matthew Niederhauser, the two creators of TULPAMANCER, an immersive, generative installation that left no one indifferent.
The implication of technology and art
Marc Da Costa – I have a PhD in cultural anthropology, which led me to study the relationship between new technologies and our society, with a real focus on climate change and the way that computer data shapes and governs our lives. I soon set up a company dedicated to the exploration of “big data”, in line with the questions I was asking myself. We worked on public data, and provided support for people wishing to better understand these new environments. At the same time, I had a parallel artistic practice that involved the same subjects, with a desire to present them through more creative works.
Matthew Niederhauser – For my part, as far back as I can remember, I’ve had a passion for taking apart computers and creating media with software. I was captivated by the possibilities of VR in the 90s, but got sidetracked working as a photojournalist in China for nearly a decade.. But new media caught up with me and in 2015 I co founded Sensorium with John Fitzgerald, an experiential studio in New York working at the forefront of immersive storytelling (OBJECTS IN MIRROR AR CLOSER THAN THEY APPEAR, ZIKR: A SUFI REVIVAL, METAMORPHIC…). My expertise also enabled me to teach at various universities and mentor projects at NEW INC where Sensorium was incubated. More recently, I joined ONX Studio, an XR production and exhibition space powered by the Onassis Foundation, as a Technical Director.

TULPAMANCER, a new discussion with an audience
M. N. – Marc and I have known each other since college. We reconnected about five years ago, and while discussing artificial intelligence and its impact on our creative practices, we quickly realised that our views on these subjects were aligned! By putting our hands on these new tools, we began to create new experiences – TULPAMANCER is the third project we’ve developed together, and certainly the most sophisticated.
M. Da. C. – The new AI generative tools are evolving very rapidly. They are now much more accessible and stable. Their use is gradually becoming more democratic. Our work has had to adapt to this context, by asking the question of our own artistic practice to be integrated into it. These are still chaotic technologies, based on raw data. If you take Photoshop, for example, you’ll know which filter or function to use to get where you want to go. Here, you have to explore, test, to find the right way. For Matthew and me, we had to relearn to understand our intuitions, our reflexes. Our role as “directors” was no longer so obvious, and no longer based on the same ideas. It’s almost more like craft labour, where you have to refine, sort and rework the results.

M. N. – On TULPAMANCER in particular, we wanted to create an interactive machine learning experience that generated something new every time for the participant. We felt we could accomplish this once we figured out how to consistently create equirectangular images with Stable Diffusion with regularity and consistency. It came from creatively exploring and defining the different kinds of outputs you can get, what the machine can offer you – and integrate it into a narrative. Over the last few months, the evolution of text, sound and now video has been incredible. We realised that it was possible to give substance to the installation experience we wanted to create.
M. Da. C. – Part of the intention was to explore the recent history of new AI-related technologies. Without spoiling the set-up, TULPAMANCER was born of the idea of provoking an encounter between the spectator and these new tools. This is very much in evidence in our set-up, with a two-part experience. And a real “vintage” design inherited from the 70s and 80s, at a time when research into artificial intelligence was still in its infancy. This creates a slightly dystopian or nostalgic atmosphere that instantly immerses us in this meta-technological research.
A physical installation with AI
M. N. – We wanted to prepare the viewer emotionally for what they would discover through the experience. And to open them up to these generative AI environments. That’s why the first part is more physical, more material. Eventually, the result of the experience disappears – all you retain is a printed version of your interactions, reinforcing the ephemeral nature of it all. Marc worked a lot on this point, playing with the virtual and real aspects of the installation. It was also fun to integrate all these objects from an old analog world.
M. Da. C. – We chose tools that were available, but that could also accompany us in the creation of the work. TULPAMANCER is a 12-scene experience in the headset, and technical limitations can quickly become apparent. It’s a huge number of images! We had to be thoughtful in order to construct this type of result. The aim was above all to bring our artistic and critical eye to bear on these avant-garde creative territories.

M. N. – For me, the most important thing during production was to see the quality of the scripts produced – and the voiceover. Being able to recreate a dialogue so faithfully convinced me that we were getting somewhere. The “personae” we were able to create, the generation of content within the story, the generative responses, all met a sufficiently high standard to finalise the experience – and not be afraid of the unknown involved. We couldn’t necessarily achieve photo-realistic work, but could deliver convincing dialogue.
M. N. – The debate is on whether or not we should be afraid of artificial intelligence. As an artist, I’m interested in understanding how far we can push its use… even if it means letting it do everything! Marc and I created the architecture, the skeleton of the experience, in constant dialogue with the machine. The final experience is based on the AI’s automation, but is inevitably influenced by our test sessions, our own voices in a sense, the texts we submit to it… The most important thing was to simplify the contact between the user and the AI, so that there were no filters or overly complex procedures. Obviously, the machine still has its limits, if we don’t guide it enough, but we’ve made progress in trying to eliminate them.
An ongoing debate… and a disruption
M. Da. C. – The TULPAMANCER project began before the Hollywood writers’ strike, and we’ve been following with interest the discussions around AI that have taken place – even if it’s only one of the points of negotiation between studios and creatives. AI offers new tools, with new opportunities, and obviously many changes in the production process for most of today’s creative content. These are changes to be taken seriously, and ones that many people in the creative industries – and all industries – will have to face. For us, the idea was to integrate this thinking into the heart of a physical installation, detached from an online environment that is ultimately too virtual or connected. We really tried to distance ourselves from a Terminator-like or pessimistic vision. After all, these are still machines powered by decisions or data linked to human activity. That’s the framework in which we need to think. By opening up a space for participants to have an intimate encounter with AI, we hoped to create a new perspective on what all these developments may actually mean.

M. N. – TULPAMANCER was built around numerous iterations. Marc tested the AI with thousands of texts and answers. Our technical director and developer, Aaron Santiago, was also very key to the development of TULPAMANCER as he built the web application to frame all the prompts and asset generation that was then rendered in Unity. In the end, we all collaborated very closely.In our view, this approach was a good example of AI-assisted creation. Obviously, these technologies are going to change a lot of things, but it’s our attitude that’s going to count. We have to remain in control of our tools, and not see them as necessarily negative.
M. Da. C. – We are learning every day about these AI machines, machine learning… To work with them is highly iterative and requires an entirely new sensibility and a new set of intuitions. TULPAMANCER is a multi-sensory experience that is part of the use of these new technologies, and illustrates their state today. And the quality is frankly breathtaking, which is why we were so excited to present our work at Venice Immersive this year. The political and economic implications that flow from AI are immense and need intense debate and oversight from the public sphere. As an artist, however, the creative possibilities are immense: when we remember how the development of photography or even tubed paints changed the artistic landscape, I am certain that something fundamental to creative work is set to transform profoundly.
Leave a Reply
You must be logged in to post a comment.