In January we shared the news of a major update for Depthkit, one of the main tools for volumetric video capture. Since the creation of our magazine, we have been in regular discussion with designers who have used this technological solution – such as KUSUNDA, END OF NIGHT or THE CHANGING SAME. Last week we met with James George, co-founder and CEO of Depthkit to talk about new opportunities for XR around the software, and its latest update.
- Depthkit is a software toolkit that enables the creation of high quality volumetric video with minimal off the shelf hardware. It is specifically designed to work with depth cameras, such as the Microsoft Kinect and the Intel RealSense, and allows creators to capture, edit, and playback 3D volumetric video in a variety of formats and platforms.
- James co-founded Scatter in 2016 after building Depthkit as an open source project which began in 2011. Before founding Scatter, James co-directed CLOUDS, the first VR project to use volumetric video. He was also the first artist-in-residence at Microsoft Research.
- The company is based in Brooklyn, New York and has its headquarters as Zero Space among other cutting edge media production technologies.
- Depthkit’s customers have now grown to include not only indie creators, but also large enterprises leading the XR industry such as Accenture and The New York Times.
- The Depthkit workflow includes two components, the Depthkit application, which is used to record and export depth data. It also includes a Unity SDK for integrating volumetric video into Unity-based applications and games.
- With Scatter, their own production company, they co-produced several projects – amongst those which used the Depthkit technology: CLOUDS, ZERO DAYS VR, BLACKOUT, THE CHANGING SAME. They won an Emmy and were nominated for a second. A bit of history.
Depthkit: a solution by an immersive filmmaker for immersive filmmakers
James George – Originally, as an artist, I am passionate about new technologies that can impact the creation and the way we tell new stories. Long before the emergence of a new era for virtual reality I was hacking cameras for audiovisual material to adapt them to my projects, or to imagine new production techniques. The arrival of VR was a real turning point, very exciting, where I immediately began to apply volumetric capture.
J. G. – Back in 2010, I was part of openFrameworks, a group of developers who were hacking Microsoft’s Kinect’s open source drivers. The Kinect camera was very important technology, but tied to the Xbox ecosystem and not intentionally built for independent creator projects. I built an open source project to allow creators to capture volumetric video with the Kinect. That was really the initial prototype of Depthkit. I used it myself as a code artist to create CLOUDS, the first non-fiction VR project to use volumetric capture. Halfway between documentary film and video game, CLOUDS featured interviews of artists who work with code and explored how networked culture advanced creativity. CLOUDS premiered at Sundance New Frontier in 2014, among the first VR projects shown there. It went on to win the Storyscapes award at Tribeca, and was exhibited at the Museum of Modern Art in New York…
J. G. – That project is where it all started. What has persisted since is the belief that digital 3D humans representing real people can be captured authentically in a way that is more akin to filmmaking. This is a radical alternative to the traditional puppet characters typical in virtual worlds. At the heart of Depthkit is the idea of volumetric filmmaking.
J. G. – While Depthkit started as an open source project originally, following the explosion of the VR industry in 2016 it was clear that we needed to accelerate our product development. That’s when we started the company. Scatter coined the term “volumetric filming” and was founded on this concept. That’s our motivation: to provide the industry with tools and examples realizing the possibilities that volumetric video promises. And especially in the creation of non-fiction, it’s essential to capture the real world.
J. G. – In 2023, Scatter announced the release of a new Depthkit Studio version which is 10 times faster to set up! Originally, Depthkit was a “2.5d” volumetric capture software with a single angle of view – which is a version now called Depthkit Core. It was relatively easy, but visually rather limited because it couldn’t capture a full body from all angles. It is useful for learning the workflow, and creating view-constrained experiences or those that celebrate the glitches as part of the story or aesthetic. We learned that the creative community needed a 360 capture tool with the same values as Depthkit. Depthkit Studio was released as a private pilot in early 2021, and commercially in 2022.
Depthkit 2023: running fast!
J. G. – Since the Depthkit Studio pilot was released, we have stayed very close to our customers to collect feedback. This year’s first release is based on that feedback – which has been surprising. Yes, as we assumed, visual quality is very important and we continue to make huge improvements there. Depthkit supporting all output formats and playback platforms is important, too. But the surprising essential need was an easy setup process. Especially for camera crews with a lot of mobility, on location or in the studio. Set up, calibration, etc… All of this was time consuming. Our latest version, in early 2023, improves hardware calibration in two main areas:
- Better management of cameras (up to 10) and associated equipment. Until now, in case of a hardware problem, you often had to rely on guesswork to solve the problem. We have added many diagnostics to the interface to report problems related to the equipment itself. Our customers will now have a lot more feedback and solutions to identify and solve hardware related bugs. And this is important, because Depthkit is often used by customer’s hardware which can be very different.
- Calibration. The software offers more functions to understand the real time quality of your calibration solve through images. This is a real mark of confidence, and a desire to offer more transparency on how Depthkit works.
J. G. – Scatter collaborates with creative teams at different levels. The fact that Scatter is also a peer in the industry creating-award winning content is a huge advantage to legitimize our know-how when it comes to the Depthkit tool. It also gives us the opportunity to be active in the relationship with other artists. We can intervene as a producer or co-producer via Scatter, as we did initially with CLOUDS or ZERO DAYS VR (which won an Emmy!). THE CHANGING SAME by Michèle Stephenson, Joe Brewster and Yasmin Elayat is a co-production with their company Rada Studio (link). In the latter case, these are artists with real non-fiction experience in more traditional media. This is a perfect example for us of new voices in XR using our solutions. Finally, the third solution is that customers use Depthkit as a self-service software tool through Depthkit subscription licenses plans. In this case we simply provide traditional technical support for their production.
Volumetric Capture for XR
J. G. – Our company’s ability to adapt, and our dynamism, meets the needs of the immersive industry. We have to remain flexible, because we organize ourselves according to our own experience as well as the customer’s feedback.
J. G. – This is what I mean by artist driving innovation. Depthkit’s product requirements come directly from artists who are the most demanding customer segment. For example, if I’m a creator going to Nepal like Nowhere Media did on KUSUNDA, I have to be able to use Depthkit without an internet connection – or swap a laptop out if my computer crashes. Depthkit can work on the creator’s own hardware, so this is possible.
J. G. – In the case of END OF NIGHT produced by Makropol, creators need to be able to record hours of footage without running out of hard drive space. Thanks to our advanced compression and performant rendering solution, this is possible! I really believe that it’s the artists who are pushing the edge of innovation forward, and we need to provide the technical opportunities for them to achieve their goals – and fulfill their demands. And where creators go, other industries and use cases inevitably follow. Volumetric capture can be used for many applications, not only for the art and filmmaking worlds! Medicine, architecture, simulation and training… Scatter has many industrial and enterprise customers. But it’s the artists who are always innovating and doing what truly hasn’t been done before.
J. G. – Recent market evolutions have guided our development. On the one hand, mobile XR headsets are becoming independent of computers. This hardware’s autonomy offers more possibilities, and the Quest, HoloLens, Magic Leap have played an important role in advancing accessibility and usability for XR. They are easier to use… but their technical capacity is more limited due the mobile processors, including in the processing of geometry and videos. And these limitations impact the volumetric capture, which requires more processing power than traditional 3D models or 2D video playback. So the software has to catch up. Depthkit has pipelines that support playback on all platforms, and we work constantly to stay up to date to power all headsets – with the challenges that this presents.
J. G. – On the other hand, WebXR is the distribution platform of the future. It’s still a very new field, with technical limitations, but it can take us away from the closed app stores to offer experiences directly online. Today, when creators are looking to publish native applications they must navigate Meta, plus Steam, plus Viveport… meanwhile WebXR allows simpler publishing with no gatekeepers. But as with mobile XR processors, we still need to make progress to make volumetric video on these media performance and high quality.
J. G. – Finally, new passthrough capabilities on recent hardware is also an interesting advancement to the possibility of collaborative experiences. This is a hunch, but I believe Depthkit’s volumetric video live streaming will be an important element of passthrough experience.
You can get started with Depthkit by visiting http://depthkit.tv and creating an account. Every account includes a free evaluation of Depthkit Core to get started with volumetric video capture right away.
Introducing the MetaBox: Immersive Hybrid Collaboration powered by Volumetric Video
