Julius Von Haast Fellowship Project: Holistic Volumetric Representation for Immersive Videos

The research project is supported by Julius von Haast Fellowship, which is a sub-programme of Catalyst: Leaders from the Royal Society Te Apārangi.

New Zealand Host: Fang-Lue Zhang, VUW Computer Graphics Group

The JVH Fellow: Prof. Reinhard Koch from Kiel University

The current research progress:

In exploring solutions for representing and editing omnidirectional content, existing methods—whether tailored for 2D perspective imagery or panoramic views—often struggle to effectively capture the intricate spatiotemporal relationships necessary for creating high-quality, editable outputs in dynamic, panoramic scenarios. Our project addresses these challenges through the development of OmniPlane, a novel approach designed to enhance the representation and editability of real-world dynamic omnidirectional environments captured casually using commodity omnidirectional cameras.

So far, we have made significant progress by implementing OmniPlane's core methodology, which leverages spherical spatiotemporal feature grids. This involves computing spatiotemporal features by fusing learnable spatial and spatiotemporal feature planes within a spherical coordinate system, along with a customized weighted sampling strategy that accounts for the inherent spherical nature of omnidirectional content. Additionally, we introduced a technique to decompose these learned feature planes into palette-based color bases, enabling flexible recoloring of omnidirectional videos.

Preliminary results, supported by extensive ablation studies and comparative analyses, indicate that our approach not only enhances the representation capabilities for omnidirectional dynamics but also provides robust tools for recolorable editing. This progress demonstrates the effectiveness of our method in advancing dynamic omnidirectional content manipulation.

Some reconstruction and editing results are shown below: