Collaborating on a physical object when two people aren’t in the same room can be quite challenging. However, a groundbreaking remote conferencing system called SharedNeRF is changing the game. This system allows remote users to manipulate a view of the scene in 3D, enabling them to assist in complex tasks like debugging complicated hardware. SharedNeRF combines two graphics rendering techniques to create a more immersive and interactive experience for remote collaboration.
SharedNeRF leverages two key graphics rendering techniques – one that is slow but photorealistic, and another that is instantaneous but less precise. This unique combination allows the remote user to experience the physical space of the collaborator in a whole new way. This system opens up the possibilities for working on tasks that were previously challenging to convey through traditional video-based conferencing systems with limited angles.
Created by Mose Sakashita, a doctoral student in the field of information science, SharedNeRF represents a paradigm shift in remote collaboration. Sakashita, who developed the system as an intern at Microsoft, worked closely with Andrew Wilson ’93, a computer science major at Cornell. The innovative work on SharedNeRF will be presented at the Association of Computing Machinery (ACM) CHI conference on Human Factors in Computing Systems (CHI’24) and has already received an honorable mention.
SharedNeRF takes a novel approach to remote collaboration by utilizing a graphics rendering method known as a neural radiance field (NeRF). NeRF uses artificial intelligence to construct a 3D representation of a scene based on 2D images, creating highly realistic depictions with reflections, transparent objects, and accurate textures. In SharedNeRF, a local collaborator wears a head-mounted camera to record the scene, which feeds into a NeRF deep learning model. This allows the remote collaborator to view the scene in 3D and rotate the viewpoint as desired.
To address the 15-second update delay of the NeRF model, Sakashita’s team combined detailed visuals from NeRF with point cloud rendering technology. The use of both rendering techniques allows remote users to view the scene from various angles with high quality while also seeing real-time movements through point clouds. SharedNeRF also incorporates an avatar of the local collaborator’s head, providing the remote user with insights into where they are looking.
In user testing, SharedNeRF outperformed standard video conferencing tools and point cloud rendering alone. Volunteers found that the system helped them see design details more clearly and gave them greater control over what they were viewing. While currently designed for one-on-one collaboration, the researchers envision extending SharedNeRF to accommodate multiple users. Future work will focus on improving image quality and offering a more immersive experience through virtual reality or augmented reality techniques.
Leave a Reply