Tele-Immersion
Participants: Klara Nahrstedt, David Padua, Wanmin Wu, Ahsan Arefin, Raoul Rivas, Albert Cohen
Executive Summary
With the current speed up in Internet and frame capturing rate of cameras, we need to speed up the 3D-reconstruction algorithm and rendering by some sort of parallelizing to meet up the real-time feature in the 3D Tele-Immersion.
Goals - Extended Description
A 3D Tele-immersive(TI) system is composed of multiple TI sites. Each site contains the infrastructure required to capture, disseminate and render the local 3D space. This space is then fused with each of the other sites in the 3D TI systems in the session. Each site consists of multiple cameras that surround the capturing area, displays rendering the virtual 3D room, and disseminating infrastructure. The TEEVE (Teleimmersion for EVerybody) is designed to manage multiple TI sites. It is modeled as a distributed publish-subscribe system. The publishers are represented by clusters of 3D cameras that capture the video frames and process them to produce a 3D video stream. Each cluster is responsible of producing a single 3D video stream. Each video stream is then forwarded to a rendezvous point (RP), residing in a Gateway Machine. The RP is responsible of disseminating the data to the subscribers of the system. Subscribers are represented by the 3D rendering displays.
Tele-immersive applications make heavy demands on both computational and network resources. The computational demand comes from real-time 3D reconstruction algorithms that are necessary to convert 2D frame images into their respective 3D frame containing the depth information. The network demand comes from the immense amount of data which is sent in multiple streams to and from active participants of the shared virtual environment. In addition, computational cycles are required to render the 3D streams received from the other remote parties. In our project, we mainly focus on the 3D reconstruction algorithms.
There are certain classes of applications that can leverage tele-immersion in its current state [1], but there are other potential users whose needs cannot be met due to tele-immersion’s large computational demands. For these tele-immersion applications to be feasible on general-purpose hardware, we need to examine methods of optimizing its most computational costly algorithms. In this project, we study an optimization strategy for this class of application that makes use of traditional loop-based compiler transformations. The transformations are applied to 3D reconstruction stereo code that is further transformed into a sequence of high-level data-parallel operations. This last transformation enables portability across parallel architectures including GPUs, multi-cores, and clusters.
Another goal of this project is to study the impact of using data-parallel primitives for productivity, portability, and the ability to optimize automatically. By implementing the tele-immersion application using a collection of high-level data-parallel constructs developed at Illinois, named Hierarchically Tiled Arrays (HTA) [2], one could have a system that encapsulates the parallelism and maps the data parallel operations across different architectures, thus simplifying the task of the domain programmer. Additionally, by expressing data-parallel operations using these constructs, the system would be able to perform autotuning on these data-parallel operations.
Results
Latency:
Improving 3D-reconstruction and parallel transmission allow real time 3D reconstruction and rendering with commodity hardware. Some graphs showing their latency are given below (click on images to enlarge).
Additional Resources
- Related videos and publications are available at TEEVE@UIUC website
External Collaborators
We would like to acknowledge our external collaborators for providing us their software and intellectual support.
- UCBerkeley: Ruzena Bajcsy, Gregorij Kurillo, Ramanarayan Vasudevan
- UC Davis: Oliver Kreylos
- NCSA: Peter Bajcsy
References
[1] Z. Yang, B. Yu, W. Wu, R. Diankov, and R. Bajscy. Collaborative dancing in tele-immersive environment. In Proc. of the 14th annual ACM international conference on Multimedia, pages 723–726, New York, NY, USA, 2006. ACM.
[2] G. Almasi, L. D. Rose, J. Moreira, and D. Padua. Programming for locality and parallelism with hierarchically tiled arrays. In Proc. of the 16th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2003, pages 162–176. Springer-Verlag, 2003.
| Graphics | Computational Imaging and Video | Natural Language Processing | Secure Web Browsing | Tele-Immersion | Video Vision Library |