가상현실 환경에서 기하학적 음향 기술 기반의 3차원 사운드 모델링 기술에 관한 연구
김정길* 종신회원
A Study of 3D Sound Modeling based on Geometric Acoustics Techniques for Virtual Reality
Cheong Ghil Kim* Lifetime Members
요 약
스마트 폰의 대중화와 고속 무선 통신 기술의 도움으로 고품질 멀티미디어 콘텐츠가 모바일 기기에서 보편화되고 있다. 특히, Oculus Rift의 출시는 소비자 시장에서 가상현실 기술의 새로운 시대를 열고 있다. 또한 컴퓨터 게임을 보다 사실적 구동을 위한 3D 오디오 기술은 곧 차세대 모바일 기기에 적용될 것이며, 시각적인 것보다 더 광범위한 사실적 경험을 제공 할 것으로 예상된다. 따라서 이 논문에서는 가상현실 기반의 응용 프로그램에서 3D 사운드 모델링을 위한 개념, 알고리즘 및 시스템에 대하여 기술하고자 하며 특히 기하학적 음향 기술 기반의 알고리즘에 초점을 맞추었다. 이를 위하여 먼저, 사운드 전파를 모델링하기 위해 물리적 기반의 기하학적 알고리즘과 다중 채널 기술 중심으로 오디오 렌더링을 위한 중요한 설계 원칙 소개와 오디오 렌더링 파이프 라인을 씬 그래프 기반 의 가상 현실 시스템 및 최신 하드웨어 구조 소개를 포함한다.
Key Words : 3D sound, virtual reality, sound trashing, smart phone, high-quality multimedia
ABSTRACT
With the popularity of smart phones and the help of high-speed wireless communication technology, high-quality multimedia contents have become common in mobile devices. Especially, the release of Oculus Rift opens a new era of virtual reality technology in consumer market. At the same time, 3D audio technology which is currently used to make computer games more realistic will soon be applied to the next generation of mobile phone and expected to offer a more expansive experience than its visual counterpart. This paper surveys concepts, algorithms, and systems for modeling 3D sound virtual environment applications. To do this, we first introduce an important design principle for audio rendering based on physics-based geometric algorithms and multichannel technologies, and introduce an audio rendering pipeline to a scene graph-based virtual reality system and a hardware architecture to model sound propagation.
*남서울대학교 컴퓨터학과 ([email protected])
접수일자 : 2016년 12월 23일, 최종게재확정일자 : 2016년 12월 27일
I. Introduction
As the fast growing interest in virtual reality, the demand of technologies associated with them has increased as well. Interactive virtual environment systems combine audio and video to simulate the experience of immersive exploration of 3D virtual world by rendering the environment from the viewpoint of users. In order to achieve a realistic immersive VR environment, it is necessary to reproduce high-quality virtual space with visual as well as acoustic
spaciousness. However, the current virtual reality technology has been developed mainly focusing on visual element; whereas, the sound area has not reflected the surrounding environment properly [1,2].
Fig. 1. OSSIC, the first 3D audio headphone
Under this circumstance, the first 3D audio headphone, OSSIC X, shown in Fig. 1, was introduced in early 2016, which instantly calibrates to the anatomy for the most accurate and immersive 3D audio [3]. Fig. 2 shows an overview of sound rendering [4] which is simulating acoustic reality in a virtual world such as the goal of graphics is simulating visual reality in virtual world. As you can see, there are lot of things that need to be covered [5, 6].
Fig. 2. Overview of sound rendering
This paper gives an overview of basic auralization methods for 3D virtual environment applications and recent research on geometric sound propagation techniques. The rest of the paper is organized as follows. In Section 2, we review the basic concept of 3D sound and system. Section 3 covers computational modeling approache based on geometrical simulations method with a new hardware architecture for sound propagations unit. Section 4 concludes this paper.
Ⅱ. Background
3D sound has become more popular as an effective mediam for human computer interfaces in many researches and industries with the feature of containing location information for specific alerts or warnings.
Therefore, it is becoming most important for scientific, commercial, and entertainment systems [7].
The basic processing pipeline is shown in Figure 3, which consists of two stages of acoustic modeling and auditory display. The input to an auralization system is a description of a virtual environment, an audio source location, an audio receiver location, and an input audio signal. The auralization system computes a model for the propagation of sound waves through the environment and constructs digital filter(s) that encode the delays and attenuations of sound traveling along different propagation paths. Convolution of the input audio signal with the filter(s) yields a spatialized sound signal for output with an auditory display device [8, 9, 11].
Fig. 3. Overview of sound rendering
In general, the multi-channel audio system or 3D sound technology with HRTF (head-related transfer function), is one method to reproduce this acoustic spaciousness [8, 9, 10]. That is, HRFT can describe process that an ear receives a sound from a point where sounds are generated in space, However, it is known that HRTF based 3D sound technologies have some limitations to reproduce realistic sound properly. This is because they do not simulate the physical effects of sound between many sound sources and the listener in the virtual 3D space.
To overcome the limits, 3D sound technology based on geometric method has been introduced [5-9]. In the geometry methods, the convergence method combining ray-tracing in 3D graphics and sound-processing is called sound-tracing. Sound-tracing is a method to create the realistic sound by tracing propagation paths between listener and sound source.
Fig. 4. Sound propagation paths
Generally, modeling sound propagation is a process of finding a solution to an integral equation expressing the wave field of points in space in terms of surrounding surfaces. Here, there are difficulties with modeling sound scattering waves in a 3D environment. Sound waves traveling from a source to a destination travel along a variety of propagation paths with different sequences of reflections, diffractions, and refractions at surfaces of the environment in Figure 4. And we can notice that sound and light are both wave phenomena, modeling sound propagation is similar to global illumination. The effect of these propagation paths includes reverberation to the original source signal as it reaches the receiver. So,
auralizing a sound for a particular source, receiver, and environment can be achieved by applying filter(s) to the source signal that model the acoustical effects of sound propagation and scattering in the environment [ ].
Ⅲ. Geometric Acoustics Techniques
For sound simulations, Eq. 1 shows the wave equation and it is described by the Helmoltz-Kirchoff integral theorem [14] with Equation 2, which is similar to Kajiya’s rendering equation [63], but also incorporates time and phase dependencies.
∂
∂′
∂∂′
∂
∂′
∂
∂′
∇′ (1)′
∞ ∞ (2)Here, boundary conditions may be defined as follows:
boundary condition is∇ ; Sound-hard boundaries is
; Sound-soft boundaries is ; impedance
boundary condition is
; Sommerfeld radiation
condition (for infinite domains):
lim
→∞
.Fig. 5. Image source method
In general, GA (Geometric Acoustics) techniques use image-source method, ray tracing, beam tracing, and ray-frustum tracing for computing sound propagation paths. Image source methods [2, 13] as shown in Fig. 5 compute specular reflection paths by considering virtual sources generated by mirroring the location of the audio source, over each polygonal surface of the environment.
Ray tracing methods [73, 133] find reverberation paths between a source and receiver by generating rays emanating from the source position and following them
through the environment until an appropriate set of rays has been found that reach a representation of the receiver position. Figs. 6 and 7 show the overview of ray tracing method. Beam tracing methods [25, 52] classify propagation paths from a source by recursively tracing pyramidal beams as shown in Fig. 8.
Fig. 6. Ray tracing method
Fig. 7. Ray tracing method
Ray theory makes the assumption that sound wavelengths are significantly smaller than the size of obstacles, and thus they are valid only for high-frequency sounds. The general approach is similar to methods used in computer graphics. A geometric algorithm is used to find significant ray paths along which sound can travel from a source to a receiver (Figure 6), and mathematical models are used to approximate the filters corresponding to source emission patterns, atmospheric scattering, surface reflectance, edge diffraction, and receiver sensitivity for sound waves traveling along each path. Finally, an impulse response is constructed by combining the filter(s) for each propagation path.
After the geometry propagation paths are constructed between each sound source and the listener, a digital filter is created for each propagation path and is convolved with the source signal in auralization pipeline. Spatial processing can be implemented in the pipeline to reproduce 3D positional audio effects in Fig. 9.
Fig. 8. Beam tracing method
Fig. 9. Auralization pipeline
The signal processing for each geometric path generally consists of 3 phases: 1) a resampling phase, 2) a “filtering”
phase, and 3) a spatial output phase. Fig. 10 shows a well-known signal processing pipeline, in which each sound path is processed independently. An alternative signal processing pipeline constructs the complete impulse response of the environment by superimposing all the filters for each propagation path. Convolution with the input audio signal is then delayed to the final stage of the process at the expense of having to convolve a longer filter [11].
Fig. 10. Signal processing pipeline
Recently, a new research on sound auralization hardware unit based on ray trashing method [12, 13] has been introduced. Fig. 11 shows the overview of the sound propagation unit at the architecture level, which consists of ray processing and sound processing unit.
Fig. 11. Sound propagation unit
Ray processing unit constis of Ray Generation, Traversal & Intersetion Test, and Hit Point Calculator.
Sound processing unit includes PPV (Propagation Path Validation), RGC (Reverb Geometry Collection), Path IR Calculator, and Reverb IR Calculator. After sound propagation unit, a hardware unit for sound auralization is placed.
Ⅳ. Conclusioin
As the growing popularity of immersive virtual reality, 3D audio technology that is currently used to make computer games more realistic will soon be applied to the next generation of mobile phone. This paper introduces the surveys of concepts, algorithms, and systems for modeling 3D sound virtual environment applications. For this purpose, this paper first introduces an important design principle for audio rendering based on physics-based geometric algorithms and multichannel technologies.
Finally, a new research of hardware based sound progagaion unit is introduced at architecture level.
ACKNOWLEDGMENT
본 논문은 2016년도 남서울대학교 교내연구비 지원에 의 하여 연구되었음
참 고 문 헌
[1] Juwon Yun, Dukki Hong, Woo-Chan Park, Cheong-Ghil Kim, “The design of a Software Development Kit for Virtual Reality 3D Audios”, Proceedings of the 2nd EEECS 2016, Science and Engineering Letters Vol. 2, (2016) 44-46 [2] Dukki Hong, Seiyoung Lee, Woo-Chan Park, “3D Audio
Down-Mixing System for Immersive Realistic Virtual Reality”, Proceedings of the 2nd EEECS 2016, Science and Engineering Letters Vol. 2, (2016) 15-16
[3]https://www.ossic.com/
[4] C Lauterbach, A Chandak, D Manocha, "Interactive sound rendering in complex and dynamic scenes using frustum tracing", IEEE Transactions on Visualization and Computer Graphics, 13 (6), (2007) 1672-1679
[4] ZhiYing Zhou, Adrian David Cheok, Yan Qiu, and Xubo Yang, "The Role of 3-D Sound in Human Reaction and Performance in Augmented Reality Environments", IEEE Transaction on Systems, MAN, and Cybernetics—PART A:
Systems and Humans, 37 (2) (2007) 262-272
[4] William W. Gaver, Randall B. Smith, Tim O'Shea, "Effective sounds in complex systems: the ARKola simulation", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (1991) 85–90
[5] Durand R. Begault, "3D Sound for Virtual Reality and Multimedia", AP Professional, Boston, MA, 1994.
[6] Fakheredine Keyrouz, Klaus Diepold, "An Enhanced Binaural 3D Sound Localization Algorithm", 2006 IEEE International Symposium on Signal Processing and Information Technology, (2006) 662-665
[7] C. Phillip Brown and Richard O. Duda, “A Structural Model for Binaural Sound Synthesis”, IEEE Transactions on speech and audio processing, 6 (5) (1998) 476-488 [8] Jürgen Herre, Johannes Hilpert, Achim Kuntz, Jan Plogstie,
"MPEG-H Audio - The New Standard for Universal Spatial / 3D Audio Coding", International Symposium on Signal Processing and Information Technology, (2006) 662-665 [9] M. Vorlander, Auralization: Fundamentals of Acoustics,
Modelling, Simulation, Algorithms and Acoustic Virtual Reality, Springer, 2008.
[10] A. Febretti et al., “Cave2: a hybrid reality environment for immersive simulation and information analysis,” in Proceedings SPIE Electronic Imaging 2013, Burlingame, CA , 2013, 864903-864903-12.
[11] Thomas Funkhouser, Nicolas Tsingos, Jean-Marc Jot, "Survey of Methods for Modeling Sound Propagation in Interactive Virtual Environment Systems, http://citeseerx.ist.psu.edu/
viewdoc/versions?doi=10.1.1.9.5567
[12] Jae-Ho Nah, Jin-Woo Kim, Junho Park, Won-Jong Lee, Jeong-Soo Park, Seok-Yoon Jung, Woo-Chan Park, Dinesh Manocha, and Tach-Don Han, "HART: A Hybrid Architecture for Ray Tracing Animated Scenes," IEEE Transactions on Visualization and Computer Graphics, 21 (3) (2015) 389-401 [13] Jae-Ho Nah, Hyuck-Joo Kwon, Dong-Seok Kim, Cheol-Ho
Jeong, JinHong Park, Tack-Don Han, Dinesh Manocha, and Woo-Chan Park, "RayCore: A ray-tracing hardware architecture for mobile devices," ACM Transactions on Graphics, 33 (5) (2014) Article 162
저자
김 정 길(Cheong Ghil Kim) 종신회원
․1987년 8월：Univ. of Redlands, USA 컴퓨터과학과 학사졸업
․2003년 8월：연세대학교 컴퓨터과학 과 공학석사 졸업
․2006년 8월 : 연세대학교 컴퓨터과학과 공학박사 졸업
․2006년 ∼ 2007년 : 연세대학교 컴퓨터과학과 박사후 연구원
․2007년 ∼ 2008년 : 연세대학교 컴퓨터과학과 연구교수
․2008년 ∼ 현재 : 남서울대학교 컴퓨터학과교수
<관심분야> : 멀티미디어 임베디드 시스템, 이기종 컴퓨팅, 모바일 AR, 3D Contents
