Mixed reality multi-person interaction research based on the calibration of the HoloLens devices

(1)

Mixed reality multi-person interaction research based on the calibration of the HoloLens devices

Zi Jie Qin

^†

, Ao Xuan Li

^††

, Hyotaek Lim

^†††

, Byung Gook Lee

^††††

ABSTRACT

Currently, the application of virtual reality technology is becoming more and more popular in all aspects of life. From virtual entertainment to industrial simulation, the new operation and working methods brought by virtual reality visualization technology have greater appeal and advantages. With the renewal and iteration of related equipment, more and more functions make its limitations continue to decrease, but its applicability continues to improve. Take the optically transparent head-mounted device as an example. It integrates more computer functions, presents and interacts in a virtual way, further integrates with daily behaviors, and shortens the distance between users and digital information.

Key words: Mixed Reality, Multi-Person Interaction, Calibration, HoloLens

※ Corresponding Author : Byung Gook Lee, Address:

(47011) Dongseo University, ChuryeRo 47, SasangGu, Busan 47011, Republic of Korea, TEL :

E-mail : [email protected]

Receipt date : Jul. 2, 2021, Revision date : Aug. 2, 2021 Approval date : Aug. 16, 2021

†

Dept of Computer Engineering, Dongseo University (E-mail : [email protected])

††††

Dept of Computer Engineering, Dongseo University (E-mail : [email protected])

††††

Dept of Computer Engineering, Dongseo University (E-mail : [email protected])

††††

Dept of Computer Engineering, Dongseo University

※ This work was supported by the National Research Foundation of Korea grant funded by the Korea govern- ment(MSIT) (No. 2020R1A2C1008589).

1. INTRODUCTION

With the continuous update and iteration of technology and design, the amount of information we can process, receive and display is also increasing. Developers have made virtual reality technology further serve some of our current en- tertainment, work areas and daily life areas [1,2].

In recent years, the development of network and cloud information transmission technology has made network transmission relatively more opti- mized, more stable, and more efficient. The cov- id-19 outbreak in early 2020 has greatly increased the demand for information and communications in isolation. Due to this epidemic, people's daily lives have been greatly hindered. Therefore, online services such as online classrooms, online confer- ences, and online offices are all used to fill gaps in society in a short period of time.

For many activities that require interaction and social interaction in people's daily lives, the space and low-cost advantages of virtual reality technol- ogy can theoretically meet part of the activities of people restricted by epidemics [3]. In this way, people can interact with friends or other people at home for interactive entertainment, collaborative work and other content, free from space and epi- demics, and provide a safer way of social behavior.

Therefore, the multi-person interaction research of virtual reality technology has important value and pioneering.

2. BACKGROUND

2.1 Mixed Reality

Mixed Reality(MR) [4] is a specific subset of

Virtual Reality (VR) related technologies [5]. It in-

volves merging the real and virtual worlds some-

(2)

where in the "virtual continuum", connecting the completely real environment with the completely virtual environment. And based on advances in computer vision, graphics processing capabilities, display technology, and input systems, the user, computer, and environment interact with each oth- er (Fig. 1).

Fig. 1. The interactions between computers, humans, and environments.

2.2 HoloLens Devices

The Microsoft HoloLens is a self-contained system running a special version of Windows 10 that it generates and projects stereographic images onto the lenses of a headset [6]. The experiment in this paper mainly uses Microsoft HoloLens 1 and 2 devices (Fig. 2) as the main test content and re- search basis.

Fig. 2. HoloLens 1 (left) and HoloLens 2 (right).

2.3 Error and offset

Although the HoloLens device has a certain cali- bration function, the displacement and error caused by some subtle differences in the human body,

movement, time and application always exist. In order to solve this problem in this paper, just refer to the traditional offline calibration method to re- duce the error [7].

2.4 Calibration research

Based on the above errors, in order to prevent the actual experience from being excessively af- fected, we refer to the more popular calibration methods [8,9]. In the specific measures of this re- search, the method used is to use eyes instead of the camera, and calculate the projection matrix of the rendering camera based on photogrammetry technology with eyes calibration in HoloLens de- vices (Fig. 3).

Fig. 3. The left is Eye calibration in HoloLens devices.

The right is camera-based calibration: C can be

regarded as the imaging position of the human

eye, and E is the coaxial camera relative to the

display.

(3)

Fig. 4. The target based on AR image tracking serves as a calibration reference.

Fig. 5. The Lobby logic structure.

Fig. 6. Multi-user connection logic structure.

2.5 Multiplayer Server

The establishment of a qualified multiplayer server is the basis of multiplayer interaction. In this article, we will build a multiplayer server as a pre-interaction link. The conditions it needs to meet include providing users with sufficient in- formation prompts and being able to synchronize information between users to ensure as much re- al-time synchronization as possible [10].

3. EXPERIMENTAL DESIGN

3.1 Offline Calibration Design The main steps [11] :

1. Physical method: adjust the position of the de- vice to ensure that the user can see the full screen range (completed with the aid of virtual markers and guidance).

2. Virtual camera adjustment: Using the target of image tracking as a reference, adjust the relative spatial position of the virtual camera to match the virtual content visible to the human eye. Refer to Fig. 4.

3.2 Server Framework Design

Photon Cloud [12] is the basis of the server, and the program script will always check that the link is normal. The second part is the "room lobby".

Users can choose to enter other people's game rooms or create their own game rooms. The user who creates the game room will decide the start of the game and the closing of the game room. See

Fig. 6.

In the Lobby section (Fig. 5). “Lobby” is the first

interface the user sees. “Create a room” is used to

(4)

Fig. 7. Virtual chessboard ( “ Pawn ” ).

Table 1. Image Targets that possess the following attributes will enable the best detection and tracking performance from the Vuforia Engine.

Attribute Example

Rich in detail Street scene, group of people, collages and mixtures of items, and sport scenes are good examples.

Good Contrast Images with bright and dark regions and well-lit areas work well.

No repetitive patterns

Employ unique features and distinct graphics covering as much of the target as possible to avoid symmetry, repeated patterns, and feature-less areas.

Format Must be 8- or 24-bit PNG and JPG formats; less than 2 MB in size; JPGs must be RGB or greyscale (no CMYK).

create a list with limited capacity. “Join the room”

is used to join the selected room (list). "Start game"

is used to lead all users into the scene. “Ready/

None” is used to confirm the user's readiness.

3.3 Content Design

As for the choice of interactive content, we hope to have as many interactive methods as possible to satisfy the perception provided to users by mixed reality (such as sound, voice, color, space, motion, and touch feedback).

Considering that the substantive content must have enough sociality to promote the interaction between users, the main interactive part is finally presented in the form of chess. The reason is that the competitive nature of chess helps to stimulate the enthusiasm of users. Secondly, the more peace- ful operation of chess can prevent some excessive movements from affecting the displacement of the user's head-mounted device. Finally, it is consid- ered that it can be substantive The activities of

things are compared for reference. The specific measure for presenting chess interaction is to real- ize "automatic" chess that can be controlled by voice and gesture interaction through various in- teractive functions of HoloLens. (Similar to chess with voice control in "Harry Potter")

At the same time, we made an animation of each chess piece, so that the chess piece will have corre- sponding action feedback when it is controlled to move, eat or be eaten, to show the fun of the virtual part in mixed reality, which is more interesting than pure chess movement. Can maintain the us- er's experience interest. Refer to Fig. 7.

3.4 Target Image Design

This part of the design mainly focuses on two

objects. The former is used to provide a reference

in the calibration part. The latter is used to place

a virtual chessboard. Two image objects are de-

signed according to the feature point standard of

Vuforia (Table 1).

(5)

Fig. 8. Multi-person online practical test.

Fig. 9. Voice recognition will affect the recognition speed due to noise interference. The picture shows a prompt for successful voice recognition.

4. TEST AND RESULTS

4.1 Test Process

The user enters the application through the HoloLens device to realize the import and installa- tion, and then the calibration part appears first [13].

Pseudo-text will be used to guide the user, and

"Next" will be used as the password to perform voice control to proceed to the next step. After completing the calibration part, the two users will directly enter the server scene. Here, the user can choose to join a "room" created by others or create a "room" by himself (refer to Fig. 5 for the proc- ess). The person who creates the room will act as the host server, and the person who joins the room will act as the guest server. Refer to Fig. 8.

The host server needs to ensure that the guest server is in a ready state before sending in-

structions to make both parties enter the chess- board scene together. In the chessboard scene, both users need to determine the position of the chess- board by finally identifying the corresponding im- age target to place the chessboard, and then the chess game can be started. White (host server) goes first, and both users can take turns using gesture control, touch control and Speech Control to control the movement and capture of the chess(the movable range of the pieces will be high- lighted). Please refer to the test video content of [13].

4.2 Test Results

Judging from the results of multiple tests:

According to the past experience of AR multi-per- son interaction on mobile terminals (please refer to the video [14]), image tracking usually causes deviation and tremor due to equipment and other reasons, but it can be identified by the space of HoloLens equipment And a more stable tracking system to optimize this function. The rest of the interactive functions basically meet the expected expectations, but there are some delays and low sensitivity issues with touch gestures, which will affect the overall interactive experience.

Secondly, because voice commands can be re-

(6)

ceived by two devices at the same time, the script will turn off the voice receiving function of the oth- er user during the corresponding game round of the user to avoid errors. However, Speech Contorl has a certain delay (Refer to Fig. 9), and after many tests, it is determined that there is a part of noise interference. Factors make the identification time longer. As an insurance, other gesture control and touch control can play a substitute role in time. The above content has been tested and feasible in HoloLens1 and HoloLens2.

In addition, when you need to pay attention, the device will occupy the use of the device camera during the screen recording, so the frame rate of the device will be reduced during the screen re- cording, and the actual operation of the device will be more smooth. In addition, due to the influence of the frame rate, some preset animations con- trolled by the number of frames will be relatively delayed, and we will find ways to improve the cor- responding problems in the future.

5. CONCLUSION

According to the observation of the existing test results, although the expected results can be ach- ieved, there are still some issues that need to be paid attention to:

- In the calibration phase : need to optimize the tracking loss problem.

- The visual data of the content scene needs to be optimized : add virtual visual information and optimize the UI.

- The voice control part needs further improve- ment : optimizing the recognition mode to avoid the impact of verbal errors.

In order to conduct further research on this ex- periment, we hope that we can interact with vari- ous types of devices (not only holographic devices, mobile phones, PCs, VR devices, etc.), and try to build a general virtual interactive system.

REFERENCE

[ 1 ] C. Anthes, R. J. García-Hernández, M. Wie- demann, and D. Kranzlmüller, “State of the art of virtual reality technology,” 2016 IEEE Aerospace Conference, pp. 1-19, 2016.

[ 2 ] C. Kwon, “A Study on the Relationship of Distraction Factors, Presence, Flow, and Learning Effects in HMD–based Immersed VR Learning,” Journal of Korea Multimedia Society, Vol. 21, No. 8, pp. 1002-1020, 2018.

[ 3 ] M.R. Miller, H. Jun, F. Herrera, J.Y. Villa, G.

Welch, and J.N. Bailenson, “Social Interaction in Augmented Reality,” PLoS ONE, 14(5):

e0216290, pp. 1-25, 2019.

[ 4 ] M. Speicher, B.D. Hall, and M. Nebeling,

“What is mixed reality?,” Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2019.

[ 5 ] What is Mixed Reality, © Microsoft (2021), https://docs.microsoft.com/en-us/win- dows/mixed-reality/discover/mixed-reality (accessed July, 20, 2021).

[ 6 ] A.K. Noor, “The HoloLens Revolution,”

Mechanical Engineering, Vol. 138, Issue 10, pp. 30-35, 2016.

[ 7 ] W.A. IJsselsteijn, Y.A.W. de Kort, and A.

Haans, “Is This My Hand I See Before Me?

The Rubber Hand Illusion in Reality, Virtual Reality, and Mixed Reality,” Presence: Teleop- erators and Virtual Environments, Vol. 15, Issue 4, pp. 455-464, 2006.

[ 8 ] N. Guo, T. Wang, B. Yang, L. Hu, H. Liu, and Y. Wang, “An Online Calibration Method for Microsoft HoloLens,” IEEE Access, Vol. 7, pp. 101795-101803, 2019.

[ 9 ] X. Hu, F.R.Y. Baena, and F. Cutolo, “Align- ment-Free Offline Calibration of Commercial Optical See-Through Head-Mounted Dis- plays With Simplified Procedures,” IEEE Access, Vol. 8, pp. 223661-223674, 2020.

[10] S. Natkin and C. Yan, “User Model in Multi-

(7)

player Mixed Reality Entertainment Applica- tions,” Proceedings of the 2006 ACM SIGCHI international conference on Advances in com- puter entertainment technology, pp. 1-9, 2006.

[11] Calibration in HoloLens(2021), https://youtu.

be/lEbC23bQLoQ (accessed July, 20, 2021).

[12] Photon: Realtime Intro(2020), https://doc.

photonengine.com/en-us/realtime/current/

getting -started/realtime-intro (accessed July, 20, 2021).

[13] Mixed Reality Auto Chess(2021), https://

youtu.be/jKV-moA1eVE (accessed July, 20, 2021).

[14] Multiplayers AR Fight Chess based on AR Core(2021), https://youtu.be/_nRlulPYTiE (accessed July, 20, 2021).

Zi Jie Qin

2019 B.S. in Multimedia Design, Shanghai University of Engineering Science 2021 M.S. in Computer

Engineering, Dongseo University

Research Interest : Multimedia Design, Argument Reality, Mixed Reality, Human Computer Interaction

Ao Xuan Li

2020 B.S. in Multimedia Design, Shanghai University of Engineering Science 2020～Master's student,

Department of Computer Engineering, Dongseo University

Research Interest : Multimedia Design, Argument Reality, Mixed Reality, Human Computer Interaction

Hyotaek Lim

1988 B.S. in Computer Science, Hongik University 1992 M.S. in Computer Science,

POSTECH

1997 Ph.D. in Computer Science, Yonsei University 1988～1994 research staff, Electronics and Telecom-

munications Research Institute 1994∼Professor, Department of Computer

Engineering, Dongseo University

Research Interest : Ubiquitous and Mobile Networking, Cloud Computing, Storage Area Networks