A Study on the Tele-medicine Robot System with Face to Face Interaction

A Study on the Tele-medicine Robot System with Face to Face Interaction

Dae Seob Shin

Abstract

Consultation with the patient and doctor is very important in the examination. However, if the consultation cannot be done directly, such as corona virus, it is difficult for the doctor to determine the patient’s condition more accurately. Recently, an image counseling system has been developed based on the Internet, but in the case of heart disease, remote medical counseling cannot be performed because it is not possible to stethoscope the heart sounds remotely. In order to solve this problem, it is necessary to develop an interactive mobile robot capable of remote medical consultation, and a doctor and a patient should be able to set a planting sound during consultation and transmit it in real time. In this paper, we developed a robot that can remotely control a medical counseling robot to move to a hospital room where patients are hospitalized, and to consult a patient in the room remotely from a doctor’s office. A remote medical imaging stethoscope system for real-time heart sound transmission is presented. The proposed system is a kind of P2P communication that transmits video information, audio information, and control signal independently through webRTC platform, so that there is no data loss.

Consults and sees doctors in real time and finds it more effective than traditional methods for patient security.

The system implemented in this paper will be able to perform remote medical care in the place where the spread of diseases between humans like the recent corona 19 as well as the remote medical care of heart disease patients in the future.

Key words：Telemedicine, Face-to-face, WebRTC, Stethoscope, Mobile robot

* Dept. of Electronic Information Communication Division, Shin Ansan University

★ Corresponding author

E-mail：[email protected], Tel：+82-2-2679-8556

※ Acknowledgment

Manuscript received Mar. 10, 2020; revised Mar. 19, 2020; accepted Mar. 24, 2020.

This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Ⅰ. Introduction

Recently, with the emergence of smartphones and tablet PCs in line with 5G technology, as the information communication technology has been developed and advanced, the interface technology between human and computer has been developed, and the interface technology between human and computer has been diversified in various forms.

As 5G technology develops, u-health technology is attracting attention as it is being combined

with an increase in the elderly population. Advances in communication also mean that there is a foundation for providing health care anywhere, anytime[1][2].

As communication technology improves, efforts

to create added value in telemedicine using video

communication have been attempted in various

fields. In recent years, telemedicine has been

revisited due to u-health. There is a need for a

service that can receive medical services anywhere

outside the hospital, measure biometric information,

293

and provide consultation with a doctor without going to the hospital.

In particular, as the elderly population increases, the demand for remote monitoring of patients without going to the hospital is increasing. In addition, when a virus that spreads between humans has appeared recently, such as a corona virus, there is a need for a system that enables a doctor to perform a medical examination remotely without directly meeting a patient.

Even in hospitals, doctors need a system that enables them to remotely manage their patients without visiting the patient’s room. Types of telemedicine are shown in Fig. 1. It can be divided into three as follows. (a) may be a medical consultation between the medical institution’s ward and the physician. (b) may provide consultation between school facilities and physicians. (c) enables telemedicine, such as homes and health care facilities. This study is an effective robotic system for conducting medical consultations with patients and medical institutions such as (a) and (c).

Fig. 1. Telemedicine Service Classification and Structure.

Fig. 2. Appearance of Remote Video Robot.

In medical consultations, doctors need a minimum of photos and a stethoscope to accurately determine a patient’s condition. Observation of the trauma of the patient is possible through a remote imaging robot. However, patients with heart disease are difficult to consult because they cannot remotely listen to heart sounds. It should be possible to measure the heart sounds during the consultation between the doctor and the patient and transmit them in real time. If a system capable of measuring such sound can be transmitted in conjunction with a remote video robot, effective telemedicine will be possible.

In this paper, we propose an video robot system that can be operated remotely to check the condition of the patient. Remote control is possible not only from PC but also from smart device, and it is configured so that doctors can conveniently control the image robot using various sensors and multi-touch technology built in smartphone or tablet PC as well as from fixed PC[3]. In addition, we designed, implemented and implemented a telemedicine video consultation system for heart sound transmission over the Internet. For video communication, we use WebRTC platform, which is more effective than the existing P2P communication, to transmit video information, audio information, and data information separately[4].

Ⅱ. Related Work

1. Remote Video Robot System Configuration The remote image robot system is designed and manufactured to be composed of three types.

* Robot Mechanism Design

First of all, the design of the robot’s external mechanical part were carried out. In general, the robot was designed with a low cost and simple structure so that it can be used from homes to medical institutions without burden.

In addition, the robot has two wheels, a 12-inch

tablet PC monitor with a height of 120cm, and

Fig. 5. Robot base manufacturing process.

the screen can be moved by Fan / Tilt. Fig. 3 shows the drawings of the remote imaging robot mechanism and modeled in 3D. In Fig. 4, the robots have a friendly feeling by covering various types of character dolls on the remote video robot.

Fig. 3. designed remote video robot base.

Fig. 4. Appearance design on remote video robot.

* Robot base production

The base part of the robot was manufactured to fit the designed robot. Two motors and ball casters were used. Fig. 5 shows the manufacturing process of the robot base.

We manufactured the robot base, developed the robot drive unit, and installed the ultrasonic sensor around the robot to move the robot safely.

We also developed a control algorithm to drive two motors.

① Development of Autonomous Driving Control Algorithm for Mobile Robot

- Intelligent control algorithm and control method for remote control driving of Mobile Robot driven by 2 axis DC motor of Differential Driving type were developed.

- The sensor part of Mobile Robot adopts IR sensor and ultrasonic sensor to enable real-time collision avoidance while the robot is driving, and the camera is mounted on the upper part of the robot so that the image of the robot in front of the robot can be remotely monitored It was sent to the operator.

Fig. 6. Autonomous Driving and Collision Avoidance Structure of Mobile Robot.

* Robot Monitor Structure

LCD monitor for video communication was

designed. The monitor was designed with the

fan / tilt moving structure and driven by two

servomotors.

Fig. 7. Robot Monitor Fan / Tilt Design.

Fig. 8. Fan / Tilt using servo motor.

Fig. 9. Remote Video robot Appearance.

We designed and built the robot to the world level to move the robot safely, and experimented to confirm the range of motion.

(Main Function Spec) Unit World Level Developed Level Speed control error cm/sec within ±10 ±10 Positioning errors cm within ±15 ±10 Ultrasonic sensor interface cm 5 2

Light sensor interface o o

Collision avoidance o o

Table 1. Acdeptable Goal

2. Electronic Stethoscope System

The remote image transmission robot was designed to work with a stethoscope to check not only the image but also the patient’s condition.

Auscultation is a diagnostic procedure performed to detect heart failure and digestive status defects.

The doctor listens to the sound of the stethoscope on the heart, lungs or intestine, the source of the sound. Fig. 10 shows the structure of the stethoscope.

Fig. 10. the structure of the stethoscope.

The stethoscope is transmitted to the ear through the earpieces as the sound source captured by the diaphragm moves up the tube.

This measured heart rate goes up the tube and you hear only low notes due to the HPF (Hight-Pass Filter). As a result, doctors cannot hear the original sound of the patient’s measured heart sound, and the sound of the low frequency band is mixed with the heart sound every time.

Fig. 11. control structure of the electronic stethoscope.

Recently, the electronic stethoscope is widely

used. It was solved by filtering and amplification

Fig. 14. Working Process of WebRTC Peer to Peer Communication.

as shown in Fig. 11. And while traditional stethoscopes only listen to doctors, electronic stethoscopes have the advantage of being able to store and record.

Fig. 12. Connect to the tablet of the electronic stethoscope.

The existing electronic stethoscope was connected to the robot and transmitted in real time as an video, and the receiver realized the medical consultation at the receiving end to realize a remote medical service requiring a stethoscope.

Fig. 12 shows the structure of an electronic stethoscope connected to a tablet. This system is installed on the video transmission robot manufactured earlier and transmits the video and the audio signal of the electronic stethoscope to the doctor.

3. Program development

It is important to obtain accurate patient’s medical information in real time for the treatment between the patient and the doctor with a remote video robot. Hardware configuration is important for accurate medical information, but software configuration is most important. Existing video communication is the structure that sends and receives the control signal of the robot through the TCP Socket while controlling by UDP Socket communication. However, it is true that accurate examination is difficult because of various problems in communication. However, in this study, we conducted experiments by securing secure communication using WebRTC platform.

Fig. 13 shows the structure of WebRTC.

Fig. 13. Architecture of WebRTC.

Browsers and mobile applications use Audio

and Video RTC (real-time communication) using

WebRTC through a simple API. The WebRTC

component has been optimized to best suit this

purpose. WebRTC-based web applications provide

rich real-time multimedia capabilities (think video

chat) on the web without plug-ins, downloads, or

installations, and help build a robust WebRTC

platform that works across platforms in multiple

web browsers. Fig. 14 shows a tank that transmits video and audio signals with the WebRTC Peer to Peer Communication procedure.

WebRTC Benefits:

• WebRTC is In-built in Firefox browser.

• Improved video and audio streaming.

• VP8 video codec and OPUS audio codec provides much less data transmission without packet loss.

Fig. 15 shows the process of running an Android program using Eclipse.

Fig. 15. Android video control program development.

Fig. 16. Android tablet Control screen.

Fig. 17. Android smartphone screen.

Fig. 18. video transfer experiment after mounting on robot.

After completing the development of the remote video robot, we made a character with a doll to have a friendly relationship with the patient when performing remote medical examination using a real robot.

Fig. 19. Equipped the character on the developed guide robot.

Ⅲ. Implementation and evaluation

In order to perform the experiment of the implemented system, the experiment is composed of the experiment of the robot control unit and the hardware control experiment.

1. Control board and video transmission experiment

In order to test the robot controller, a program

written in C language was downloaded to the

ATmega128 board. After the Bluetooth pairing

was performed, the program was set up to 0×31,

0×32, 0×34 ... 0×38 and the servo was driven

using data such as 0×40, 0×41, 0×42, 0×43. Then, a motion control experiment was performed using a remote control app.

Fig. 20. Experimental procedure of video transmission and motor drive.

2. Evaluation item experiment of development technology

For the evaluation of the developed product, the experiment was carried out at the temperature:

(20 ± 2) ℃ and humidity: (56 ± 5)% R. H. In this section, we will briefly summarize the results and data of the experiment.

Table 2-1. Experiment item and evaluation method.

No. Test Item Evaluation mothod 1 Video transmission

speed

⦁Video transmission time measurement.

2 Sound size capacity

⦁Speaker output noise measurement.

⦁Measure 1m from the front.

3 Operating time

⦁Check the remaining charge after operation / operation when fully charged.

4 Max Speed ⦁Maximum moving speed measurement

5 Battery usage indicator

⦁LED indication.

⦁Display remaining battery Check.

6 weight ⦁Weight measurement.

7 Robot height ⦁Robot height measurement.

8 After sensor response Stop speed

⦁Stop speed measurement after sensor response.

⦁Stop after detecting a forward object while moving

9 Stethoscope Frequency Measurement

Stored Stethoscope Measurements

Sample photo (front) Sample photo (rear)

Table 2-2. Evaluation results.

No. Test Items Target value Evaluation results 1 Video transmission

speed 20 Frame Video call with WebRTC platform 2 Sound size capacity 60 dBA/m 70.8 dBA

3 Operating time 8H 10H

4 Max Speed 30 cm/s 40.62 cm/s

5 Battery usage

indicator LED display LED display

6 weight 38 kg 17.80 kg

7 Robot height 120 cm 132.6 cm

8 After sensor response

Stop speed 1 s 0.20 s

9

Stethoscope Frequency Measurement

8000Hz 8000Hz

3-3. Experiment content

The experiment was carried out according to the evaluation method for each item, and the evaluation of the sound and speed of the robot and the frequency of the stethoscope sound were summarized.

3-3-1. Robot output sound measurement experiment

Speaker maximum output noise was measured during video call with tablet PC. The noise measuring room (width×length×height) was 4.98

×3.1×3,42 m and the distance from the robot was

measured at 1 m from the front. The noise

measuring instrument was performed by NA-27

(RION) and there were 10 measuring functions,

which made it easy to measure. The experiment

was performed three times to find the average

value.

[Measurement Data]

NO Room noise (dBA) Noise measurement (dBA) 1

41.4

70.1

2 69.5

3 71.4

Average value 70.8

[Evaluation results]

- Speaker maximum output noise result (69.5～

71.4) dBA.

Noise meter Noise measurement pictures Fig. 21. Experimental process photo.

3-3-2. Stop speed experiment after sensor response In order to improve the interaction between the patient and the doctor through the robot, the doctor responds quickly to the robot’s sensor response to speed up the interaction and environment recognition of the remote robot. The instrument used was a stopwatch and a HS-6 (CASIO) instrument.

[Measurement Data]

NO Reaction time (s) etc

1 0.22

2 0.18

3 0.17

Average value 0.20

[Evaluation results]

- Result of measurement of downtime after reaction of object detection sensor (0.17～0.22) s.

Photos before the move Still photo after detecting an object while moving Fig. 22. Experimental process photo.

3-3-3. Stethoscope storage experiment

Received the stethoscope sound transmitted from the remote video robot and added the function to store and play the stethoscope sound on the tablet PC, and experimented through the storage and playback screen on the screen. Fig. 23 shows a screen for storing and playing stethoscope sounds on the tablet screen.

Fig. 23. Experimental process photo.

Since the stethoscope sound must be transmitted and stored, it is encoded to 8000Hz 8bit mono type and it is configured to provide the function of playing and recording the stethoscope sound received by the doctor. The stethoscope sound sent from the patient can be stored in the doctor’s monitor and managed for video consultation history. The following figure shows the playing and stopping of the stored stethoscope. Through experiments, we could hear and diagnose a stethoscope sound through a remote imaging robot. In addition, since the images are transmitted using WebRTC, more than 20 frames of images have been acquired for remote medical examination, and it was confirmed through experiments that the images were transmitted safely without any breaks in the image system.

Ⅳ. Conclusions

In this study, we developed an video transmission robot system that can move freely in the structure proposed in the fixed PC for telemedicine service.

Even if the doctor and the patient are far away,

the video call is performed in real time as if they

are close, and the doctor can operate the robot

remotely to examine the patient’s condition in

various ways. For the configuration of the video

system, real-time video transmission is implemented using the H263 codec, which is a video communication.

Unlike the method of providing the login function to the server agent by using the TCP socket, the webRTC platform is configured to separate the video signal, the audio signal, and the control signal so that the video is not interrupted and is transmitted stably at a transmission speed of 20 frames or more per second. It confirmed that it became. In addition, since the electronic stethoscope system is mounted on the medical robot, the doctor will consult with the patient to check the planting in real time and receive and examine the patient.

Using the system implemented in this study, it is possible to monitor and remotely refer to elderly patients at home or patients discharged after heart disease surgery. Since the minimum sound quality required for accurate diagnosis by medical staff should be guaranteed, future research projects will compare the sound quality before and after the transmission of the stethoscope sound, and conduct continuous experiments and analysis for the effectiveness of the stethoscope image counseling system.

In addition, various researches will be needed, such as interactions that can safely control remote robots using various sensors and multi-touch technology mounted on smartphones or tablet PCs, and environmental awareness technology of remote robots.

References

[1] T. Cohen, “Medical and Information technologies.

converge,” IEEE Engineering in Medicine and Biology Magazine, Vol.23, No.3 pp.59-65, 2004.

DOI: 10.1109/MEMB.2004.1317983

[2] W. Yuan, D. H. Guan, S. Y. Lee, and H. J. Lee,

“Using Reputation System in Ubiquitous Healthcare,”

The Proc. of 9

IEEE Int’l Conference on e-Health Networking, pp.182-186, 2007.

DOI: 10.1109/HEALTH.2007.381626

[3] E. Pacchierotti, H. I. Christensen and P. Jensfelt,

“Human-Robot Embodied Interaction in Hallway Settings: a Pilot User Study,” IEEE International Workshop on Robots and Human Interactive Comm, pp.164-171, 2005.

DOI: 10.1109/ROMAN.2005.1513774

[4] S. J Baek, R. H. Lee, C. S. Yi, “Design and Development of A Systemic Structure to Ensure the Interoperability between the WebRTC-based Video Conferencing Systems and Heterogeneous Terminals,” KIISE Transactions on Computing Practices, Vol.23, No.4, pp.238-243. 2017.

DOI: 10.5626/KTCP.2017.23.4.238

BIOGRAPHY

Dae Seob Shin (Member)

1996：BS degree in Electronics Engineering, Howon University.

1998：MS degree in Electronics Engineering, Inha University.

2014：Ph. D. degree in Electrical and Biomedical Engineering, Hanyang University

2019～Present：adjunct Professor, Shin Ansan University

<Research Interests>

Image processing, neural network, Adaptive control, Signal

Processing, Embedded Control, Rehabilitation robots.