Development of living body information and behavior monitoring system for nursing person

monitoring system for nursing person

Ai Ichiki

․ Hidetoshi Sakamoto

․ Yoshifumi Ohbuchi

*

Kumamoto University

ABSTRACT

The non-contact easy detecting system of nursing person's body vital information and their behaviors monitoring system are developed, which consist of “Kinect” sensor and thermography camera. The “Kinect” sensor can catch the body contour and the body moving behavior, and output their imaging data realtime. The thermography camera can detect respiration state and body temperature, etc. In this study, the practicability of this system was verified.

Keywords: Engineering Education

I. Introduction ¹⁾

The items of the respiratory state, the body temperature and the behaviors for the nursing are very important observation items. Especially, the state of respiratory monitoring is an important item which is indispensable　in the detection of the “Sudden Infant Death Syndrome”(SIDS) and the apnea syndrome. Polysomnograph(PSG) is widely used in determining respiratory states. However, the largest shortcoming of PSG is that it is expensive and its low tolerance for the nursing patients and infants　by relatively high invasiveness of the PSG .

In this study, the easy monitoring system of non-contact respiratory state and body temperature for early detecting the SIDS and the apnea syndrome was proposed .

The validation for practical use as follows was carried out.

1) Verification of the posture detecting performance of

“Kinect” sensor.

2) Automatically detecting the state of respiratory by thermography camera.

3) Development and verification of the new monitoring system with “Kinect” sensor.

Received 21 June, 2014; Revised 21 June, 2014 Accepted 30 July, 2014

† Corresponding Author: [email protected]

II. Non-contact monitoring system of state of respiratory state

The monitoring system outline of non-contact respiration is shown in Fig 1. The initial system consists of the infrared camera, CCD camera, room temperature measurement unit and two personal computers. The CCD camera detects a patient posture and his face contour, who is lying in bed. The camera'a informations are used for the thermography camera control. The thermography camera measures his respiratory and his body temperature.

Fig. 2 shows an example of the tracking image of the face detected with CCD camera. Recognizing his head area by CCD camera, the face area is automatically displayed by image processing. Next, the face area is extracted from head area by skin color image matching. The mouth area is initially set up manually. The face position is decided

Fig. 1 The body vital information monitoring system

Fig. 2 The example of Posture tracking system by CCD camera

on position of mouth area by the face area's pattern matching.

Fig. 3 shows the picture of image processing of infrared image. The infrared camera detects the thermal information around the face. By image processing of these thermal information, the state of respiratory and the body temperature can be obtained. The head region and face area are detected by binary pictures of the face in infrared camera. In setting up the mouth area in the face, the face digital image obtained from CCD camera was used.

The state of respiratory is detected by using temperatures fluctuate in the current of air by breath around the mouth.

However, this CCD camera has some problems about the face recognition and the posture detection. So, we replaced this camera with “Kinect” sensor for solving these problems. Because this sensor can recognize the shape and depth of the head, the image of the head can be quickly monitored. The “Kinect” sensor is shown in Fig. 4.

Fig. 3 The picture of image processing by infrared camera

Fig. 4 “Kinect” sensor

III. Validation of the monitoring system

1. Verification of the posture detecting performance of “Kinect” sensor

a. Detection ability of posture by “Kinect” sensor.

In this study, CCD camera was replaced with “Kinect”

sensor, and the verification about two items, that is, the face recognition and the posture detection, were carried out. Table 1 shows the comparison between CCD camera and “Kinect” sensor. The front face tracking image is shown in Fig. 5 By using “Kinect” sensor, the following three items were improved.

• Auto detection of mouth area.

• Skip of skin color image process.

• Wide traceability of face rotation.

Table 1 Comparison between CCD camera and “Kinect” sensor CCD camera "Kinect" sensor Face recognition Setting skin color

extraction Infrared distance sensor Posture detection Relative position in

mouth area to face area

The angle of face rotation

Fig. 5 The front face tracking image by "Kinect" sensor

Table 2 Face recognition angle Distance from sensor and face/

Rotation velocity 60cm 70cm 80cm 90cm 100cm 110cm 120cm

45deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚

90deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚

180deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚

the face was changed, and the face rotation speed and the face recognition angle were examined. Table 2 shows the results. From this table, the recognition angle is independent of the distance from sensor and face.

2. Evaluation of automatically respiratory state detecting by infrared camera.

a. Automatic setting of threshold by infrared camera.

• Threshold 1(two-valued of head area)

The histogram of two-valued image shows the bimodality. The two-valued image separate the face area and the background clearly. We calculated of threshold of the valley part in histogram with mode method. Fig. 6 shows an image processing as an example of thermography camera. Fig. 6 (a) and (b) show the origin picture and the picture after image processing respectively.

• Threshold 3( Breath detection)

The value of threshold 3 fluctuates widely by room temperature and body temperature. We tried every threshold value from 100 to 200. When the breath detection flag appeared, we decided the value of threshold 3 and determined as threshold. Fig. 7 shows an example of image processing in infrared camera.

b. Validation of threshold 1 by infrared camera.

Here, when the body temperature change, the variance of the threshold 1 was examined. In threshold 1 obtained by above trial, the face recognition ratio is 100% when the body temperatures are from 30 to 37 degree Celsius.

(a) The face origin picture

(b) The picture after face image processing Fig. 6 Image processing example of thermo camera

Fig. 7 Nasal breathing

(a) RGB image

(b) Depth image

(c) Infrared image Fig. 8 Image processing example of “Kinect” sensor

IV. Development of the new monitoring system by using “Kinect” sensor

In this research, CCD camera was replaced with “Kinect”

sensor (4.1.1) for the system improvement because of there were some problems in the face recognition and the posture detection. In the result, using “Kinect” sensor, it was found that the face recognition, posture and respiration are non-contact measurable by non-contact . So, the new monitoring system by using only “Kinect” sensor was developed(System 2) and compared with conventional monitoring system (System1).

Fig. 8 shows image processing of “Kinect” sensor. Fig. 8(a) is RGB image, Fig. 8(b) is Depth image and Fig. 8(c) is Infrared image, respectively.

1. The monitoring system of “Kinect” sensor

“Kinect” sensor detects the patient's posture who is lying in bed and measures his respiratory state with RGB camera and depth sensor. In the posture measurement, the multi-“Kinect” sensors are available. This system

(a) Posture measurement (b) Respiration measurement Fig. 9 The body vital information monitoring system (System2)

use two sensor and one is set at foot side and the other is head side. Fig. 9 shows the body vital information monitoring system (System 2). Fig. 9(a) is posture measurement, Fig. 9(b) is respiration measuring, respectively.

2. Development of the monitoring system with

“Kinect” sensor(System2)

The posture measurement system tracks the body motion and saves the image when the body moves.

Respiration measuring system detects the breathing state and records the breathing count every minute.

a. Development of the posture measurement system The posture is obtained by measuring the shortest distance from “Kinect” sensor to nursing person. This detection use the distance measuring function of “Kinect”

sensor. Fig. 10 shows the image processing of the posture measurement system. The RGB camera of “Kinect”

sensor measures the body from foot side and head side.

The light blue point of the depth image is the minimum

Fig. 10 The image processing of the posture measurement

system

Fig. 11 The night measuring with the infrared camera

Fig. 12 The recoded data

depth and detects his posture. In the measurement at night, RGB camera changes into the infrared camera equipped with “Kinect” sensor. The night measuring image with the infrared camera is shown in Fig. 11.

Also, this system records the RGB processing image when the body moves. RGB image recording is decided by the movement of the minimum depth. The recording data is recorded the current data and time. Fig. 12 shows the recorded data.

b. Development of the respiration measuring system We developed the respiration measuring system by using RGB camera and depth camera of “Kinect” sensor.

The image processing is shown in Fig. 13. The breathing state is detected by movement of depth change of the chest. Fig. 14 shows the flowchart of the breathing detection.

In the measurement at night, RGB camera changes into the infrared camera. In the night measuring, RGB camera is switched the infrared camera. The night measuring with the infrared camera is shown in Fig. 15. Also, the system records the breathing count every minute. Fig. 16 shows the logging data of the starting measurement time and the breathing count.

Fig. 13 The image processing

Fig. 14 The flowchart of the breathing detection

Fig. 15 The night measuring with the infrared thermo camera

Fig. 16 The logging data of the breathing count

V. The comparison of system 1 with system 2

In the posture measurement of the system 1, the traceable limit of face rotation is 45～55 degree. In the system 2, the traceable limit of face and body rotation are 90 degree.

In the respiration measuring system, we evaluated the precision of the breathing state. The breathing data of system 2 corresponded to the standard breathing. The accuracy of measuring the breathing improved by using “Kinect” sensor.

VI. Conclusions

<System 1>

• Using “Kinect” sensor instead of CCD camera, the system improvement was carried out for practical use as follows.

(a) Automatic detection of mouth area and face area.

　 (b) Skip of the skin color detection process . 　 (c) Improvement of traceable limit of face rotation.

• By calculating threshold 1 and 3 automatically, the setting time for measurement has been greatly shortened.

<System 2>

• “Kinect” sensor is an effective sensor to the posture measurement and the respiration measurement.

• The performance of the posture measurement and the respiration measuring is improved by replacing the CCD camera into “Kinect” sensor.

• By combining the system 1 and system 2 organically, the higher precise non-contact monitoring system is obtained.

References

1. Nobuhiro YOSHITAKE, An Implementation of Posture Detection Functions to Inpatient Monitoring Systems using Kinect, IPSJ SIG Technical Report (2013), 1－8

2. Yuhki TAKAHASHI, Development of System for Prevention of Midnight Prowl Using Kinect, 2013 The Institute of Industrial Applications Engineers Japan (2013), 42-43 3. Mariko AKIMOTO, A sheet-type device for home-monitoring

sleep apneas in children, Sleep and Biological Rhythms (2011), 103-111

4. Tomohito HAYASHI, Study of Non-Restrictive Sleep Monitor With Air-Matt Sensor, The Japan Society Mechanical Engineers

(2002), 71-74

5. Shunji HYUGA,”Let’s make the Kinect for Windows application! ”(2012)

6. Tomoaki UEDA, The sensing world changing by “Kinect”

sensor, http://www.neo-tech-lab.co.uk/arsensing/

Ai Ichiki

Engineer of TOSHIBA Cooperation LTd.

Received BS (2012), MS (2014) in Mechanical System Engineering from Kumamoto University, Japan. Her work experiences are Product Engineer of TOSIBA Coopration LTd.

(2014), Her current research focuses on the production technology of turbine for power electronics.

Phone: +81963423735 Fax: +81963423729 E-mail: [email protected]

Hidetoshi Sakamoto

Professor, Doctor of Engineering, at the Mechanical System Division, Graduate School of Science and Technology, Kumamoto University, Japan. He is received Master degree of Mechanical Engineering by Kumamoto University, Japan in 1977, and got his doctor's degree of Engineering from Kyushu University, Japan in 1991. He was the Kyushu Branch Head of The Society of Materials Science, Japan, 2006-2008 and now the Director of Infrared Thermography Committee of The Society of Experimental Mechanics, Japan. He is also a member of WIT international Science committee of “Computational Mechanics and Experimental Methods”, “High performance Structures and Materials”, “Contact and Surface Treatments”, and an editorial board member of WIT International Journal of Modeling and Simulation He is the director of International conference on Far East Fracture of Strength. The field of his research includes Solid Mechanics, Computer mechanics, Sheet metal forming, High-speed fracture and deformation analysis, Biomaterial materials strength evaluation and Engineering educational support program development, etc.

Phone: +81-96-342-3735 Fax: +81-96-342-3729

E-mail: [email protected]

Yoshifumi Ohbuchi

Associate Professor, Doctor of Engineering, Creative Engineering and Design Education Center Faculty of Engineering, Kumamoto University, Japan. He is received Master degree of Mechanical Engineering by Kumamoto University, Japan in 1985, and got his doctor's degree of Engineering from Tokyo Institute of Technology, Japan in 2002. He was a member of Kumamoto University from 1985 to 1993. He was Visiting Researcher of Tokyo Institute of Technology from 1993 to 1994. He was Associate Professor of Fukuoka Institute of Technology, Japan from 2003 to 2005. Since 2005, he is a Creative Engineering & Design Education Center Faculty of Engineering, Kumamoto University, Japan. Interesting Research Area are Engineering Education, Creative Engineering and Design, Succession Methods of Traditional Craftsmanship and Skill, Metal cutting and grinding simulation.

Phone: +81-96-342-3732 Fax: +81-96-342-3729

E-mail: [email protected]