3. Multi-Sensory Media Service

The Ontology for a Multi-Sensory Media Service

Seungwon Oh and Minsoo Hahn

Digital Media Lab, Information and Communications University, Seoul 135-854 Korea {aegis901, mshahn}@icu.ac.kr

Abstract- Under the development of ubiquitous technologies, people need a novel media system. That means the multi-sensory media service which satisfies the five senses and makes people feel realistic. It is necessary to build knowledge about sensory effects and devices in order to provide the service. In this paper, we propose an ontology-based knowledge modeling. The proposed ontology model consists of Effect Ontology and Device Ontology. The Effect Ontology represents semantic information about multimedia contents and sensory effects synchronized with the contents. The Device Ontology represents semantic information about devices generating sensory effects. By using the model, we can infer sensory effects and their attributes from the predefined semantic information of multimedia contents. We proved the usefulness of our ontology model by showing results according to scenario-based experiments.

Keywords - ontology, sensory, media, reality, semantic, representation.

1. Introduction

Ubiquitous computing has become an integral concept in a variety of areas. Through ubiquitous technologies, the devices for daily use have had various functions and features over traditional functions and features. In other words, a traditional environment which had just predefined simple functions has been changed to an environment providing user-focused complex services with ubiquitous technologies. The remarkable development of ubiquitous technologies indicates the necessity of a rather improved novel media system.

Now, people can enjoy an audio-visual multimedia content at home equipped with advanced audio-visual devices.

However, it entertains only our eyes and ears. In the future, people will need the novel media service which satisfies the five senses and which lets people feel realistic [1]. In this paper, we will use the 'Multi-Sensory Media Service' as the terminology of this noble service. Figure 1 shows the architecture of the multi-sensory media service.

III I

As shown in figure 1, the service provides users with sensory effects generated by sensory devices and audio-visual contents.

A media player not only plays contents but also communicates with other sensory devices. In this case, a sensory effect means an effect to augment feelings by stimulating human sensory organs according to a multimedia content. A sensory device means a device to generate sensory effects such as light, wind, heat, incense, and etc.

In order to provide the multi-sensory media service, many related researches should be studied. First, noble contents have to be created and devices which can make sensory effects have to be developed. Next, the service requires information about sensory effects for representing multimedia contents and sensory devices for generating sensory effects. Finally, the service needs a mechanism which manages the information and controls sensory devices.

In order to meet the requirements for the multi-sensory media service, we propose an ontology-based semantic knowledge representation for the service. Ontologies are useful in building machine-readable semantic knowledge.

Ontology-based semantic knowledge has important advantages. First, it is easy to share knowledge between heterogeneous devices. Next, it has a good structure to expand knowledge with increasing contents and devices. Last, it is useful for knowledge reuse and logical inference based on a semantic structure.

Therefore, we proposed an ontology-based knowledge modeling. The proposed ontology model represents semantic information about multimedia contents, sensory effects synchronized with the contents and sensory devices.

2. Related Work

Ubiquitous computing is a computing paradigm that a computing power becomes invisible in our lives. We can easily access to computing power anywhere, anytime and anyway through intelligent interfaces. Mark weiser says about Ubiquitous computing as follows: "Its highest ideal is to make a computer so embedded, so fitting, so natural, that we use it without even thinking about it." [2]. One of the main goals of ubiquitous computing is to support relevant information when it is needed. Therefore, in order to provide a service in ubiquitous environment, a knowledge modeling which represents the context about a user or environment is necessary.

Ontologies are widely used for modeling context in ubiquitous environment because an ontology is useful in knowledge sharing, logic inference and knowledge reuse [4].

There are many researches that utilize ontology to provide context-aware services, for examples, CoBrA-ONT (2003) [3],

CONON (2004) [4], SOUPA (2004) [5], GAS (2005) [6], and Ubiquitous User Modeling [7].

First, CoBrA-ONT (Context Broker Architecture-Ontology) represents the infonnation about places, agents, events and their associated properties in an intelligent meeting room domain [3]. In order to support pervasive context-aware systems, CoBrA-ONT was developed as a component of the Context Broker Architecture.

Second, CONON (Context Ontology) is an ontology which has a hierarchical structure. In pervasive computing environments, CONON defines context modeling and supports logic-based context reasoning [4]. CONON consists of an upper ontology and a domain-specific ontology. The upper ontology represents general concepts about domain-independent context and the domain-specific ontology represents domain-dependent concepts for extensibility.

Third, SOUPA (Standard Ontology for Ubiquitous and Pervasive Applications) was developed in order to model and support pervasive computing applications [5]. SOUPA was implemented with OWL. In order to represent intelligent agents, SOUPA includes various concepts such as user profiles, actions, events, space, time, beliefs, desires, intentions, and policies for security and privacy. In order to represent the above concepts, SOUPA references other ontologies, for example, the COBRA-ONT, the Regional Connection Calculus, the Friend-Of-A-Friend ontology, the spatial ontologies in OpenCyc, the DAML_Time, the MoGATU BDI ontology, and the Rei policy ontology.

Fourth, GAS (Gadgetware Architectural Style) Ontology is an ontology that was developed in order to represent the semantics of the main concepts of a ubiquitous computing environment and define their inter-relations [6]. The GAS Ontology consists of the GAS Core Ontology and the GAS Higher Ontology. The GAS Core Ontology describes the common language to communicate and/or collaborate between different domains. The GAS Higher Ontology represents both the description ofeach domain and its acquired knowledge.

Last, Heckmann proposed a complete framework to realize the new ubiquitous user modeling [7].

SITUATIONALSTATEMENTS and USERML were developed on the syntactic level, while UBISWORLD ontology and GUMO, the general user model ontology were

developed on the semantic level. The

SITUATIONALSTATEMENTS represents descriptions of situations like user model entries, context infonnation or low-level sensor data.

Previous ontology researches focused on the user's state and behavior for context-aware services. Many useful ontology models were implemented for context-aware services.

However, they are not sufficient to provide the multi-sensory media service.Itis necessary to develop the novel ontology model that includes not only user context knowledge but also multimedia content knowledge, service context knowledge and sensory device knowledge. Also, the ontology model has to allow for an expansion possibility and an inference possibility. In this paper, we proposed the new ontology model considering contents and devices for the service.

3. Multi-Sensory Media Service

The multi-sensory media service has the overall architecture as shown in figure 1. As shown in figure 1, the service provides users with sensory effects generated by sensory devices and audio-visual contents. In order to provide the service, first, a fonnal communication protocol must be defined in order to communicate between the player and sensory devices. Second, the standardization of stimulus which makes people feel realistic must be defined. Based on the standardization, many sensory devices can stimulate people to feel realistic. Third, knowledge which represents infonnation about multimedia contents and sensory devices is necessary. According to the given multimedia contents infonnation, the player defines sensory effects synchronized with the contents and sensory devices provide people with the sensory effects. Although not representing in figure 1, the context ofa user or environment may be considered because of the nature of the service.

The service is not a predefined simple service but a flexible complex service based on the infonnation of multimedia contents and sensory devices. Also the service consists of different combinations of sensory devices and provides various multimedia contents. Considering the characteristics of the service, the knowledge which has the following three features is necessary for the service. First, the knowledge has to be interpreted semantically for the semantic interoperability among different devices. Second, the knowledge has to be standardized for various devices to share and exchange the knowledge each other. Last, the knowledge has to be possible to infer new infonnation. Because the knowledge cannot prepare infonnation for all situations, the service must be possible to infer new infonnation based on the usable knowledge.

:lickwiad. ...i8ccae. __

Figure 2. Ontology-based MUlti-Sensory Media Service

Therefore, in this paper, we propose an ontology-based semantic knowledge representation as a method of representing infonnation for the multi-sensory media service.

Figure 2 shows the ontology-based service architecture. In figure 2, the Effect Ontology means the semantic representation about sensory effects and multimedia contents.

The Device Ontology means the semantic representation about sensory devices installed in the service environment. The Inference Engine means the mechanism to infer necessary infonnation from given infonnation.

ISBN 978-89-5519-139-4 -1026- Feb. 15-18, 20091CACT 2009

and their relationships. Also we added concepts for the sensory effects to the Effect Ontology.

CIIIS Property

I~

The Place Class represents the place information about each scene ofmultimedia contents. In this case, the role ofthe Place Class is to define influences on Heating and Cooling effects of each scene. Although the Place Class meant location information in many different ontologies, we designed five subclasses of the Place Class according to climate as shown in figure 6. Therefore, we can define influences on the Heating and Cooling effects according to the climatic characteristics of places.

The Time Class also represents the time information about each scene of multimedia contents. The time information also has influence on the sensory effect. The Time Class has the Season Class and the Day Class as subclasses as shown in figure 7. The Day Class has the following individuals: dawn, morning, forenoon, noon, afternoon, evening and night. The Season Class has the following individuals: spring, summer, fall and winter.

c:::>

---+ Property .•••.•• Subclus

The Scene Class represents the description of multimedia contents. The information for the multi-sensory media service is organized by the unit of a scene. The Scene Class has relations with different classes such as Place, Time and Element as shown in figure 5. In other words, The Scene Class includes the information of place, time and elements predefined by a content producer.

hasElcmcnt

~

~Macc~

hasTirne

Figure 5. Scene Class

~

Figure 4. Physical Environment for the Multi-Sensory Media Service In summary, an ontology is suitable for representing knowledge necessary for the multi-sensory media service because of the following advantages:

- A formal and explicit knowledge representation - A machine-readable semantic knowledge representation - Knowledge sharing between heterogeneous systems - Easy knowledge expansion

- Knowledge reuse and logical inference based on a semantic structure

Figure 3 and figure 4 show the physical experiment environment for the multi-sensory media service. To stimulate user's five senses, we constitute various devices such as a display for video, a 5.1 channel speaker for sound, a light for brightness, a heater for temperature, a fan for wind and a perfume spray for smell. Practically, we constructed the system which satisfies user's senses except for taste.

.,...-

Figure 3. Physical Environment for the Multi-Sensory Media Service

Subclus

I::?

CT:::?

0··C~v

Figure 7. Time Class

The Element Class represents the subject of sensory effects, for example, Sun, Wind, Water, Fog, and so on. In other words, we define sensory effects for representing element information

4. Effect Ontology

The Effect Ontology represents the information of sensory effects synchronized with multimedia contents. Appendixes show the whole structure of the Effect Ontology. Before representing the sensory effects, the content information must be defined ahead. A scene means the minimum unit of multimedia contents like movies. A producer puts content into a movie by the scene. Each scene has basically several components such as place, time, actor, object, event, and so on.

Therefore, in the Effect Ontology, we defined the concept of a scene and then described many concepts included in scenes

of each scene predefined by a content producer. Figure 8

shows the structure of the Element Class.

5. Device Ontology

-+ Property

••••••• Subdus

<::> c.

0 ^Value

-+ ^Property

...•

~ ^o

^claas

ATTRIBL'TE

huDircdion

...I 1

hasEfrect

Figure 10. Device Class

~

Figure 10 shows the structure of the Device Class. The Device Class has the information of installed position in the service environment by the hasDirection relation. Also, the Device Class has the information of possible generating sensory effects and effect attributes.

The Device Ontology represents the information of sensory devices generating sensory effects in the multi-sensory media service. Appendixes show the whole structure of the Device Ontology. We defined the Device Ontology in order to describe each sensory device. In other words, the Device Ontology defines characteristics of sensory devices installed in the service environment.

The Device Ontology shares the Attribute Class and the Effect Class in the Effect Ontology.

6. Ontology Development and Evaluation

The Web Ontology Language (OWL) is a language for defining and instantiating Web ontologies. In order to expand the limited expressiveness of RDF Schema, a more expressive OWL has been defined by the World Wide Web Consortium (W3C) [8]. So far, OWL is the most widely used and outstanding ontology language in terms of expressiveness and reasoning capabilities. That's why we selected OWL among many ontology languages.

In order to build OWL-based knowledge, we used Protege 3.4 [9] as an ontology editing tool. Protege is a free, open source ontology editor and knowledge-base framework.

Protege supports GUI environments for building ontologies.

As ontologies are widely used in a variety of areas, ontology evaluation has become an important issue. The goal of an ontology evaluation is to reuse a useful ontology which defines correctly an interesting domain. To evaluate an ontology, the following criteria were identified: consistency, completeness and conciseness [10].

First, consistency means whether contradictory knowledge cannot be inferred from other axioms. Second, completeness means whether knowledge can represent explicitly a related domain. Last, conciseness means whether unnecessary redundancies or definitions can exist.

In order to prove our ontology, first, we utilize Protege [9]

with a consistency evaluation tool. Protege also supports the correctness of syntax and logic in building ontologies.

Figure 8. Element Class

The Element Class has the effect information related to each element by the hasEffect relation and direction information by the hasDirection relation. In figure 8, the dotted classes represent the classes that are not fully considered in this paper.

The Effect Class represents the sensory effects to stimulate people. The information of the Effect Class is inferred by the information of a scene. Figure 9 shows the structure of the Effect Class. In figure 9, the dotted classes represent the classes that are not fully considered in this paper.

We can define each effect according to each element. Each effect has attributes defined by information belonging to each scene. The Attribute Class represents the attributes what effects can have. We defined the following attributes:

BreadthHorizon, BreadthVertical, Color, Frequency, RotateHorizon, RotateVertical, and Temperature. The BreadthHorizon means that an effect spread horizontally. The BreadthVertical means that an effect spread vertically. The Color defines a color of an effect. The Frequency defines the number of occurrences of a repeating effect per unit time. The RotateHorizon means that an effect rotates horizontally. The RotateVertical means that an effect rotates vertically. The Temperature means that an effect changes circumference temperature.

Also, the Effect Class can have direction and intensity information. The direction of an effect is inferred from the direction of an element defined by a content producer. The intensity of an effect is defined as the relative value inferred from information belonging to each scene.

"..---.) e

'~:-~~mm

---l

cdJ::~~::~:: ~~ D1RECno~

LIGHT \ ••••

1

eaLj···.~:~

CtmA~:>

Figure 9. Effect Class

ISBN 978-89-5519-139-4 -1028- Feb. 15-18, 20091CACT 2009

As shown in above tables, we can confirm that proper sensory effects are inferred by the proposed ontology model.

Our ontology model can fully represent multimedia contents and sensory devices for the multi-sensory media service. Also, ifnew contents and devices are added, the ontology model can be easily expanded by adding just related information without its structural change because of the model design consideration. Therefore, a content provider can determine the sensory effects synchronized with contents by defining simple scene information in order to provide the multi-sensory media service.

Second, we show the result of various scenarios. Scenarios consist of different scenes of multimedia contents. We explain the inferred each effect information from each scene.

The experiment scenarios are as follows. The following figures are screenshots of scenes. Each table shows the inferred information according to predefined information in each scene.

Figure 11. Dessert Scene

Table1. Semantic Representation of the Dessert Scene

Given Information

Place Time

Temperate Afternoon

Element Tornado

Inferred Information Effect Attribute Fan RotateHorizon

RotateVertical

Given Information Inferred Information

Place Time Element Effect Attribute

Dry Noon Sun Heating Temperature

Light Color

Figure 12. Evening Glow Scene

Table 2. Semantic Representation of the Evening Glow Scene

Given Information Inferred Information

Place Time Element Effect Attribute

Temperate Evening Sun Light Color

BreadthHorizon

Figure 13. Snowfield Scene

Table 3. Semantic Representation of the Snowfield Scene

Given Information Inferred Information

Place Time Element Effect Attribute

Frigid Morning Sun Cooling Temperature

Light BreadthHorizon BreadthVertical Color

Figure 14. Tornado Scene

7. Conclusion

We discussed a framework for the Multi-Sensory Media Service. And then, we introduced our ontology model for the service. The proposed ontology model consists of Effect Ontology and Device Ontology. The Effect Ontology represents semantic information about multimedia contents and sensory effects synchronized with the contents. The Device Ontology represents semantic information about sensory devices generating sensory effects. With our ontology model, the noble service can make users feel realistic and entertain users.

With the proposed ontology model, we infers sensory effect information from the given multimedia contents information.

In order to evaluate the proposed ontology model, we demonstrated various scenarios. Each scenario is a scene of a multimedia content such as a movie. After we defined the semantic information about the scenes, we confirmed inferred results. With scenario-based experiments, we proved the effectiveness of the proposed ontology model.

Therefore, by describing the simple scene information, a content provider can define the sensory effects for the multi-sensory media service.

8. Future Work

As a future work, we are planning to consider the context of a user and environment. The satisfaction of sensory effects may be under the influence of user preference and physical information of service environment, for example, the size of space, the type of space, and so on. Considering these factors, a personalized multi-sensory media service will be possible.

Also, we will research about inferring alternative sensory devices for generating similar sensory effects when matched sensory devices do not exist.

ApPENDIXES

•

The Effect Ontology

The Device Ontology

REFERENCES

[1] C. Timmerer. A Summary of the Representation of Sensory Effects(RoSE). 2008.

[2] M. Weiser, "The Computer for the 21st Century," Scientific American 265, Nr. 3, S. 94-101,1991

[3] H. Chen, T. Finin, and A. Joshi. An ontology for context-aware pervasive computing environments. Knowl. Eng. Rev., vol. 18: 197-207,2003.

[4] X. Wang, D. Zhang, T. Gu, and H. Pungo Ontology based context modeling and reasoning using OWL. Pervasive Computing and Communications Workshops, 2004. Proceedings of the Second IEEE Annual Conference on: 18-22, 2004.

[5] H. Chen, F. Perich, T. Finin, A. Joshi. SOUPA: standard ontology for ubiquitous and pervasive applications. Mobile and Ubiquitous Systems:

Networking and Services, 2004. MOBIQUITOUS 2004. The First Annual International Conference on: 258-267,2004.

[6] E. Christopoulou and A. Kameas. GAS Ontology: An ontology for collaboration among ubiquitous computing devices. International Journal of Human-Computer Studies, vol. 62: 664-685,2005.

[7] D. Heckmann. Ubiquitous User Modeling. PhD Thesis. 2005.

[8] D. L. McGuinness, F. Harmelen, "OWL Web Ontology Language

Overview," W3C Recommendation,

http://www.w3.org/TR/owl-features/, 2004.

[9] Protege, http://protege.stanford.edu/.

[10]A. Gomez-Perez, "A Framework to Verify Knowledge Sharing Technology," Expert Systems with Application. Vol. 11, N. 4. pp.

519-529, 1996.

ISBN 978-89-5519-139-4 -1030- Feb. 15-18,2009 ICACT 2009