Categorisation of Audience Relationship between Action and Visualisation in Interactive Art Installations

(1)

Categorisation of Audience Relationship between Action and Visualisation in Interactive Art Installations

Je-ho Oh, Chung-kon Shi GSCT, KAIST, South Korea

Abstract

The aim of this paper is to categorise audience participation in interactive art according to a visualisation method. The research studied categories of audience in interactivities and applied artworks at the 7th Seoul Media Art Biennale. To establish the method of categorisation, the participation of the audience is classified into two factors: ‘to act’ or ‘to be represented’

in the visualisation realm. Based on these two factors, the research proposed a new graph for visualisation of human action in interactive art, categorising interactive art into four categories. The audience in interactive artwork performs the following roles: object, coordinator, transformer, character. Ultimately, this research predicts the meaning of the human being in virtual reality and utilises basic categories to visualise audience actions.

Keywords--- interactive art, categorisation, audience visualisation, audience action, object, coordinator, transformer, character.

1. Introduction 1.1. Overview

The aim of this research is to classify audience participants based on relationships between action and visualisation in interactive art installations.

1.2. Background and Motivation

Interactive art consists of three factors: audience action, installation, and contents. It seeks expression of the embodiment of audience action. The audience undergoes experiences in the interactive artwork, becoming a digital being or character. Their action is a form of two-way communication between a real situation and a virtual situation; members of the audience react to real objects or virtual objects.

The main difference from earlier artwork, either visual art or formative art, is that contents are transformed in interactivity by the audience. Depending on factors of

action from the audience, the content output of interactive art changes. The author considers factors of action and expresses the embodiment using these factors.

Visualisation is a method to express embodiment of audience action, creating defamiliarisation of experience.

These processes create the aesthetic of interactive art.

Until now, the main issue in interactive art has been how to trigger interactive art from audience action. It is important not only to develop the audience participant system, but also to address how to visualise audience action in interactive art.

Is interactive art included in one of the visual arts, formative arts, or fine art? Through the development of media technology, the visual arts have been changed into various categories. Digital media mediated by computer science expands the territory of visualisation, creating questions about how to apply visualisation better as well as how to address issues of technologies upgrade. The use of vision, motion, and other sensing technologies to trigger autonomous actions and behaviors means that artworks can interact with audiences in lifelike and unpredictable ways [1].

Interactive art is one of the visual arts mediated by digital technologies and included in the formative arts. In the digital age, visual art as a convergence for multimedia and audience experience expands the auditory, olfactory, palate, tactile, and synesthesia senses.

Mixed reality (MR) environments are progressively upscaling the everyday convergence of the technological and the human, dissolving the usual boundaries found at the interface of the screen, as digital images can enter the user's space. In the interaction relationship between users and digital characters, each affects the way the other behaves and interprets the world [2].

The audience participates in the artwork through their hands, eyes, or movements of the entire body, triggering operation of the art implement.

How will audience action be represented by digital technologies? How will the audience react to the installation and how will that reaction be expressed by visualisation through digital media? The artwork visualisation exploration process contains not only information, but also the transmission meaning and aesthetics.

2013 17th International Conference on Information Visualisation

357

555

(2)

It is time to apply an accurate analysis to classify audience action and to express visualisation technology.

1.3. Research Scope

This research focused on artwork which expresses an embodiment of audience action in visualisation. It deals with the sense of vision, not the relationship with the audience and not the other senses—auditory, olfactory, palate, tactile and synesthesia.

2. Previous Work

2.1. Woo’s Categorisation and Limitation

Woo’s research proposed a new concept of interactivity based on four categories related to degrees of communication and the degree of physical [3]. This research’s new concept classifies control-based interaction and communication-based interaction by medium, and liminal interaction and transitive interaction by physical sense.

However, these categories did not consider the audience’s action and visualisation. The audience only controlled an interface and communicated with installations. Woo’s research evaluated the degree of interaction, but did not consider visualisation of interactive action. Audience action with an interactive interface produces an output. In this research, audience experience is not to be represented by the installation.

2.2. Fel’s Categorisation and Limitation

Fel’s research suggested a basic concept of communication between the audience and installation [4]

[5]. Fel proposed four categories: response, control, contemplation, and belonging by embodiment. These factors feature output by the audience communicating with the object (installation).

The limitation of this research is that it does not propose the output of a relationship between user and object. Fel studied the audience’s experience through the audience itself and the object; he did not consider an audience represented by installations.

2.3. Kim’s Categorisation and Limitation

Kim classified interactive art by two categories:

communication style and mirror style, based on the effect of output [6]. Communication style means that the audience communicates with the installation. In this style, the audience as participator becomes an operator in the art installation. Mirror style means that the audience is represented by the installation, like a reflection in a mirror.

The aesthetic in interactive art is that the audience experiences their own action or watches their own image through the artwork in a situation of narcissism.

This categorisation addresses only two elements of interactive art. It provides a starting point for research, but the categories are too large. In mirror style, the audience

is only to be represented in narcissism. This fails to address variety as the aim of visualisation. This representation is not from action. In the communication style category, it considers communication audience-to- audience or audience-to-installation. The development of digital technology, reflection, and communication is complicated and needs to consider multimodal features.

3. Research Goals 3.1. Research Needs

The experience of the audience in interactive artwork is converted by a computer system or digital devices.

Visualisation in the audience experience is an important factor determined by the audience participant or aesthetic moment. No research deals with the interactive experience changed into visualisation.

It is useful to analyse the interrelation of audience action and visualisation in interactive artworks. This analysis will help the author create interactive artworks and the audience to participate in the interaction situation.

3.2. Research Goals

The aim of this paper is to categorise audience participation in interactive art according to the relationship of audience action and visualisation.

First, this research analyses previous methods of categorisation and presents a new method of categorisation.

Second, this research applies a new method to new trends in artwork and extracts basic factors of categorisation.

Third, this research classifies the roles of audience interactive art based on new categories.

Fourth, this research suggests the audience role in interactive art visualisation.

Fifth, this research predicts the relationship with action and visualisation in a virtual or augmented-reality world.

4. Case Study—Research on Studies and Artworks

4.1. Aim of the Case Study

The aim of this case study is to observe the relationship with action and visualisation in recent and masterpiece works of interactive art. Based on this examination, this research revised previous methods of categorisation in interactive art.

4.2. Method of Analysis

The chapter on previous work analyses three categories dealing with interactive art. Basic factors in interactive art were extracted to frame the analysis of audience action and visualisation: installation, audience, and the visualisation and interpretation view. These

358 358 358 556 556 556 556

(3)

factors are subdivided by two variables: the first, the fundamental element, and the second variable is the application of the first element.

Table 1 Method of Analysis

Factors First Variable Second Variable Installation

view

Installation type

Control-based or communication- based interaction Audience view Audience

action

Audience role Visualisation

view

Visualisation output

Audience representation Interpretation

view

Meaning of artwork

Aesthetics of artwork

4.3. Artworks Collection

The new method was applied to these four interactive artworks:

1) Be Your Own Souvenir by blablabLAB (2011) 2) Spell on the City by NMARA (Zune Lee & Kenny Kyungmi Kim) (2011)

3) In Between by HYBE (2012)

4) Inter-Scenery by Joon Y. Moon (2012)

These works were exhibited at the 7th Seoul International Media Art Biennale, “Spell on You,” 11 September 2012-4 November 2012 at the Seoul Museum of Art and Digital Media City Gallery [7]. Four interactive artworks were exhibited out of a total of 49 artworks.

This research analyses four artworks through the revised method. However, this case study omits an interpretation view to exclude a subjective analysis.

4.4. Case Study Results 1) Be Your Own Souvenir

Figure 2 Be Your Own Souvenir [7]

This artwork uses a technical process that measures the audience with 3D scanners and uses a 3D printer. In this process, the audience poses as models and various posed objects are made by the system. The 3D printed object is then given to the audience as a souvenir.

The analysis of this artwork is as follows:

Table 2

Analysis of Be Your Own Souvenir Factors First Variable Second Variable Installation

view

3D scanner, 3D printer, Kinect, ABS plastic

Control-based interaction

Audience view Pose an image as a model

Visual 3D model

Visualisation view

A souvenir Audience representation 2) Spell on the City

Figure 2 Spell on the City [7]

The audience in front of the artwork can tweet their feelings about and hopes for Seoul City, tagging the Twitter account of the artist using their smart devices. The process is that an analysis of audience tweets is extracted by automatic system and categorized under seven emotions: happy, sad, afraid, surprised, disgusted, angry, and neutral. These emotions send amulets to outdoor media boards around Seoul City. Through this system, the audience casts a spell on Seoul automatically and generatively.

Table 3

Analysis of Spell on the City

view

SNS

interaction and media board network

Communication- based interaction

Audience view Tweet their feelings

Sender tweets, operator and emotion

generator

359 359 359 557 557 557 557

(4)

Send the amulet corresponding to the emotion to outdoor media boards around Seoul

No representation on the screen

3) In Between

Figure 3 In Between [7]

This artwork uses a motion-tracking installation to interact with the audience. It senses the position of audience movement as they approach the installation and fictional footsteps of an invisible and virtual character come to the audience.

Table 4 Analysis of In Between

view

Projection, tracking cam, PC, sound

Communication- based interaction

Audience view Audience approaches another audience

Operator making reaction

Fictional footsteps on installation

No

representation on screen

4) Inter-Scenery

Figure 4 Inter-Scenery [7]

This artwork is an interactive art installation based on human representation. As the audience enters the gallery space, a camera captures their silhouettes, which remain imprinted on the wall screen which is set with a background showing a foreign virtual situation.

Table 5

Analysis of Inter-Scenery

Factors First

Variable

Second Variable

Installation view

Screen Control-based interaction Audience view Strike a pose

and pull a string

Operator and 2D model

Audience image in virtual reality screen

Audience representation

4.5. Discussion

In a visualisation view, the audience directly participates in interactive artwork or watches the output of content as an objective observer. These factors were already studied by Fel’s research - contemplation and belonging [4] [5]. However, Fel’s research did not address the concept of the degree to which the contents are changed by audience action; he restricted audience action to visualisation contents. The audience in interactive art performed their role by their actions. In this viewpoint, audience action affects visualisation and visualisation affects audience action. These relationships reflect the categories of interactive art due to expanded audience experience. The new concept of categories should include the process of audience action changing into visualisation in relationship with the audience role subdivided by their action and visualisation.

360 360 360 558 558 558 558

(5)

From the viewpoint of a new concept, the participation of the audience is classified into two categories: ‘to act’ or ‘to be represented’. ‘To act’ means that the audience directly acts on interactive art installations or objective screens, while ‘to be represented’

in visualisation means the self-image of the audience is represented or expressed, not the self-image of the art installation. That is because the main trigger in interactive art is the audience itself. The audience identification is operating an interactive situation and generating contents.

The audience is expressed through other images transformed or mediated or visualised by artwork. ‘To be represented’ is an important issue to make artwork and to determine the visualisation of interactivity.

The following figure is a masterpiece of interactive art, representing the ‘to act’ type or the ‘to be represented’

type.

Figure 5 ĞTo act’ Type, Jeffrey Shaw, Legible City, 1989-1991

Figure 6 ĞTo be represented’ Type, Herman Maat, Paranoid Panopticum, 1999 The following is a four-factor combination of action and representation.

1) Not to act and not to be represented 2) To act and not to be represented 3) Not to act and to be represented 4) To act and to be represented

4.6. Producing a New Method for Categorisation This method classifies interactive art into four categories by audience action and representation. The

action is plotted along the Y axis; the visualisation is plotted along the X axis.

Figure 7 Categorisation by Four Factors 5. Categorisation

5.1. Categorisation of the Role of Audience Participation

According to the category graph, this research proposes the categorisation of the role of audience participation by four factors:

Figure 8 Categorisation of the Role of Audience Participation by Four Factors

1) Object type

The audience that does not act and is not represented becomes an object. The audience as object means it is a part of the installation.

To express visualisation of object type, the audience in this situation is considered an object as background or environment. The audience participates in the artwork, but does not control output, only belonging to a part of the contents.

2) Coordinator type

The audience that acts and is not represented becomes a coordinator. The audience as a coordinator means it is an operator and transistor of the artwork. The audience

361 361 361 559 559 559 559

(6)

aim is to operate the art installation and to develop the contents of the artwork by audience choice.

To express visualisation of the coordinator type, the audience is considered by the performer to produce contents of a visualised environment or output in this situation. The author considers the trigger input by the audience performance.

3) Transformer type

The audience that does not act and is represented becomes a transformer. Transformer means the audience is changed into another being in the artwork image. The audience’s facial image changes or the whole figure alters into other conditions. Defamiliarisation of the image by artwork is the main aesthetic thrust. The audience watches itself through the artwork and experiences the aesthetic developed by the authors.

To express visualisation of the transformer type, the audience is considered by image and defamiliarisation to produce contents. The audience identification is the main element by reflection.

4) Character type

The audience that acts and is represented becomes a character in narrative. This means the audience takes part in the virtual simulation, preparing the setting, virtual role, and mission. Audience as character plays two subtypes in interactive art: avatar or character in narrative. The avatar type means that audiences do not have their own will; they move by means of an automatic system made by the artist and author. The character type means that the audience does enact their own will. As the audience reacts, the narrative in the artwork develops.

To express visualisation of character, the audience is considered by image and action to produce the contents of the environment or installation in this situation.

5.2. Two Realms of Visualisation for Audience Participants

The audience action is the trigger of interactive art, and embodiment of interactive art is audience action applying to visualisation. In the end, audience identification is relevant for these interrelationships. In virtual or augmented reality, a human being is the only real actor in a virtual situation regardless of whether he/she acts as an individual or as a virtual being. This research propose two features of humans from the viewpoint of action and visualisation.

Briefly, the audience is inserted into the situation of the artwork as individuals or as transformations in interactive artworks.

First, in the artwork in which the audience becomes an ‘object’ or ‘coordinator’ in the outcome of art according to their actions, the audience inserts itself.

Audience identification is still maintained in the artwork.

Next, in the artwork in which the audience becomes a

‘transformer’ or ‘character’ in the outcome of art according to their actions, the audience inserts itself as a

transformation. Audience identification becomes another existence in the artwork.

6. Conclusion 6.1. Summary

This research proposes the basis of categories of relationship with action and visualisation of the audience in interactive art. By using these four categories, the participation of the audience in interactive art is classified and a new role of the audience in artwork is found. This research proposes a basic graph illustrating audience action and visualisation.

The audience participates in interactive art according to roles as the object, coordinator, transformer, or character. Based on these four categories, the audience participation varies in specific ways. Audience action in visualisation is the basic element for interactive interface or installation. From this viewpoint, this paper is also relevant for audience participation design given the installation in a virtual or augmented-reality world.

6.2. Limitations and Future Work

The limitation is that the research scope focuses on the visual realm. In interactive art, the audience experience includes properties of multimedia using six senses including the synesthetic sense.

The future work would be to make an interactive art framework applying these categories. Based on this framework, the next phase of research could evaluate the proportions of audience action and visualisation. From a different viewpoint, categories are needed to study audience action and multimedia through six senses.

References

[1] Maria Miranda, Interactive Art: Digital Entities and the Audience Experience, Multimedia, IEEE Volume: 15, Issue: 4, 2008

[2] Dennis Del Favero, Scenario: Co-Evolution, Shared Autonomy and Mixed Reality, IEEE International Symposium on Mixed and Augmented Reality 2010 Arts, Media, & Humanities Proceedings, 2010

[3] Tack Woo, Kwangyun Wohn, Nigel Johnson, CATEGORISATION OF NEW CLASSES OF DIGITAL INTERACTION, Leonardo, Vol. 44, no. 1, pp. 90–91, 2011

[4] Sidney Fels, Intimacy and Embodiment: Implications for Art and Technology, MULTIMEDIA '00 Proceedings of the 2000 ACM workshops on Multimedia, 2000 [5] Brigid Costello, Lizzie Muller, Shigeki Amitani & Ernest

Edmonds, Understanding the Experience of Interactive Art: Iamascope in Beta_space, Proceedings of the Second Australasian Conference on Interactive Entertainment, 2005

[6] Jaewha Kim, Interactive Media Art: A Study on Interactivity, Soongsil University doctoral dissertation, 2008

[7] http://www.mediacityseoul.kr/

362 362 362 560 560 560 560