⳾⹪㧒䢮䢮ἓ㠦㍲ ❻⩂┳㦚䢲䢲㣿䞲㦮⹎₆⹮ 㧊㧊⹎㰖㠊⏎䎢㧊㎮㦚㦚㥚䞲㧊㧊⹎㰖䌲⁎ ㍺㍺Ἒ ⹥ ῂ ῂ䡚

(1)

⳾⹪㧒䢮䢮ἓ㠦㍲ ❻⩂┳㦚䢲䢲㣿䞲㦮⹎₆⹮ 㧊㧊⹎㰖㠊⏎䎢㧊㎮㦚㦚㥚䞲㧊㧊⹎㰖䌲⁎ ㍺㍺Ἒ ⹥ ῂ ῂ䡚

㔶㥺⹎*, 㞞㰚䡚**, 㧚☯䡗***

*䢎㍲╖䞯ᾦ 䅊䜾䎆Ὃ䞯ὒ

**㩲㭒╖䞯ᾦ ἓ㡗㩫⽊䞯ὒ

***䢎㍲╖䞯ᾦ 䅊䜾䎆㩫⽊Ὃ䞯⿖

e-mail : dhim@ hoseo.edu

Design and Implementation of Deep-Learning-Based Image Tag for Semantic Image Annotation in Mobile Environment

YoonMi Shin*, Jinhyun Ahn**, Dong-Hyuk Im***

*Department of Computer Engineering, Hoseo University

**Department of Management Information Systems, Jeju National University

***Division of Computer and Information Engineering, Hoseo University

殚檃檃

⳾⹪㧒㦮 ₆㑶 ⹲㩚ὒ ㏢㎲⹎❪㠊㌂㣿㦮㯳Ṗ⪲ 㑮㠜㧊 Ⱔ㦖 Ⲗ䕆⹎❪㠊䆮䎦䁶✺㧊㌳㎇♮ἶ 㧞┺. 㧊⩂䞲 Ⱔ㦖㟧㦮䆮䎦䁶㭧㠦㍲㌂㣿㧦Ṗ 㤦䞮⓪ 㧊⹎㰖⯒ 䣾㥾㩗㦒⪲ 㺔₆ 㥚䟊㦮⹎ ₆⹮

㧊⹎㰖 Ỗ㌟㦚㧊㣿䞲┺. 㧊 Ỗ㌟ ₆⻫㦖㧊⹎㰖㠦㦮⹎ 㧞⓪ 㩫⽊✺㦚㧊㣿䞮㡂㌂㣿㧦Ṗ 㺔ἶ 㧦䞮⓪ 㧊⹎㰖⯒ 㩫䢫䞮Ợ 㺔㦚㑮㧞┺. ⽎㡆ῂ㠦㍲⓪ ⳾⹪㧒䢮ἓ㠦㍲㧊⹎㰖Ṗ Ṗ㰞㑮㧞⓪ 㦮

⹎㩗㩫⽊⯒ 㠊⏎䎢㧊㎮䞮ἶ 㧊㢖 ▪⿞㠊 ⳾⹪㧒㠦㧞⓪ 㧊⹎㰖㠦䛣㎇䞲㠊⏎䎢㧊㎮㦚㥚䟊 ❻⩂

┳ ₆㑶㦚㧊㣿䞮㡂 ┺㟧䞲䌲⁎✺㦚㧦☯ ㌳㎇䞮☚⪳ ῂ䡚䞮㡖┺. 㧊⩝Ợ ㌳㎇♲ 㠊⏎䎢㧊㎮㩫⽊

✺㦖㦮⹎㩗 ₆⹮ 䌲⁎⯒ 䐋䟊 RDF 䔎Ⰲ䝢⪲ 䢫㧻♲┺. SPARQL 㰞㦮㠊⯒ 㧊㣿䞮㡂㦮⹎ ₆⹮ 㧊⹎

㰖 Ỗ㌟㦚䞶㑮㧞┺.

1. 昢嵦

⳾⹪㧒㔲㧻㦮 ⹲╂ὒ ┺㟧䞲㏢㎲ ⹎❪㠊㦮䢲㣿㯳Ṗ⪲ 㧎䞮㡂 ╖⨟㦮㧊⹎㰖䆮䎦䁶Ṗ 㯳Ṗ䞮ἶ 㧞

┺. ➆⧒㍲㧊⩂䞲 Ⱔ㦖㟧㦮㧊⹎㰖⯒ 㩖㧻䞮ἶ ὖ Ⰲ䞮⓪ ộ㧊㭧㣪䞮┺[1,2]. 㧊⯒ 㥚䟊 Ⱔ㦖㟧㦮㧊⹎

㰖䆮䎢䁶㏣㠦㍲䣾㥾㩗㦒⪲ Ỗ㌟䞶㑮㧞⓪ 㧊⹎

㰖㠊⏎䎢㧊㎮ ₆⻫㧊㩲㞞♮㠞┺[3,4]. 㧊⹎㰖㠊⏎䎢㧊㎮ ₆⻫㦖㧊⹎㰖Ṗ Ṗ㰖ἶ 㧞⓪ 㦮⹎ 㩫⽊✺㦚㧊

⹎㰖㢖䞾℮ 㩖㧻䞮㡂㧊⹎㰖⯒ ┺㟧䞮Ợ 䚲䡚䞮⓪

₆⻫㧊┺. 㧊 ₆⻫㦚䐋䟊 Ⱔ㦖㟧㦮㧊⹎㰖 ◆㧊䎆

㏣㠦㍲㌂㣿㧦Ṗ 㤦䞮⓪ 㧊⹎㰖⯒ 㩫䢫䧞㺔㦚㑮㧞

┺.

㧊㩚㡆ῂ㠦㍲⓪ ⳾⹪㧒䢮ἓ㠦㍲㠑㦖㠊⏎䎢㧊㎮

㩫⽊⯒ 㧊⹎㰖㢖䞾℮ 㩖㧻䞲┺[5]. ⳾⹪㧒䢮ἓ㠦㍲

㧊㣿䞶㔲㧊⹎㰖㠦 ╖䞲㌗䢿㩫⽊㢖㌂㣿㧦Ṗ 㧛⩻

䞲䌲⁎ 㩫⽊✺㦚㧊㣿䞮㡂㠊⏎䎢㧊㎮㦚䞲┺. ⳾⹪

㧒䢮ἓ㠦㍲㧦☯㦒⪲ 㠑㠊㰚㌗䢿㩫⽊⓪ 㡾䏾⪲㰖㠎㠊㧎 RDF 䔎Ⰲ䝢㦮 ⁎⧮䝚䎢㧊䎆⪲ ⼖䢮䞮㡂㠊

⏎䎢㧊㎮䞲┺. ㌂㣿㧦Ṗ 㰗㩧㧛⩻䞲䌲⁎✺㦖 DBPedia⯒ 㧊㣿䞮㡂 RDF 䔎Ⰲ䝢⪲ ⳾◎Ⱇ 䞲┺.

⽎㡆ῂ㠦㍲⓪ ⳾⹪㧒 ₆₆㠦㍲㦮㠊⏎䎢㧊㎮ ⹥ Ỗ㌟㔲㓺䎲㍺Ἒ[5]⯒ ₆⹮㦒⪲ RNN(Recurrent

Neural Network)[6]㦮䡫䌲㭧 One to Many ⹿㔳㦮 ❻

⩂┳ ⳾◎㦚㧊㣿䞲┺. 㧊 ⹿㔳㦖㧊⹎㰖䃷㎮㌳㎇

㠦㍲ Ⱔ㧊㌂㣿♮⓪ ⳾◎⪲㍲䞮⋮㦮㧛⩻Ṩ㦒⪲ 㡂

⩂ Ṳ㦮㿲⩻Ṩ㦚㠑㦚㑮㧞┺. 㧊 ⳾◎㦚㿪Ṗ䞮㡂㧛⩻ 㧊⹎㰖㠦 ὖ⩾♲ 䌲⁎✺㦚㧦☯㦒⪲ ㌳㎇♮⓪ 䡫䌲⪲ 䢫㧻䞮㡖┺.

2. 洢橎穢柢枪癢

⽎㡆ῂ㠦㍲⓪ ⳾⹪㧒䢮ἓ, 㧊⹎㰖㠊⏎䎢㧊㎮ ₆ 㑶㠦 CNN(Convolutional Neural Network)ὒ Multimodal RNN(Multimodal Recurrent Neural Network)㦚 ἆ䞿䞲

⳾◎[7]㦚㿪Ṗ⪲ ㌂㣿䞲┺.

(⁎Ⱂ 1)㧊⹎㰖 ὖ⩾ 䌲⁎ 㧦☯ ㌳㎇ ὒ㩫 2019년 추계학술발표대회 논문집 제26권 제2호 (2019. 11)

- 895 -

(2)

(⁎Ⱂ 1)㦖㧛⩻ 㧊⹎㰖㢖 ὖ⩾♲ 㧦☯ 䌲⁎ ㌳㎇ ὒ 㩫㦚⽊㡂㭖┺. Ⲓ㩖㧊⹎㰖Ṗ 㧛⩻Ṩ㦒⪲ ✺㠊ṖỢ

♮Ⳋ CNN 㦚㧊㣿䞮㡂㧊⹎㰖㦮䔏㰫㦚㿪㿲䞮Ợ ♲

┺. 㿪㿲♲ 䔏㰫ⱋ㦖 1 㹾㤦㦮䡫䌲⪲ ⼖䡫♮⓪◆ 㧊䡫䌲Ṗ ₆㫊 CNN ⳾◎㦮㢚㩚㡆ἆ Ἒ䂋㧊┺. 㢚㩚㡆ἆ Ἒ䂋㦖 RNN 㦮䧞✶ ⩞㧊㠊㦮㽞₆Ṩ㦒⪲ ✺㠊 ṖỢ ♲┺. RNN 㦮㼁⻞㱎㎖㦮㧛⩻Ṩ㦖 ⶎ㧦䏶䋆㦒

⪲ <s>Ṗ ✺㠊ṖỢ ♲┺. 㧊⯒ 㔲㧧㦒⪲ 㧊⹎㰖㢖 ὖ

⩾♲ 䌲⁎✺㦚㡞䁷䞮Ợ ♲┺. <e>⧒⓪ ⶎ㧦䏶䋆㧊 ⋮ 㡺Ợ ♮Ⳋ 㡞䁷㦖㫛⬢♲┺. 㧊⩝Ợ 㠑㠊㰚䌲⁎✺ὒ

⳾⹪㧒䢮ἓ㠦㍲㔲Ṛ, 㧻㏢䌲⁎✺₢㰖䐋䞿♮㠊㧊

⹎㰖䌲⁎ 㩫⽊✺㦚 Ṭ⓪┺.

㥚 ὒ㩫㦚㿪Ṗ䞲㩚㼊㔲㓺䎲 ῂ㫆⓪ (⁎Ⱂ 2)㠦㍲

⽊㡂㭖┺.

(⁎Ⱂ 2)㩚㼊㔲㓺䎲 ῂ㫆

㌂㣿㧦Ṗ ⳾⹪㧒䢮ἓ㠦㍲㤦䞮⓪ 㧊⹎㰖⯒ 㧛⩻ 䞲

┺. 㧛⩻ 㧊⹎㰖㠦 ╖䟊 Tag Process ὒ㩫㦚䐋䟊㔲Ṛ, 㥚䂮㩫⽊㢖 ❻⩂┳ ⳾◎㦚䐋䟊䌲⁎ 㩫⽊✺㦚㧦☯

㦒⪲ ㌳㎇♲┺. 䌲⁎ ◆㧊䎆 SPO ⼖䢮 ⳾✞㠦㍲㌳㎇

♲ 䌲⁎ 㩫⽊✺㦚㧊㣿䞮㡂㠊⏎䎢㧊㎮ RDF 䔎Ⰲ䝢䡫䌲⪲ 䢫㧻䞲┺. 㠊⏎䎢㧊㎮䞲 ◆㧊䎆⓪ MySQL ὒ Jena TDB 㠦㩖㧻♲┺. SPARQL 㰞㦮㔲㩖㧻 ♲ 㧊⹎

㰖 ID ⯒ 㺔₆ 㥚䟊 RDF 䔎Ⰲ䝢㦚㌂㣿䞮㡂 ⍺㧚✲

⁎⧮䝚䡫䌲⪲ 㧎◇㕇䞮㡂 Jena TDB 㠦㩖㧻䞲┺.

3. 柢枪癢割笊

(⁎Ⱂ 3)㌂㰚㾂㡗 (⁎Ⱂ 4)㧊⹎㰖䌲⁎ Ⰲ㓺䔎 (⁎Ⱂ 3)㦖 ⳾⹪㧒 ₆₆⪲ ㌂㰚㹣₆ ⡦⓪ ㌂㰚 ❇⪳

㦚䞮⓪ 䢪Ⳋ㧊┺. 㧊 ὒ㩫㦚䐋䟊㌂㰚㦚㧛⩻䞮Ợ

♮Ⳋ (⁎Ⱂ 4)㻮⩒ ❻⩂┳ ὒ㩫㦚䐋䟊㌂㰚㠦 ╖䟊

㧦☯㦒⪲ ὖ⩾♲ 䌲䋂㩫⽊ Ⰲ㓺䔎⯒ ⽊㡂㭖┺. 㿪Ṗ

⪲ ㌂㣿㧦Ṗ 㑮☯㦒⪲ 㧊⹎㰖㠦 ╖䞲䌲⁎⯒ 㧛⩻䞶㑮☚ 㧞┺.

4. 冶嵦

⳾⹪㧒䢮ἓ㦮 ⹲㩚㦒⪲ 㧊⹎㰖䆮䎦䁶Ṗ 㯳Ṗ䞮㡂㦮⹎㩗㧎㧊⹎㰖 Ỗ㌟㧊㭧㣪䟊㪢┺. ⽎㡆ῂ㠦㍲⓪

⳾⹪㧒䢮ἓ㠦㍲ ❻⩂┳ ⳾◎ 㭧 CNN ὒ Multimodal RNN 㦚㧊㣿䞮㡂㧊⹎㰖㠦 ╖䞲䌲⁎ 㩫⽊✺㦚㧦☯

㦒⪲ ㌳㎇䞲┺. ➆⧒㍲㌂㣿㧦Ṗ 㰗㩧㧊⹎㰖㠦䌲⁎

⯒ 㧛⩻䞮⓪ ⻞Ệ⪲㤊 ┾㩦㦚⽊㢚䞲┺. ⡦䞲㧦☯㦒

⪲ ㌳㎇ ♲ 䌲⁎⯒ 㧊㣿䞮㡂䛣⿖䞲㠊⏎䎢㧊㎮㦚 ῂ

㎇䞲┺.

䟻䤚 ὒ㩲⪲⓪ DBPedia 㦮㑶㠊㩫⽊⯒ ㌂㣿䞮㰖㞠ἶ 䌲⁎ Ṛ㠦㑶㠊㩫⽊⯒ 㡆ἆ 䞶㑮㧞☚⪳ ❻⩂

┳ ⳾◎㦚㧊㣿䞮㡂㧊⹎㰖㠦 ╖䞲㦮⹎㩗 ₆⹮ 䌲⁎

⯒ 㧦☯㦒⪲ 䞶㑮㧞☚⪳ 䞶 Ἒ䣣㧊┺.

Acknowledgement

㧊 ⏒ⶎ㦖 2017 ⎚☚ 㩫⿖(⹎⧮㺓㫆ὒ䞯⿖)㦮㨂㤦㦒⪲ 䞲ῃ㡆ῂ㨂┾㦮㰖㤦㦚 ⹱㞚㑮䟟♲ 㡆ῂ (No.NRF-2017R1C1B1003600)㧊Ⳇ, 2018 ⎚☚ 㩫⿖(ᾦ㥷

⿖)㦮㨂㤦㦒⪲ 䞲ῃ㡆ῂ㨂┾㦮㰖㤦㦚 ⹱㞚㑮䟟♲

₆㽞㡆ῂ㌂㠛㧚(No. NRF-2018R1D1A1B07048380).

⡦䞲, ⽎㡆ῂ⓪ ὒ䞯₆㑶㩫⽊䐋㔶⿖ ⹥ 㩫⽊䐋㔶₆ 㑶㰚䦻㎒䎆㦮 ╖䞯 ICT 㡆ῂ㎒䎆㰖㤦㌂㠛㦮㡆ῂἆὒ

⪲ 㑮䟟♮㠞㦢 (IITP-2019-2018-0-01417).

焾処怾竒

[1]⏎㔏⹒, and 䢿㧎㭖. "Ⲗ䕆⹎❪㠊 Ỗ㌟㔲㓺䎲㦮

㍺Ἒ ⹥ ῂ䡚." 㩫⽊ὒ䞯䣢⏒ⶎ㰖: ◆㧊䌖⻶㧊㓺 30.5 (2003): 494-506.

[2]㧊㡺㭖, et al. "㏢㎲゛◆㧊䎆⯒ 㧊㣿䞲㡗䢪䦻䟟㣪㧎⿚㍳." 䞲ῃ䆮䎦䁶䞯䣢⏒ⶎ㰖 14.10 (2014): 527-538.

[3]Im, D. H., Park, G. D.: Linked tag: image annotation using semantic relationships between image tags. Multimed Tools Appl. April 2015, vol. 74, Issue 7, pp2273-2287 (2015) [4]Im, Dong-Hyuk, and Geun-Duk Park. "STAG:

semantic image annotation using relationships between tags." 2013 International Conference on Information Science and Applications (ICISA). IEEE, 2013.

[5]⏎䡚▫, ㍲ὧ㤦, and 㧚☯䡗. "⳾⹪㧒䢮ἓ㠦㍲

㦮⹎ ₆⹮ 㧊⹎㰖㠊⏎䎢㧊㎮ ⹥ Ỗ㌟." Ⲗ䕆⹎❪

㠊䞯䣢⏒ⶎ㰖 19.8 (2016): 1498-1504.

[6]ڨۄۆۊۇۊۑڇٻ گۊۈ܀ݫ ڇٻ ۀۏٻ ڼۇډٻ ٽڭۀھېۍۍۀۉۏٻ ۉۀېۍڼۇٻ network based language model." Eleventh annual conference of the international speech communication association. 2010.

[7]Karpathy, Andrej, and Li Fei-Fei. "Deep visual-semantic alignments for generating 2019년 추계학술발표대회 논문집 제26권 제2호 (2019. 11)

- 896 -

(3)

image descriptions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

2019년 추계학술발표대회 논문집 제26권 제2호 (2019. 11)

- 897 -

⳾⹪㧒 䢮 䢮ἓ㠦㍲ ❻⩂┳㦚 䢲 䢲㣿䞲 㦮⹎₆⹮ 㧊 㧊⹎㰖 㠊⏎䎢㧊㎮㦚 㦚 㥚䞲 㧊 㧊⹎㰖 䌲⁎ ㍺ ㍺Ἒ ⹥ ῂ ῂ䡚