How the Learning Speed and Tendency of Reinforcement Learning Agents Change with Prior Knowledge

㌂

㌂㩚

㰖㔳㠦 㦮䞲 ṫ䢪䞯㔋 㠦㧊㩚䔎㦮 䞯㔋 ㏣☚㢖 ἓ

䟻㎇

⼖

⼖䢪

*ἶ⩺╖䞯ᾦ 䅊䜾䎆䞯ὒ

gameboyjisoo@korea.ac.kr, booksky@korea.ac.kr, harrykim@korea.ac.kr

G

How the Learning Speed and Tendency of Reinforcement

Learning Agents Change with Prior Knowledge

Jisoo Kim*, Eun Hun Lee*, Hyeoncheol Kim* *Dept. of Computer Science and Engineering, Korea University

殚 檃檃

䞯㔋 ㏣☚Ṗ ⓦⰆ ṫ䢪䞯㔋㦚 ⻪㣿㩗㦒⪲ 䢲㣿䞶 㑮 㧞☚⪳ 㡆ῂṖ 䢲⹲䞮Ợ 㧊⬾㠊㰖ἶ 㧞┺. ㌂㩚 㰖㔳㦚 㩲Ὃ䟊㍲ 䞯㔋 ㏣☚⯒ ⏨㧒 㑮 㧞㰖Ⱒ, 㧮ⴑ♲ ㌂㩚 㰖㔳㦚 㩲Ὃ䟞㦚 㥚䠮㧊 㫊㨂䞲┺. ⽎ 㡆ῂ⓪ ⿞䢫㔺䞮Ệ⋮ 㧮ⴑ♲ ㌂㩚 㰖㔳㧊 䞯㔋㠦 㠊⟺ 㡗䟻㦚 ⹎䂮⓪㰖 ㌊䘊⽎┺. OpenAI Gym ⧒㧊ぢ⩂Ⰲ⯒ 㧊㣿䟊㍲ Ⱒ✶ Gamble 䢮ἓ, Cliff 䢮ἓ, ⁎Ⰲἶ Maze 䢮ἓ㠦㍲ 㔺䠮㦚 㰚䟟䟞┺. ⁎ ἆὒ ㌂㩚 㰖㔳㦚 䐋䟊 㠦㧊㩚䔎㦮 䟟☯㠦 ἓ䟻㎇㦚 ⿖㡂䞶 㑮 㧞┺⓪ ộ㦚 䢫㧎䟞┺. ⡦䞲, ἓ⪲ 䌦㌟㠦 㧞㠊㍲ 㧮ⴑ♲ ㌂㩚 㰖㔳㧊 㠒Ⱎ⋮ 䞯㔋㦚 ⹿䟊䞮⓪㰖 㞢㞚⽊㞮┺. 䋺㤢✲: ₆Ἒ䞯㔋, ṫ䢪䞯㔋, ㌂㩚 㰖㔳, Q-learning, ṫ䢪 䞯㔋 㠦㧊㩚䔎 1. 昢嵦 ₆Ἒ䞯㔋㦮 ⿚㟒 㭧 䞮⋮㧎 ṫ䢪䞯㔋㦖 trial-and-error ⹿㔳㦒⪲ 㭒㠊㰚 䢮ἓ(environment)㠦㍲ ⽊㌗㠦 ☚╂䞮⓪ 㩫㺛(policy)㦚 ⺆㤆⓪ 㞢ἶⰂ㯮㧊┺. ṫ䢪䞯 㔋㠦㍲⓪ 䟟☯㦮 㭒㼊Ṗ ♮⓪ 㠦㧊㩚䔎(agent)Ṗ Ṗ⓻ 䞲 䟟☯(action) 㭧 䞲 Ṳ⯒ Ἒ㏣ ㍶䌳䟊㍲ 㾲㫛㩗㦒⪲ ⳿䚲㠦 ☚╂䞮⓪ ộ㦚 ⳿䚲⪲ ㌒⓪┺[1]. ┾Ἒ㩗㦒⪲ ⶎ㩲⯒ 䛎⓪ 㧊 ⹿㔳㦖 䡚㫊䞮⓪ ⹿㔳 㭧 ㌂⧢㦮 ⶎ 㩲䛖㧊 ⹿㔳ὒ Ṗ㧻 㥶㌂䞮┺⓪ 䔏㰫㧊 㧞┺[2]. ⁎⩂⋮ ṫ䢪䞯㔋㦖 ⽋㧷䞲 䢮ἓ㠦㍲ Ⱔ㦖 䞯㔋㔲Ṛ 㧊 䞚㣪䞮┺⓪ ┾㩦㦚 Ṗ㰖ἶ 㧞┺[3]. 㧊⯒ 䟊ἆ䞮⓪ ⹿⻫ 㭧 䞮⋮Ṗ 㠦㧊㩚䔎㠦Ợ ㌂㩚 㰖㔳(prior knowledge)㦚 㞢⩺㭒⓪ ộ㧊┺[4]. Q-learning[5]㦚 㝆⓪ ⪲⽝㠦Ợ ㌂㩚 㰖㔳㦚 㞢⩺㭒㠞㦚 ➢Ṗ ⺇㰖㌗䌲 (tabula rasa)㠦㍲ 䞯㔋䞮⓪ ộ⽊┺ ▪ ザ⯊Ợ 䞯㔋䞮ἶ 㫡㦖 ㎇ὒ⯒ ⋮䌖⌎ ㌂⪖Ṗ 㧞┺[6][7]. ⡦䞲, ㌂⧢㧊 ⽊㡂㭒⓪ ㍶⪖⯒ ⳾⹿䞮㡂 䞯㔋䞮⓪ 㠦㧊㩚䔎☚ 㫡㦖 ἆὒ⯒ ⽊㡂㭖 㩗㧊 㧞┺[8]. ⁎⩂⋮ 㧊⩆ ⹿⻫㧊 ㎇ Ὃ䞶 㑮 㧞㠞▮ 㧊㥶⓪ 䟊╏ ⶎ㩲㠦 ╖䟊 㩫╋㠦 䟊 ╏䞮⓪ 㰖㔳㦚 㩲Ὃ䞶 㑮 㧞㠞₆ ➢ⶎ㧊┺[9]. 㩫╋㧊 ⳛ䢫䞮㰖 㞠㦖 ⶎ㩲㠦 ╖䟊㍲⓪ 㧎Ὃ㰖⓻㠦Ợ 㧮ⴑ♲ 㰖㔳㦚 㩚╂䞶 㑮 㧞㦒Ⳇ, ⁎ộ㧊 㧮ⴑ♲ 䟟☯㦒⪲ 㧊㠊㰞 㑮☚ 㧞┺[10][11]. ⁎⩝₆ ➢ⶎ㠦 㧊⩆ 㧶㨂㩗 㥚䠮㧊 㠊⟺ ἓ㤆㠦 ⹲㌳䞶 㑮 㧞⓪㰖 ㌊䘊⽒ 䞚㣪Ṗ 㧞┺. 㧊㠦 ⽎ 㡆ῂ⓪ Q-learning 㦚 㝆⓪ 3 Ṗ㰖 䢮ἓ㠦㍲ 㧊⩆ ㌂㩚 㰖㔳㦮 㡗䟻⩻㦚 䢫㧎䞲┺. 2 Ṳ㦮 ἓ⪲ 䌦 ㌟ 䢮ἓ㠦㍲⓪ 䌦㌟㦚 ⹿䟊䞮⓪ ㌂㩚 㰖㔳ὒ 㹾㍶㺛 㦚 㰖䟻䞮⓪ ㌂㩚 㰖㔳㧊 㾲㫛 ἓ⪲ ἆ㩫㠦 㠊⟺ 㡗 䟻㦚 ⹎䂮⓪㰖 ⽎┺. ☚⹫ 䢮ἓ㠦㍲⓪ 㞞㩫㎇ 䢏㦖 ☚⹫㎇㦚 㿪ῂ䞮⓪ ㌂㩚 㰖㔳㦮 ṫ☚㠦 ➆⧒㍲ 㠊⠑ Ợ 䟟☯㧊 ⹪≢⓪㰖 䢫㧎䞲┺. ⽎ 㡆ῂ㦮 2 㧻㠦⓪ ὖ⩾ 㡆ῂ㠦 ╖䟊㍲ ㍲㑶䟞┺. 3 㧻㠦⓪ ῂ㿫䞲 䢮ἓ㠦 ╖䞲 ㎎⿖ 㩫⽊⯒ ₆㑶䟞㦒 Ⳇ, 4 㧻㠦⓪ ṗ 䢮ἓ㠦㍲ 㰚䟟䞲 㔺䠮㠦 ╖䟊 ㍺ⳛ䞮 ἶ 㔺䠮 ἆὒ⯒ ⿚㍳䟞┺. ἆ⪶㠦⓪ ⽎ 㡆ῂ⯒ 䐋䟊 㠑㦖 ἆὒ ⹥ 㡆ῂ㦮 䞲Ἒ㠦 ╖䟊㍲ ㍲㑶䟞┺. ڍډ ὖ⩾ٻ㡆ῂ㢖ٻ゚ᾦٻ Q-learning 㠦 ㌂㩚 㰖㔳㦖 㔺䠮 㔲㧧 㩚⿖䎆 㞢ἶ 㧞⓪ 㰖㔳㧊Ⳇ, 㧊⩆ ㌂㩚 㰖㔳㦚 㩗㣿䟊㍲ 䞯㔋 ㏣ ☚㢖 ㎇⓻㦚 ⏨㧎 㡆ῂṖ 㧞┺. ⁎ 㭧 Dixon[6]㦖 ⿖ ⿚㩗㧎 ㌂㩚 㰖㔳㧊 䞯㔋㦚 㠒Ⱎ⋮ ☫⓪㰖 䢫㧎䟞┺. ⼓㦚 ⋒ἶ 㧊☯䞮⓪ ⪲⽝㧊 Ṗ㧻 䦪䞲 12 Ṳ㦮 ㌗䌲 㠦 ╖䞲 㩫╋㦚 㞢Ⳋ 㠒Ⱎ⋮ ▪ ザ⯊Ợ 䞯㔋䞮⓪㰖

512

⽺┺. ⁎ ἆὒ 㧒⿖ ㌗䢿㠦Ⱒ 㩗㣿䞲 ㌂㩚 㰖㔳㧊 䞯 㔋 ㏣☚⯒ 7.5 ⺆₢㰖 ⏨㧒 㑮 㧞┺⓪ ộ㦚 䢫㧎䟞┺. Moreno[7]⓪ ⽋㧷䞲 䢮ἓ㦚 ┾Ἒ⼚⪲ ⋮⑶㍲, 㑶⧮ 㧷₆ 䢮ἓ㠦㍲ 䝢⩞㧊㠊Ṗ 㑶⧮⯒ 䞒䟊㍲ ⳿䚲 㰖㩦 㠦 ☚╂䞮☚⪳ 䞯㔋㔲䆆┺. 㻮㦢㠦⓪ ⳿䚲 㰖㩦Ⱒ 㫊 㨂䞮⓪ 䢮ἓ㠦㍲ 䞯㔋㔲䆆㦒Ⳇ, 䢮ἓ㠦 㧋㑯䟊㰖Ⳋ 㑶⧮㢖 䝢⩞㧊㠊 䞲 ⳛ㦚 㹾⪖╖⪲ 㿪Ṗ䟊㍲ 䞯㔋㔲 䆆┺. ⁎ ἆὒ ㌂㩚 㰖㔳㦚 㝊 䝢⩞㧊㠊㦮 㔏⮶㧊 㟓 2 ⺆ ⏨㡖㦒Ⳇ, 䞯㔋 㔲Ṛ ⡦䞲 㩞⹮ 㧊䞮⪲ 㭚㡖┺. ⚦ 㡆ῂ⯒ 䐋䟊㍲ ㌂㩚 㰖㔳㦒⪲ 㩫╋㦚 㮂㦚 ➢ 䞯㔋 㔲Ṛ㦚 㭚㧊ἶ ㎇⓻㦚 ⏨㧒 㑮 㧞┺⓪ ộ㦚 䢫 㧎䟞┺. Dixon 㦖 ㌂⧢㧊 㩲Ὃ䞮⓪ ㌂㩚 㰖㔳㦮 䣾ὒ ⯒ 䢫㧎䟞㦒Ⳇ, ṯ㦖 ⶎ㩲⯒ ┺⯎ 䢮ἓ㠦㍲ 䛖 ➢☚ ⻪㣿㩗㧎 ㌂㩚 㰖㔳㧊 㝆㧒 㑮 㧞┺⓪ ộ㦚 㞢Ợ ♮ 㠞┺. Moreno ⓪ 㧒⿖ ㌗䢿㠦 ╖䞲 㩫╋㦚 ⺆㤆Ⳋ, ⁎ ộ㦚 ₆⹮㦒⪲ ▪ ⽋㧷䞲 ㌗䢿㦚 ザ⯊Ợ 䞯㔋䞶 㑮 㧞┺⓪ ộ㦚 䢫㧎䟞┺. ṫ䢪䞯㔋 㠦㧊㩚䔎☚ ┾Ἒ⼚ ⪲ ⶎ㩲⯒ ⋮⑶㍲ 䛖 㑮 㧞┺⓪ ộ㦚 Ỗ㯳䟞┺. ⁎⩂⋮ Dixon, Moreno ⳾⚦ ㌂㩚 㰖㔳㦒⪲ 㩫╋㦚 㭒⓪ ἓ㤆㠦 ╖䟊㍲Ⱒ 䢫㧎䟞┺. ➢⪲⓪ ㌂⧢㧊 㞢ἶ 㧞⓪ ㌂㩚 㰖㔳㧊 㧮ⴑ♦㦚 㑮☚ 㧞㦒Ⳇ, 䞯㔋㠦 ☚ 㤖㧊 㞞♮⓪ ⶎ㩲㦮 㧒⿖⯒ Ⲓ㩖 䞯㔋㔲䌂 㑮 㧞┺. ⡦䞲, 㩫╋㧊 㠜⓪ ⶎ㩲㠦 ╖䟊㍲ 䢫㧎䞮㰖 㞠㞮┺. ⁎⩝₆㠦 ⽎ 㡆ῂ⓪ 㧮ⴑ♮Ệ⋮ ἓ䟻㎇㦚 㭒⓪ ㌂㩚 㰖㔳㧊 㠊⠑Ợ 䞯㔋㠦 㡗䟻㦚 㭒⓪㰖 䢫㧎䞲┺. 3. 筞筞凃 暒暒儢

㡺䝞㏢㓺 ⧒㧊ぢ⩂Ⰲ㧎 OpenAI Gym[12]㦮 Kelly Coinflip, Cliff, Frozen Lake 䢮ἓ㦚 㺎ἶ䟊㍲ 㾲㫛㩗㦒 ⪲ Gamble, Cliff, Maze 䢮ἓ㦚 Ⱒ✺㠞┺. ⳾✶ 䢮ἓ㠦 ㍲ 㻮㦢㠦⓪ 䌦䠮㦚 㤆㍶㔲䞶 㑮 㧞☚⪳ ⶊ㧧㥚㎇㦚 㠊ⓦ 㩫☚ ⿖㡂䞮ἶ, 䞯㔋㧊 㰚䟟♮Ⳋ㍲ ⶊ㧧㥚 䟟☯ 㦚 䀾䞶 䢫⮶㦚 㩦㹾 Ṧ㏢㔲䆆┺. ⡦䞲 ⳾✶ 䢮ἓ㠦 ㍲ 䞶㧎Ἒ㑮(discount rate)⓪ 0.97, 䞯㔋⮶(learning rate) 㦖 0.01 ⪲ ἶ㩫䟞┺. ἓ⪲ 䌦㌟ 䢮ἓ㠦㍲⓪ ⳿䚲 㰖㩦₢㰖 Ṗ⓪ 㾲㩗 ἓ ⪲⯒ 㺔㦚 㑮 㧞☚⪳ 㧊☯㠦 ╖䞲 ⿖㩫㩗㧎 ⽊㌗㦚 㩗㣿䟞┺. 䞲 ⻞ 䟟☯㦚 䀾䞶 ➢ ⽊㌗㦚 -1 ⪲ 㰖㩫䞾 㦒⪲ 㠦㧊㩚䔎Ṗ 䣾㥾㩗㦒⪲ 㧊☯䞮☚⪳ 㥶☚䟞┺. 3.1 Gamble 䢮ἓ Gamble 䢮ἓ㠦㍲⓪ ☯㩚 ⛺㰧₆⪲ ☚⹫䞮Ⳋ㍲ 㾲 ╖䞲 ☞㦚 ⳾㞚㟒 䞲┺. ⽊㥶䞲 㧦㌆㦮 㧒㩫 ゚㥾㦚 ⻶䕛䞮ἶ, 㞴Ⳋ㧊 ⋮㡺Ⳋ ⻶䕛 ⁞㞷Ⱒ䋒 㠑ἶ ✍Ⳋ㧊 ⋮㡺Ⳋ ⻶䕛 ⁞㞷Ⱒ䋒 㧙⓪┺. 䞲 䞯㔋 䣢㹾⓪ 㽳 10 ⧒㤊✲ ☯㞞 㰚䟟♮Ⳇ, ⁎ 㩚㠦 ☞㦚 ┺ 㧙Ệ⋮ 㾲╖ ⁞㞷₢㰖 ⻢Ⳋ 䣢㹾Ṗ 㫛⬢♲┺. 㽞₆ 㧦⁞㦖 20 㤦, ☯㩚 㞴Ⳋ 䢫⮶㦖 0.62, 㾲╖ ⁞㞷㦖 150 㤦㦒⪲ 㺛㩫 䟞┺. ⁎ 㧊㥶⓪ 䟊╏ 㽞₆Ṩ㦒⪲ 䟞㦚 ➢, 㠦㧊㩚䔎 㦮 ㎇䟻㧊 㫆⁞㝿 ┺⯊Ợ ⋮㢪㦒Ⳇ, ㌂㩚 㰖㔳㦒⪲ 䟟☯㧊 㠊⠑Ợ ⼖䞮⓪㰖 㞢㞚⽒ 㑮 㧞㠞₆ ➢ⶎ㧊┺. 䡚㨂 ⽊㥶 ⁞㞷㦚 5 ⪲ ⋮⑞ ộ(⋮Ⲏ㰖⓪ ⻚Ⰶ┺)㧊 㠦㧊㩚䔎㦮 ㌗䌲Ṗ ♮ἶ, ⽊㥶 ⁞㞷㠦 ➆⧒㍲ 㽳 30 Ṳ㦮 ㌗䌲Ṗ 㫊㨂䞲┺. Ṗ⓻䞲 䟟☯㦖 ⽊㥶 㧦㌆㦮 10%, 20%, 40%, 60%⯒ ☚⹫䞮⓪ ộ㦒⪲ 㽳 4 Ṗ㰖Ṗ 㧞┺. 㻮㦢㠦 ⶊ㧧㥚 䟟☯㦚 䀾䞶 䢫⮶㦚 100%⪲ ㍺ 㩫䟞㦒Ⳇ, decay rate ⯒ 0.999 ⪲ ㍺㩫䟞┺. 3.2 Cliff 䢮ἓ (⁎Ⱂ 1) Cliff 䢮ἓ ⌃⟶⩂㰖⯒ 䞒䟊㍲ ⳿䚲 㰖㩦㦒⪲ Ṗ⓪ ἓ⪲ 䌦㌟ 䢮ἓ㧊┺(⁎Ⱂ 1). ⳿䚲 㰖㩦㠦 ☚╂䞮Ⳋ +30 ⽊㌗㦚 㠑ἶ, ⌃⟶⩂㰖㠦㍲ ⟾㠊㰖Ⳋ -30 ⽊㌗㦚 㠑㦒Ⳋ㍲ 䞯 㔋 䣢㹾Ṗ 㫛⬢♲┺. 㩲㧦ⰂỎ㦢㦚 ⹿㰖䞮₆ 㥚䟊㍲ ⼓㠦 ⿖➀䧞Ⳋ -10 㦮 ⽊㌗㦚 㠑Ợ 䟞┺. 㠦㧊㩚䔎㦮 㥚䂮Ṗ 䡚㨂㦮 ㌗䌲㧊Ⳇ, 㽳 72 Ṳ㦮 ㌗ 䌲Ṗ 㫊㨂䞲┺. ⁎ 㭧 28 Ṳ⓪ ⼓㧊Ⳇ 㔺㩲⪲ ㌂㣿䞮 ⓪ ㌗䌲⓪ 44 Ṳ┺. Ṗ⓻䞲 䟟☯㦖 ㌗䞮㫢㤆 㧊☯ ⍺ Ṗ㰖┺. 㻮㦢㠦 ⶊ㧧㥚 䟟☯㦚 䀾䞶 䢫⮶㦚 60%⪲ ㍺ 㩫䟞㦒Ⳇ, decay rate ⯒ 0.99 ⪲ ㍺㩫䟞┺. 3.3 Maze 䢮ἓ (⁎Ⱂ 2) Maze 䢮ἓ Cliff 䢮ἓὒ ṯ㧊 ἓ⪲ 䌦㌟ 䢮ἓ㧊┺ (⁎Ⱂ 2). ⳿ 䚲 㰖㩦㠦 ☚╂䞮Ⳋ +30 ⽊㌗㦚 㠑ἶ, 㩲㧦ⰂỎ㦢㦚 ⹿㰖䞮₆ 㥚䟊㍲ ⼓㠦 ⿖➀䧞Ⳋ -20 㦮 ⽊㌗㦚 㠑⓪┺. ἓ⪲ 1 㦖 㾲㩗 ἓ⪲⪲ 22 ⻞㦮 䟟☯ 䤚㠦 ⳿䚲 㰖 㩦㠦 ☚╂䞮Ⳇ, ἓ⪲ 2, 3 㦖 ṗṗ 24 ⻞, 34 ⻞ Ⱒ㠦 ⳿ 䚲 㰖㩦㠦 ☚╂䞲┺. 㠦㧊㩚䔎㦮 㥚䂮Ṗ 䡚㨂㦮 ㌗䌲㧊Ⳇ, 㽳 121 Ṳ㦮 ㌗䌲Ṗ 㫊㨂䞲┺. ⁎ 㭧 70 Ṳ⓪ ⼓㧊Ⳇ 㔺㩲⪲ ㌂㣿 䞮⓪ ㌗䌲⓪ 51 Ṳ┺. Ṗ⓻䞲 䟟☯㦖 ㌗䞮㫢㤆 㧊☯ ⍺ Ṗ㰖┺. 㻮㦢㠦 ⶊ㧧㥚 䟟☯㦚 䀾䞶 䢫⮶㦚 10%⪲ ㍺㩫䟞㦒Ⳇ, decay rate ⯒ 0.99 ⪲ ㍺㩫䟞┺. ڏډ 柪竞 愕 㔺䠮ٻἆὒٻ ㌂㩚 㰖㔳㧊 㠦㧊㩚䔎㠦 ⹎䂮⓪ 㡗䟻㦚 ⽊₆ 㥚䟊 ㍲ 㾲㩗䢪⯒ ☫⓪ ㌂㩚 㰖㔳, 㾲㩗䢪⯒ ⹿䟊䞮⓪ ㌂ 㩚 㰖㔳, 㹾㍶㺛㦒⪲ 㥶☚䞮⓪ ㌂㩚 㰖㔳, ⁎Ⰲἶ ἓ 䟻㎇㦚 ⿖㡂䞮⓪ ㌂㩚 㰖㔳㦚 㩦㹾 㩗㣿䟊⽺┺.

513

<䚲 1> Gamble 䢮ἓ㠦㍲㦮 ṗ ⻶䕛 ゚㥾㠦 ╖䞲 ㌂㩚 㰖㔳 㩗㣿 ἆὒ ㌂㩚 㰖㔳 肚 10% ⻶䕛 㥶☚ 20% ⻶䕛 㥶☚ 40% ⻶䕛 㥶☚ 60% ⻶䕛 㥶☚ ㌂㩚 㰖㔳 䘟‶ 䚲㭖 䘎㹾 䘟‶ 䚲㭖 䘎㹾 䘟‶ 䚲㭖 䘎㹾 䘟‶ 䚲㭖 䘎㹾 䘟‶ 䚲㭖 䘎㹾 0.2 389.7 121.27 327.65 113.35 400.65 149.68 374 129.54 355.3 105.70 0.4 306.75 98.12 355.15 133.68 372.45 152.95 434.75 149.05 0.6 286.55 97.50 342.25 128.98 423.75 141.79 457.9 148.20 0.8 277.45 107.23 351.85 100.52 358.5 123.64 440.25 211.78 1.0 265.95 66.13 320.55 91.16 426.35 139.19 455.75 172.02 㤦⧮ ⳾✶ Ṩ㧊 0 㦒⪲ 㔲㧧䞮⓪ Q-table 㦮 㧒㩫 ῂ Ṛ㠦 㟧㑮Ṩ, 㦢㑮Ṩ㦚 㩗㣿䞮⓪ ⹿㔳㦒⪲ 㔺䠮㠦㍲ ㌂㩚 㰖㔳㦚 ⿖㡂䟞┺. 4.1 Gamble 䢮ἓ 㽳 300 ⻞ 䞯㔋㔲䋾 䤚 ㌳㎇♲ Q-table 㦚 ₆⹮㦒⪲ Ợ㧚㦚 䝢⩞㧊䞮☚⪳ 䟞┺. ⳾✶ 䎢㓺䔎⓪ 䝢⩞㧊 20 䣢⪲ 㰚䟟♦㦒Ⳇ, 䘟‶ὒ 䚲㭖䘎㹾⯒ ゚ᾦ䟞┺. ㌂㩚 㰖㔳㧊 㠜㦚 ➢, Q Ṩ㧊 㟓 0.2~5 ㌂㧊㠦 Ⲏⶒ ⩖┺. ⡦䞲 䘟‶ 389.7 㤦㦚 ⳾㦖 ㌗䌲⪲ Ợ㧚㧊 㫛⬢ ♮㠞㦒Ⳇ, Ợ㧚 Ṛ㦮 䚲㭖䘎㹾⓪ 㟓 121.27 㧊㠞┺. ㌂㩚 㰖㔳㦮 㩫☚㠦 ➆⧒㍲ 㠦㧊㩚䔎㦮 䟟☯㧊 㠊 ⠑Ợ ⹪≢⓪㰖 ㌊䘊⽊㞮┺. ☢㞚ṖⳊ㍲ 䞲 䟟☯㠦 ╖ 䟊 0.2, 0.4, 0.6, 0.8, 1.0 㦮 Q Ṩ㦚 㩗㣿䟞┺. ἓ䟻㎇㦮 ⼖䢪⯒ ⋮䌖⌎ ἆὒ⓪ 䚲 1 ὒ ⁎Ⱂ 3 㠦 ⋮䌖⌞㦒Ⳇ, ㏢㑮㩦 㞚⧮ 2 㧦Ⰲ₢㰖 ⹮㢂Ⱂ䟞┺. (⁎Ⱂ 3) Gamble 䢮ἓ㠦㍲ ㌂㩚 㰖㔳㦒⪲ 䟟☯ ㍶䌳㠦 ⼖䢪⯒ ╖䚲㩗㦒⪲ ⋮䌖⌎ ㌂⪖✺㦮 䧞䔎ⱋ ㌂㩚 㰖㔳㦚 ṫ䞮Ợ ⿖㡂䞮Ⳋ 㠦㧊㩚䔎㦮 䟟☯㠦 ἓ䟻㎇㦚 ⿖㡂䞶 㑮 㧞┺⓪ ộ㧊 䢫㧎♮㠞┺. 䔏䧞 㞞㩚㎇㦚 㿪ῂ䞮⓪ 10% ⻶䕛ὒ ☚⹫㎇㦚 㿪ῂ䞮⓪ 60% ⻶䕛㠦 ╖䞲 ㌂㩚 㰖㔳㦖 䢫㔺䞲 ⼖䢪⯒ ⽊㡂㭖 ┺. ㌂㩚 㰖㔳㦚 ṫ䞮Ợ 㩗㣿䟞㦚 ➢⓪ Ệ㦮 䟃㌗ 䟊 ╏ 䟟☯㦚 ㍶䌳䞮⓪ ộ㧊 䢫㧎♮㠞┺. 㞞㩚㎇㦚 㿪ῂ䞮⓪ ㌂㩚 㰖㔳㧊 ṫ䞮Ⳋ 䘟‶ 㽳㑮 㧋㧊 㟓 30%₢㰖 㭚㰖Ⱒ, 䚲㭖䘎㹾⓪ 㩞⹮₢㰖 Ṧ㏢ 䞲┺. ☚⹫㎇㦚 㿪ῂ䞮⓪ ㌂㩚 㰖㔳㧊 ṫ䞶㑮⪳ 䘟‶ 㽳㑮㧋㧊 㟓 17% ⓮㰖Ⱒ, 䚲㭖䘎㹾☚ 2 ⺆₢㰖 ⓪┺. 4.2 Cliff 䢮ἓ 㻮㦢㦒⪲ 㾲㩗 ἓ⪲⯒ 㺔⓪ 䞯㔋 䣢㹾⯒ 䢫㧎䞮⓪ 㔺䠮㦚 㰚䟟䟞┺. ⳾✶ 䎢㓺䔎⓪ 䝢⩞㧊 20 䣢⪲ 㰚䟟 ♦㦒Ⳇ, 䘟‶ὒ 䚲㭖䘎㹾⯒ ゚ᾦ䟞┺. ㌂㩚 㰖㔳㧊 㠜㦚 ➢, Q Ṩ㧊 㟓 -9 㠦㍲ 20 ㌂㧊㠦 Ⲏⶒ⩖┺. ⳿䚲 㰖㩦㠦 Ṗ₢㤎㑮⪳ 㟧㑮 Ṩ㧊 䅎㦒Ⳇ, ⌃⟶⩂㰖 㰗㩚㠦㍲ ⌃⟶⩂㰖⪲ 㧊☯䞮⓪ 䟟☯㠦 ╖䞲 Q Ṩ㧊 Ṗ㧻 ⌄㞮┺. 䘟‶㩗㦒⪲ 300 ⻞㱎 䣢㹾㠦 䞯 㔋㧊 ♦㦒Ⳇ, 䚲㭖䘎㹾⓪ 㟓 92.53 㧊㠞┺. ⁣㩫㩗, ⿖㩫㩗 ㌂㩚 㰖㔳㦮 㩫☚㠦 ➆⧒㍲ 䞯㔋 ㏣☚Ṗ 㠒Ⱎ⋮ ⹪≢⓪㰖 ㌊䘊⽊㞮┺. ⌃⟶⩂㰖⪲ ⟾ 㠊㰞 㑮 㧞⓪ 8 Ṳ㦮 ㌗䌲㠦 㾲㩗 ἓ⪲⯒ Ṗ⯊䂮⓪ ㌂㩚 㰖㔳(㤆䁷㦒⪲ 㧊☯)ὒ ⌃⟶⩂㰖⪲ 㥶☚䞮⓪ ㌂ 㩚 㰖㔳(㞚⧮⪲ 㧊☯)㦚 㩗㣿䟞┺. ⡦䞲, 㧊 ⚦ Ṗ㰖 㭧 䞲 Ṗ㰖㠦 㟧㑮㦮 Q Ṩ㦚 㩗㣿䞮Ⳋ, ┺⯎ ἓ㤆㠦 ╖䟊㍲ ṯ㦖 Q Ṩ㦚 㦢㑮⪲ 㩗㣿䟞┺. 㡂⩂ 䎢㓺䔎⯒ 䟊⽎ ἆὒ, 0.1~0.6 ㌂㧊㦮 Q Ṩ✺㧊 Ṗ㧻 㥶㦮⹎䞲 ἆὒ⯒ ⋮䌖⌞₆ ➢ⶎ㠦 䟊╏ Ṩ㦒⪲ 㾲㫛 㔺䠮㦚 㰚䟟䟞┺. 㾲╖ 1000 ⻞₢㰖 䞯㔋㦚 䞶 㑮 㧞☚⪳ ㍺㩫䟞㦒Ⳇ, ⁎➢₢㰖 䞯㔋㧊 㞞 ♮Ⳋ 㔺 䕾䟞┺ἶ Ṛ㭒䟞┺. ἆὒ⓪ 䚲 2 㠦 ⋮䌖⌞㦒Ⳇ, ㏢㑮 㩦 㞚⧮ 2 㧦Ⰲ₢㰖 ⹮㢂Ⱂ䟞┺. <䚲 2> 㾲㩗 ἓ⪲㢖 ⌃⟶⩂㰖⪲ ⟾㠊㰖⓪ ἓ⪲㠦 ╖ 䞲 ㌂㩚 㰖㔳㦚 㩗㣿䟞㦚 ➢ 䞯㔋 ἆὒ 㾲㩗 ἓ⪲ ⌃⟶⩂㰖 ㌂㩚 㰖㔳 䘟‶ 䚲㭖 䘎㹾 䘟‶ 䚲㭖 䘎㹾 㔺䕾 䣵㑮 0.1 148 155.18 339.1 92.92 0 0.2 46.25 115.57 339.15 78.83 0 0.3 21.7 54.14 364.2 83.78 0 0.4 9.85 6.51 358.05 77.15 0 0.5 9.8 4.88 406.38 84.06 7 0.6 8.05 4.70 439.44 111.86 11 㾲㩗 ἓ⪲⯒ 㺔⓪◆ ☚㤖㧊 ♮⓪ ㌂㩚 㰖㔳㦮 ἓ㤆, 㩗㦖 㟧㦚 ⿖㡂䞮▪⧒☚ 䞯㔋 ㏣☚⯒ 䋂Ợ ⏨㧒 㑮 㧞㠞┺. ⌃⟶⩂㰖⪲ 㥶☚䞮⓪ ㌂㩚 㰖㔳㦮 ἓ㤆㠦☚ 㩗㦖 㟧㦒⪲☚ 䞯㔋 ㏣☚⯒ 10%~20% ⓼㿲 㑮 㧞㠞 ┺. ⡦䞲, 㧮ⴑ♲ ㌂㩚 㰖㔳㧊 ⍞ⶊ 䄺㰖Ⳋ 㩞⹮ 㧊㌗ 㦮 ἓ㤆㠦 䞯㔋 㧦㼊⯒ 㔺䕾䞲┺. 㧊⯒ 䐋䟊 Ṛ┾䞲 ἓ⪲ 䌦㌟ 䢮ἓ㠦㍲ 㧧㦖 㟧㦮 ㌂㩚 㰖㔳☚ 䋆 㡗䟻 㦚 ⋒䂶 㑮 㧞┺⓪ ộ㦚 䢫㧎䟞┺.

514

4.3 Maze 䢮ἓ 3 Ṳ㦮 ἓ⪲ 㭧 䞲 ἓ⪲⪲ 䞯㔋㦚 㥶☚䞮⓪ ㌂㩚 㰖㔳㦚 ⿖㡂䟞㦚 ➢ 㠊⟺ ἓ⪲⯒ ㍶䌳䞮⓪㰖 䢫㧎䞮 ⓪ 㔺䠮㦚 㰚䟟䟞┺. 䞯㔋ὒ 䎢㓺䔎⯒ ⼧䟟䟞㦒Ⳇ, 㽳 3000 ⻞ 䞯㔋㦚 㔲䋺Ⳋ㍲ 䞲 ἓ⪲Ⱒ Ἒ㏣ ㍶䌳䞮Ợ ♮⓪ 㔲㩦㦚 ₆⪳䟞┺. ἓ⪲⯒ 㺔㦚 ➢ 100 ⻞ 㧊㌗ 䟟☯㦚 䀾䟊☚ ⳿䚲 㰖㩦㦚 㺔㰖 ⴑ䞮Ⳋ, ┺㦢 䞯㔋 䣢㹾⪲ ⍮㠊Ṗ☚⪳ 䟞┺. ㌂㩚 㰖㔳 㠜㧊 5 䣢 䎢㓺䔎䟞㦚 ➢, 㟓 1780 䣢 䞯 㔋㦚 㰚䟟䞮Ⳋ 㾲㩗 ἓ⪲⯒ 䢫㩫㩗㦒⪲ 㺔₆ 㔲㧧䟞 ┺. 㧊➢ Q Ṩ㧊 㟓 -10 㠦㍲ 20 ㌂㧊㠦 Ⲏⶒ⩖┺. 㟧 㑮 Ṩ㦖 ⳿䚲 㰖㩦 ⁒㻮⧒⓪ ộ㦚 ἶ⩺䞮Ⳋ, 㭧Ṛ ἓ ⪲㦮 Q Ṩ㦖 -10 ₢㰖 Ṛ┺ἶ 䕦┾䟞┺. Q-learning 䔏㎇㌗ 䞲 ⹿䟻㦒⪲ 㧊☯䞮Ợ 䞮₆ 㥚䟊 ㍲ ⋮Ⲏ㰖 ㎎ ⹿䟻㠦 㦢㑮㧎 Q Ṩ㦚 㩗㣿䟞┺. ⁎⩝ 㰖 㞠㦒Ⳋ, ㌂㩚 㰖㔳㦒⪲ ⿖㡂䞲 㟧㑮㦮 Q Ṩ㧊 ⋮ Ⲏ㰖 3 ⹿䟻ὒ ṯ㞚㰞 ➢₢㰖 ⹮⽋㩗㦒⪲ ㌂㩚 㰖㔳 㧊 ⿖㡂♲ ㌗䌲 ⌊㠦㍲Ⱒ 㧊☯䟞₆ ➢ⶎ㧊┺. ἓ⪲ 1, 2, 3 㦒⪲ 㥶☚䞮⓪ ㌂㩚 㰖㔳㦚 㩗㣿䞮ἶ ⁎ ἆὒ⯒ ㌊䘊⽊㞮┺. 㡂⩂ 䎢㓺䔎⯒ 䟊⽎ ἆὒ, 1~30 ㌂㧊㦮 Q Ṩ✺㧊 Ṗ㧻 㥶㦮⹎䞲 ἆὒ⯒ ⋮䌖⌞ ₆ ➢ⶎ㠦 䟊╏ Ṩ㦒⪲ 㾲㫛 㔺䠮㦚 㰚䟟䟞┺. 㧊➢, 䢮ἓ㦮 ⽋㧷☚⯒ ⹮㡗䞮㡂 ⿖⿚㩗㦒⪲Ⱒ ㌂㩚 㰖㔳㦚 㩗㣿䟞┺. ἓ⪲ 㩚㼊㠦 ╖䟊 ㌂㩚 㰖㔳㦚 㩗㣿䞮㰖 㞠ἶ 㔲㧧 㰖㩦⿖䎆 㩚㼊 ἓ⪲㦮 㟓 3 ⿚㦮 1 㠦 ╖䟊 ㌂㩚 㰖㔳㦚 ⿖㡂䟞┺. ἆὒ⓪ 䚲 3 㠦 ⋮䌖⌞┺. <䚲 3> ἓ⪲⼚⪲ ㌂㩚 㰖㔳㦚 㩗㣿䟞㦚 ➢ 䞯㔋㦚 Ⱎ 䂮⓪ 䞯㔋 䣢㹾 ㌂㩚 㰖㔳 肚 ㌂㩚 㰖㔳 蛞 ἓ⪲ 1 ἓ⪲ 2 ἓ⪲ 3 1783 1 1987 1818* 1846* 1791 5 2181 1605* 1733* 1777 10 1973 1985* 1532* 1780 20 1405 1593 2387 1799 30 1414 1570 2401 * 䟊╏ ἓ㤆㠦⓪ ἓ⪲ 1 ⪲ 䞯㔋㧊 ⰞⶊⰂ♮㠞┺. ⁏┾㩗㧊㰖 㞠㦖 ㌂㩚 㰖㔳☚ 䞯㔋㠦 ☚㤖㦚 㭖┺ ⓪ ⽊㧻㧊 㠜㠞┺. 㾲㩗 ἓ⪲㧎 ἓ⪲ 1 㠦 ╖䟊㍲ -10 㧊㌗㦮 ㌂㩚 㰖㔳㦚 ⿖㡂䟞㦚 ➢Ⱒ 䞯㔋 ㏣☚Ṗ ゾ⧒ 㰖⓪ ộ㧊 䢫㧎♮㠞㦒Ⳇ, ⁎⽊┺ 㩗㦖 ㌂㩚 㰖㔳㦖 䞯㔋 ㏣☚⯒ ⏨㧊㰖 ⴑ䟞┺. ⋮Ⲏ㰖 ἓ⪲㦮 ἓ㤆㠦☚ Q-table 㠦 㧒⹮㩗㦒⪲ ⋮䌖⋮⓪ Ṩ 㧊㌗㦒⪲ ⿖㡂䟞 㦚 ➢Ⱒ 㠦㧊㩚䔎Ṗ ㍶䌳䞮⓪ 㾲㫛 ἓ⪲⯒ ⹪∖ 㑮 㧞㠞┺. ⡦䞲, ἓ⪲ 3 㦮 ἓ㤆㠦⓪ ⁏┾㩗㧎 ㌂㩚 㰖 㔳㦚 㩗㣿䞮▪⧒☚ ▪ Ⱔ㦖 䞯㔋 䣢㹾Ṗ 䞚㣪䟞┺. 㧊⩆ ἆὒ⯒ 䐋䟊㍲ Q-learning 㠦㍲ ㌂㩚 㰖㔳㦮 䞲Ἒ⯒ ㌊䘊⽒ 㑮 㧞㠞┺. ἓ⪲Ṗ ₎㠊㰞㑮⪳ ⳿䚲 㰖㩦㦮 ⽊㌗㧊 ἓ⪲ 㩚㼊㠦 㩗㣿♶ ➢₢㰖 Ⱔ㦖 䞯㔋 䣢㹾Ṗ 䞚㣪䞮┺. ⡦䞲, ㌂㩚 㰖㔳㦚 ⿖㡂䞮⓪ ⹿㔳 ➢ⶎ㠦 㩗㦖 ㌂㩚 㰖㔳㧊 㡺䧞⩺ 䞯㔋㦚 ⹿䟊䞮⓪ 䣾 ὒ⯒ Ṗ㪎㢪┺. ڐ ڐډ ἆ⪶ٻ ⽎ 㡆ῂ⓪ 3 Ṳ㦮 䢮ἓ㠦㍲ 㰚䟟䞲 Q-learning 㔺䠮 㦚 䐋䟊㍲ 㾲㩗䢪♮㰖 㞠㦖 ㌂㩚 㰖㔳㦮 㡗䟻㠦 ╖䟊 ㌊䘊⽊㞮┺. ⁎ ἆὒ 㩫╋㧊 㠜⓪ 䢮ἓ㠦㍲ 㠦㧊㩚䔎 㦮 䟟☯ ἓ䟻㦚 ㎇Ὃ㩗㦒⪲ ⹪∾┺. ⡦䞲, ⿖㡂䞮⓪ ㌂㩚 㰖㔳㦮 㩫☚㠦 ➆⧒㍲ 䟟☯ ἓ䟻㦚 ▪ Ⱔ㧊 ⹪ ∖ 㑮 㧞┺⓪ ộ☚ 䢫㧎䟞┺. ἓ⪲ 䌦㌟ 䢮ἓ㦮 ἓ㤆, ㌂㩚 㰖㔳㧊 㾲㩗 ἓ⪲ 䌦㌟㦚 ⹿䟊䞮Ⳋ, 䞯㔋㧊 ⓦ ⩺㰖Ệ⋮ 㔺䕾䞶 㑮 㧞┺⓪ ộ㦚 䢫㧎䟞┺. ⁎Ⰲἶ 䢮ἓ㧊 ⽋㧷䟊㰖Ⳋ ⁏┾㩗㧎 ㌂㩚 㰖㔳㦚 ⿖㡂䟊㟒㰖 Ⱒ 䞯㔋㠦 㡗䟻㦚 ⋒䂶 㑮 㧞┺⓪ ộ㦚 䢫㧎䟞┺. 䟻䤚㠦⓪ Q-learning 㣎 ┺⯎ ṫ䢪䞯㔋 ₆⻫㠦㍲ 㩫 ╋㧊 㞚┢ 㡂⩂ 㫛⮮㦮 ㌂㩚 㰖㔳㧊 㠊⟺ 㡗䟻㦚 ⋒ 䂮⓪㰖 ㌊䘊⽒ 䞚㣪Ṗ 㧞┺. ⡦䞲, ₆㫊 㡆ῂ㠦 㝆㡖 ▮ 㭒⼖ ㌂ⶒ㦚 㧎㔳䞮⓪ ὖἚ㩗 ㌗䌲 ὋṚ㠦㍲☚ ┺ ⯎ 㫛⮮㦮 ㌂㩚 㰖㔳㦮 㡗䟻㦚 䢫㧎䞶 䞚㣪䞮┺. 焾処怾竒

[1] Kaelbling, L. P., Littman, M. L., & Moore, A. W. Reinforcement learning: A survey. Journal of artificial intelligence research, 4, 237-285. 1996.

[2] Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT Press. 1998.

[3] Driessens, K., & Džeroski, S. Integrating guidance into relational reinforcement learning. Machine Learning, 57(3), 271-304. 2004.

[4] Smart, W. D., & Kaelbling, L. P. Practical reinforcement learning in continuous spaces. ICML. 2000. 903-910. [5] Watkins, C. J., & Dayan, P. Q-learning. Machine learning,

8(3-4), 279-292. 1992.

[6] Dixon, K., Malak, R. J., & Khosla, P. K. Incorporating prior knowledge and previously learned information into reinforcement learning agents. Carnegie Mellon University, Institute for Complex Engineered Systems. 2000.

[7] Moreno, D. L., Regueiro, C. V., Iglesias, R., & Barro, S. Using prior knowledge to improve reinforcement learning in mobile robotics. Proc. Towards Autonomous Robotics Systems. Univ. of Essex, UK. 2004.

[8] Abbeel, Pieter, and Andrew Y. Ng. Exploration and apprenticeship learning in reinforcement learning. Proceedings of the 22nd international conference on Machine learning. 2005.

[9] Argall, B. D., Chernova, S., Veloso, M., & Browning, B. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5), 469-483. 2009.

[10] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. 2016.

[11] Everitt, T., Krakovna, V., Orseau, L., Hutter, M., & Legg, S. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417. 2017.

[12] https://gym.openai.com/

515