æ ªåŒäŒç€ŸSOARIGã§ãšã³ãžãã¢ãããŠãããElleryã§ãã ä»åã¯ãAttention Is All You Needã®è«æãèªãã ã®ã§ããŸãšããŠã¿ãŸããã
Attention Is All You Needãšã¯
ãAttention Is All You Needãã¯ã人工ç¥èœãå匷ããäžã§ã¯é¿ããŠã¯éããªããTransformerããšããä»çµã¿ãææ¡ããè«æã§ãã
ãããŸã§ãèªç¶èšèªåŠçåéã§ç¢ºåºããå°äœãç¯ããŠãããååž°æ§é ïŒRNNïŒãç³ã¿èŸŒã¿æ§é ïŒCNNïŒãå®å šã«æé€ããAttentionæ©æ§ã®ã¿ã«åºã¥ããã¢ãŒããã¯ãã£ã§åŸæ¥ã®ã¢ãã«ã®ç²ŸåºŠãå€§å¹ ã«æŽæ°ãã倧ãã泚ç®ãéããŸããã
ãŸããTransformerã¯ãã®äŸ¿å©ãããçŸåšã§ã¯èªç¶èšèªåŠçã®åéãé£ã³åºããæ§ã ãªåéã§æŽ»çšãããã¢ãŒããã¯ãã£ãšãªã£ãŠããŸãã
ä»åã¯ããã®Transformerã®äžæ žéšåã§ãããAttentionæ©æ§ã«ã€ããŠè§£èª¬ããŠãããããšæããŸãã
Attentionãšã¯
Attentionã¯æ¥æ¬èªã ãšãã泚ææ©æ§ããšèš³ãããããã«ããéèŠãªæ å ±ã«æ³šæãåããããããšããã¢ã«ãŽãªãºã ã«ãªããŸãã
äŸãã°ããç§ã¯äººå·¥ç¥èœã§ããããšããæç« ããI am an artificial intelligence.ããšããæç« ã«ç¿»èš³ãããšãã«ããIããšããåèªãæšè«ãããšãã¯ããç§ããšããåèªã«ã泚æããåãããããartificial intelligenceããšããåèªãæšè«ãããšãã¯ã人工ç¥èœããšããåèªã«ã泚æããåãããããšãã£ãå ·åã§ãã
äžèšã¯æšè«æã®ãã¯ãã«ã®éã¿ïŒååèªã®é¢ä¿ïŒãç¢å°ã§è¡šçŸããå³ã«ãªããŸãã
ãIããšããåèªãšãç§ããšããåèªã¯ä»ã®åèªã«æ¯ã¹ãŠããã匷ãçŽã¥ããŠããããartificialããšããåèªãåæ§ã«ã人工ç¥èœããšããåèªãšåŒ·ãçŽã¥ããŠããŸãã
泚æãåãããããšããã®ã¯ããã®ãã¯ãã«ã®éã¿ã倧ãããããšããããšã§ããããã®ãã¯ãã«ã®éã¿ïŒç¢å°ã®å€ªãïŒãåŠç¿ããã®ãAttentionãšããã¢ãŒããã¯ãã£ã«ãªããŸãã
Attentionã®ã¢ãŒããã¯ãã£
Attentionã®ã€ã³ãããã¯ãã¢ãã«ã®é äŒæãšããŠæµããŠããå€ãšEncoderããåºåãããmemoryãšèšãããå€ã§ãããããŠãAttentionã®æçµç®çã¯å ¥ã£ãŠããmemoryã«å¯ŸããŠé©åãªweightãæãç®ããŠãæšè«ãã¹ãå€ïŒinputïŒã«å¯Ÿããã泚æããåããããããšã§ãã
é©åãªweightãã©ã®ããã«æ±ããããšãããšãinputãšmemoryã®å ç©ãèšç®ããããšã§æ±ããŸãããã¯ãã«ã®å ç©ã¯ãïŒã€ã®ãã¯ãã«ãã©ãã ãåãæ¹åãåããŠããããèšç®ãããã®ãªã®ã§ãinputãšmemoryã®å ç©ã®èšç®ã¯ãã®ïŒã€ã®ãã¯ãã«ã®ãé¡äŒŒåºŠããèšç®ããããšã«åœãããŸãããã®ããã«èšç®ããweightã¯å¿ ç¶çã«ãæšè«ãã¹ãå€ãšé¡äŒŒããŠããmemoryã®ãã¯ãã«ã«å¯Ÿãããã®ã倧ãããªãã®ã§ããã®weightãšmemoryãæãç®ããããšã«ãã£ãŠãããæ£ç¢ºãªã泚æããåããããäºãã§ããã®ã§ãã
Self-AttentionïŒèªå·±æ³šææ©æ§ïŒ
æåã«çŽ¹ä»ããAttentionã¯ãmemoryããšãinputããš2ã€ã®å ¥åãå¿ èŠãšããã¢ãŒããã¯ãã£ã«ãªã£ãŠããŸããããSelf-Attentionã¯å€éšã®ããŒã¿ãå¿ èŠãšããã1ã€ã®ã€ã³ãããã®ã¿ã§å®çµããã¢ãŒããã¯ãã£ã«ãªããŸãã
åŸæ¥ã®Attentionã§ã¯ãã©ãã«æ³šæãåããããšããã®ã¯ãå€éšã®ããŒã¿ã«å€§ããäŸåããŠããã®ã«å¯ŸããSelf-Attentionã§ã¯å€éšã®ããŒã¿ã«äŸåããèªåèªèº«ã®ããŒã¿ã®ã¿ã§ã©ãã«æ³šæãåããããšããã®ãã³ã³ãããŒã«ã§ããããã«ãªã£ãŠãããšããã®ã倧ããªå€æŽç¹ã«ãªããŸãã
Source Target-Attention
æåŸã«ãTransformerå ã§äœ¿çšãããŠãããSource Target-Attentionã«ã€ããŠçŽ¹ä»ããŸãã
Source Target-Attentionã¯ããinputããšãmemoryãã®2ã€ã®å ¥åãå¿ èŠãšããã¢ãŒããã¯ãã£ã§ãå ¥åãããããŒã¿ãAffineã¬ã€ã€ã§å€æãè¡ã£ãŠãããéã¿ãèšç®ãããšãããæåã«çŽ¹ä»ããAttentionãæ¡åŒµããæ§é ã«ãªã£ãŠããŸãã
ãŸãšã
ä»åã¯ããAttention Is All You Needããèªãã ã®ã§ããã®äžæ žéšåã§ããAttentionæ©æ§ã«ã€ããŠè§£èª¬ããŠããŸããããããŸã§è§£èª¬ãã以å€ã«ãè«æå ã§ã¯ããã¹ã¯ã®è©±ãã¹ã±ãŒãªã³ã°ã«é¢ããŠã®è©±ãMulti-Head Attentionã«é¢ããŠã®è©±ãªã©ãããã®ã§ãããè€éã«ãªãã®ã§å²æããããŸãããã©ããã粟床ã«é¢ããéèŠãªè©±ãªã®ã§ãæ°ã«ãªãæ¹ã¯è«æã®ã»ãã«ç®ãéããŠããã ããã°ãšæããŸãã