• English
    • suomi
  • English 
    • English
    • suomi
  • Login
View Item 
  •   Home
  • Artikkelit
  • Luonnontieteiden ja metsätieteiden tiedekunta
  • View Item
  •   Home
  • Artikkelit
  • Luonnontieteiden ja metsätieteiden tiedekunta
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

End-to-End Listening Agent for Audiovisual Emotional and Naturalistic Interactions

Thumbnail
View/Open
Article (834.4Kb)
Self archived version
published version
Date
2018
Author
El Haddad, Kevin
Rizk, Yara
Heron, Louise
Hajj, Nadine
Zhao, Yong
Kim, Jaebok
Ngo Trong Trung
Lee, Minha
Doumit, Marwan
Lin, Payton
Kim, Yelin
Cakmak, Huseyin
Unique identifier
10.7559/citarj.v10i2.424
Metadata
Show full item record
More information
Research Database SoleCris

Self-archived article

Citation
El Haddad, Kevin. Rizk, Yara. Heron, Louise. Hajj, Nadine. Zhao, Yong. Kim, Jaebok. Ngo Trong Trung. Lee, Minha. Doumit, Marwan. Lin, Payton. Kim, Yelin. Cakmak, Huseyin. (2018). End-to-End Listening Agent for Audiovisual Emotional and Naturalistic Interactions.  Journal of science and technology of the arts, 10 (2) , 49-61. 10.7559/citarj.v10i2.424.
Licensed under
All rights reserved
Abstract

In this work, we established the foundations of a framework with the goal to build an end-to-end naturalistic expressive listening agent. The project was split into modules for recognition of the user’s paralinguistic and nonverbal expressions, prediction of the agent’s reactions, synthesis of the agent’s expressions and data recordings of nonverbal conversation expressions. First, a multimodal multitask deep learning-based emotion classification system was built along with a rule-based visual expression detection system. Then several sequence prediction systems for nonverbal expressions were implemented and compared. Also, an audiovisual concatenation-based synthesis system was implemented. Finally, a naturalistic, dyadic emotional conversation database was collected. We report here the work made for each of these modules and our planned future improvements.

Subjects
listening agent   smile   laughter   head movement   eyebrow movement   speech emotion recognition   nonverbal expression detection   sequence-to-sequence prediction systems   multimodal synthesis   nonverbal expression synthesis   emotion database   dyadic conversa   
URI
https://erepo.uef.fi/handle/123456789/7321
Link to the original item
http://dx.doi.org/10.7559/citarj.v10i2.424
Publisher
Escola das Artes, Universidade Catolica Portuguesa
Collections
  • Luonnontieteiden ja metsätieteiden tiedekunta
University of Eastern Finland
OpenAccess
eRepo
erepo@uef.fi
Send Feedback
OpenUEF
Service provided by
the University of Eastern Finland Library
Library web pages
Twitter
Facebook
Youtube
Library blog
 

Browse

All of the ArchiveResource types & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

Login
University of Eastern Finland
OpenAccess
eRepo
erepo@uef.fi
Send Feedback
OpenUEF
Service provided by
the University of Eastern Finland Library
Library web pages
Twitter
Facebook
Youtube
Library blog