Show simple item record

dc.contributor.authorEl Haddad, Kevin
dc.contributor.authorRizk, Yara
dc.contributor.authorHeron, Louise
dc.contributor.authorHajj, Nadine
dc.contributor.authorZhao, Yong
dc.contributor.authorKim, Jaebok
dc.contributor.authorNgo Trong Trung
dc.contributor.authorLee, Minha
dc.contributor.authorDoumit, Marwan
dc.contributor.authorLin, Payton
dc.contributor.authorKim, Yelin
dc.contributor.authorCakmak, Huseyin
dc.date.accessioned2019-01-15T12:17:34Z
dc.date.available2019-01-15T12:17:34Z
dc.date.issued2018
dc.identifier.urihttps://erepo.uef.fi/handle/123456789/7321
dc.description.abstractIn this work, we established the foundations of a framework with the goal to build an end-to-end naturalistic expressive listening agent. The project was split into modules for recognition of the user’s paralinguistic and nonverbal expressions, prediction of the agent’s reactions, synthesis of the agent’s expressions and data recordings of nonverbal conversation expressions. First, a multimodal multitask deep learning-based emotion classification system was built along with a rule-based visual expression detection system. Then several sequence prediction systems for nonverbal expressions were implemented and compared. Also, an audiovisual concatenation-based synthesis system was implemented. Finally, a naturalistic, dyadic emotional conversation database was collected. We report here the work made for each of these modules and our planned future improvements.
dc.language.isoenglanti
dc.publisherEscola das Artes, Universidade Catolica Portuguesa
dc.relation.ispartofseriesJournal of science and technology of the arts
dc.relation.urihttp://dx.doi.org/10.7559/citarj.v10i2.424
dc.rightsAll rights reserved
dc.subjectlistening agent
dc.subjectsmile
dc.subjectlaughter
dc.subjecthead movement
dc.subjecteyebrow movement
dc.subjectspeech emotion recognition
dc.subjectnonverbal expression detection
dc.subjectsequence-to-sequence prediction systems
dc.subjectmultimodal synthesis
dc.subjectnonverbal expression synthesis
dc.subjectemotion database
dc.subjectdyadic conversa
dc.titleEnd-to-End Listening Agent for Audiovisual Emotional and Naturalistic Interactions
dc.description.versionpublished version
dc.contributor.departmentSchool of Computing, activities
uef.solecris.id59592692en
dc.type.publicationTieteelliset aikakauslehtiartikkelit
dc.relation.doi10.7559/citarj.v10i2.424
dc.description.reviewstatuspeerReviewed
dc.format.pagerange49-61
dc.relation.issn1646-9798
dc.relation.issue2
dc.relation.volume10
dc.rights.accesslevelopenAccess
dc.type.okmA1
uef.solecris.openaccessOpen access -julkaisukanavassa ilmestynyt julkaisu


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record