Multimodality. Cognition. Society.

|   August to September 2022   |

@ZiF - Center for Interdisciplinary Research  /  Bielefeld, Germany - Aug 28 to Sep 6 2022   

EU COST ACTION Multi3Generation  /  @Schloss Etelsen, Germany - Sep 26-30 2022   


The Institute on Multimodality provides a consolidated perspective on the theoretical, methodological, and practical understanding of multimodality research. The institute offers a novel interdisciplinary view of recent advances centralising multimodality research from diverse viewpoints, including:

  • artificial intelligence - machine learning
  • cognition - media reception
  • multimodal interaction - semiotics
  • media and communications
  • data science - media - practice
Through a confluence of perspectives from Minds, Media, and Technology, the institute will position and enable an understanding of the manner in which multimodality shapes the socio-semiotic interpretation and propagation of interactional, communicative, and collaborative acts/artefacts in everyday life and work.
The Institute on Multimodality features the following:

  • Dissemination Event: Summer School + Doctoral Colloquium (consisting of keynotes, lectures, tutorials)
  • Closing Workshop of ZiF Research Group on ``Multimodal Rhetoric in Online Media Communications''
  • Public Event, colocated as an informal evening session during the closing event

One of the central agenda of the institute pertains to dissemination; through a summer school, for instance, this will be achieved through focussed lectures and tutorial sessions devoted to the topic under consideration. The institute will --through its multidisciplinary faculty and technical programme-- present a comprehensive and outward-looking view of multimodality research at the interface of Minds, Media, and Technology. The scientific agenda of the institute will explicate the relationships between multimodality studies across areas such as cognition, artificial intelligence, visuo-auditory media, and interaction studies, particularly highlighting the significance of multimodality research towards the design and engineering of next-generation digital technologies.

@ZiF - Center for Interdisciplinary Research  /  Bielefeld, Germany - Aug 28 to Sep 6 2022.  /  JOIN US!   


Representation Mediated Multimodality:  
Grounded Representation, Reasoning, and Learning for Interactive Interpretation

The training school on ``Representation Mediated Multimodality'' provides a consolidated perspective on the theoretical, methodological, and applied understanding of representation mediated multimodal sensemaking at the interface of language, knowledge representation and reasoning, and visuo-auditory computing. Intended purposes -addressed in the school- encompass diverse operative needs such as explainable multimodal commonsense understanding, multimodal generation/synthesis for communication, multimodal summarisation, multimodal interpretation guided decision-support, adaptation & autonomy, and analytical visualisation.

At the heart of the thematic focus of the school are the themes of grounding and grounded meaning-making:

linguistic expressions and/or relational categorisations are called grounded when they are linked to non-linguistic, especially quantitative perceptual data, such as information coming from modalities such as vision and audition in space-time. Such perceptual data could pertain to, for instance, dynamic spatio-temporal phenomena both in an embodied as well as disembodied interaction context. Grounding is in essence a key aspect of semiotic construction, e.g., enabling high-level meaning acquisition, analogy, and has been a long-standing challenge in Artificial Intelligence and related disciplines.

The training school particularly consolidates and presents a novel and emerging view of state-of-the-art multimodal, neurosymbolic grounding primarily through a confluence of methods in Artficial Intelligence, Computational Linguistics, Vision, Speech and their impact on interactional and communicative acts/artefacts in everyday life and specialist professional work. Here, multimodal interpretation is to be construed in the context of diverse forms of imagery, e.g., encompassing perceptual and communicative data sources such as image, video, language, audition, text, eye-tracking, neurophysiological markers in behavioural / clinical settings.
In this backdrop, the training school particularly focusses on grounded knowledge representation, reasoning, and learning (for linguistic purposes) through the following emerging research-driven themes that are of high interest:

  • Space and Language
  • Computational Linguistics
  • Cognitive Linguistics
  • Cognitive Vision
  • Vision and Language
  • Neurosymbolism
  • Narrative - Discourse
  • Computational Models of Narrative
  • Speech and Language Technologies
  • Human-Centred AI

The principal training school programme --consisting of keynotes, invited talks, and tutorials-- will categorically address questions such as:

  • Explainability and transparency in multimodal models
  • Neurosymbolism / Interaction between symbolic & sub-symbolic representations in models
  • Commonsense knowledge representation and reasoning, and its role in multimodal models
  • Complementarity / redundancy among data sources or modalities
  • Situated reasoning, e.g., for language generation, decision-making
Of particular significance to the training school are themes that benefit both public and private human-centred technological services most directly affecting EU citizens in an accessible, multilingual Europe, and have strong economic and societal impacts. The training school will focus on human-machine interaction considerations in select application areas of emerging societal significance, such as autonomous systems, cognitive robotics, multi-lingual language technologies.

EU COST ACTION Multi3Generation  /  @Schloss Etelsen, Germany - Sep 26-30 2022.  /  JOIN US!