Multimodality. Cognition. Society.

|   August to September 2022   |

   INSTITUTE ON MULTIMODALITY   >   MINDS.  MEDIA.  TECHNOLOGY.
@ZiF - Center for Interdisciplinary Research  /  Bielefeld, Germany - Aug 28 to Sep 6 2022   

   TRAINING SCHOOL   >   REPRESENTATION MEDIATED MULTIMODALITY
EU COST ACTION Multi3Generation  /  @Schloss Etelsen, Germany - Sep 26-30 2022   

The Institute on Multimodality provides a consolidated perspective on the theoretical, methodological, and practical understanding of multimodality research. The institute offers a novel interdisciplinary view of recent advances centralising multimodality research from diverse viewpoints, including:

• artificial intelligence - machine learning
• cognition - media reception
• multimodal interaction - semiotics
• media and communications
• data science - media - practice

Through a confluence of perspectives from Minds, Media, and Technology, the institute will position and enable an understanding of the manner in which multimodality shapes the socio-semiotic interpretation and propagation of interactional, communicative, and collaborative acts/artefacts in everyday life and work.

• Dissemination Event: Summer School + Doctoral Colloquium (consisting of keynotes, lectures, tutorials)
• Closing Workshop of ZiF Research Group on ``Multimodal Rhetoric in Online Media Communications''
• Public Event, colocated as an informal evening session during the closing event

One of the central agenda of the institute pertains to dissemination; through a summer school, for instance, this will be achieved through focussed lectures and tutorial sessions devoted to the topic under consideration. The institute will --through its multidisciplinary faculty and technical programme-- present a comprehensive and outward-looking view of multimodality research at the interface of Minds, Media, and Technology. The scientific agenda of the institute will explicate the relationships between multimodality studies across areas such as cognition, artificial intelligence, visuo-auditory media, and interaction studies, particularly highlighting the significance of multimodality research towards the design and engineering of next-generation digital technologies.

   INSTITUTE ON MULTIMODALITY   >   MINDS.  MEDIA.  TECHNOLOGY.
@ZiF - Center for Interdisciplinary Research  /  Bielefeld, Germany - Aug 28 to Sep 6 2022.  /  JOIN US!   

The training school on ``Representation Mediated Multimodality'' provides a consolidated perspective on the theoretical, methodological, and applied understanding of representation mediated multimodal sensemaking at the interface of language, knowledge representation and reasoning, and visuo-auditory computing. Intended purposes -addressed in the school- encompass diverse operative needs such as explainable multimodal commonsense understanding, multimodal generation/synthesis for communication, multimodal summarisation, multimodal interpretation guided decision-support, adaptation & autonomy, and analytical visualisation.

At the heart of the thematic focus of the school are the themes of grounding and grounded meaning-making:

linguistic expressions and/or relational categorisations are called grounded when they are linked to non-linguistic, especially quantitative perceptual data, such as information coming from modalities such as vision and audition in space-time. Such perceptual data could pertain to, for instance, dynamic spatio-temporal phenomena both in an embodied as well as disembodied interaction context. Grounding is in essence a key aspect of semiotic construction, e.g., enabling high-level meaning acquisition, analogy, and has been a long-standing challenge in Artificial Intelligence and related disciplines.

The training school particularly consolidates and presents a novel and emerging view of state-of-the-art multimodal, neurosymbolic grounding primarily through a confluence of methods in Artficial Intelligence, Computational Linguistics, Vision, Speech and their impact on interactional and communicative acts/artefacts in everyday life and specialist professional work. Here, multimodal interpretation is to be construed in the context of diverse forms of imagery, e.g., encompassing perceptual and communicative data sources such as image, video, language, audition, text, eye-tracking, neurophysiological markers in behavioural / clinical settings.
In this backdrop, the training school particularly focusses on grounded knowledge representation, reasoning, and learning (for linguistic purposes) through the following emerging research-driven themes that are of high interest:

• Space and Language
• Computational Linguistics
• Cognitive Linguistics
• Cognitive Vision
• Vision and Language
• Neurosymbolism
• Narrative - Discourse
• Computational Models of Narrative
• Speech and Language Technologies
• Human-Centred AI

The principal training school programme --consisting of keynotes, invited talks, and tutorials-- will categorically address questions such as:

• Explainability and transparency in multimodal models
• Neurosymbolism / Interaction between symbolic & sub-symbolic representations in models
• Commonsense knowledge representation and reasoning, and its role in multimodal models
• Complementarity / redundancy among data sources or modalities
• Situated reasoning, e.g., for language generation, decision-making

   TRAINING SCHOOL   >   REPRESENTATION MEDIATED MULTIMODALITY
EU COST ACTION Multi3Generation  /  @Schloss Etelsen, Germany - Sep 26-30 2022.  /  JOIN US!