Potentials of Multimodal User Interfaces

A Comparison of Approaches of Human-Machine Interaction for the Digitalized Production Environment

Daniel-Leonhard Fox, André Ullrich ORCID Icon, Norbert Gronau ORCID Icon
JournalIndustrie 4.0 Management
Issue Volume 37, 2021, Edition 5, Pages 49-52
Open Accesshttps://doi.org/10.30844/I40M_21-5_S49-52
Bibliography Share Cite Download

Abstract

Digitalization is changing manufacturing substantially. The design of user interfaces in the digitalized production environment is of utmost importance. In this article, potential combinations of approaches to human-machine interaction are shown and examples of applications are presented. Multimodal user interfaces offer a high degree of immersion. As a result, approaches of (among others) VR-, gesture- and speech-based types of interaction are compared with the dialogue principles and their suitability in practice as work and learning support for employees is presented.

Keywords

, , , ,

Bibliography

[1]  Handbuch Usability: ISO 9241. URL: www.handbuch-usability.de/iso-9241.html, Abrufdatum 15.06.2020.
[2]  Vogel-Heuser, B.; Bauernhansl, T.; Ten Hompel, M. (Hrsg): Handbuch Industrie 4.0. Bd. 4: Allgemeine Grundlagen, 2. Auflage. Berlin 2017.
[3]  Dumas, B.; Lalanne, D.; Oviatt, S.: Multimodal Interfaces: A Survey of Principles, Models and Frameworks. In: Lecture Notes in Computer Science (2009), S. 3-26.
[4]  Deutsche Akademie der Technikwissenschaften (Hrsg): Innovationspotenziale der Mensch-Maschine-Interaktion. München 2016.
[5]  Reply: Zukunftskonzepte für Human Machine Interfaces – So kommen Menschen den Maschinen näher. URL: www.reply.com/de/topics/ artificial-intelligence-and-machine-learning/human-machine-interfaces-trend-report, Abrufdatum 15.06.2020.
[6]  Héder, M.: From NASA to EU: the evolution of the TRL scale in Public Sector Innovation. In: The Innovation Journal 22 (2017) 1.
[7]  Goertz, W.; Gartner: Hype Cycle for Human-Machine Interface. 2018. URL: www.gartner. com/doc/3882879/hype-cycle-humanmachine-interface-, Abrufdatum 15.06.2020.
[8]  Liu, H.; Wang, L.: Gesture recognition for human-robot collaboration: A review. In: International Journal of Industrial Ergonomics 68 (2018), S. 355-367.
[9]  Vukelic, M.; Pollmann, K.; Peissner, M.; Spath, D.: Mensch-Technik-Interaktion mit Emotionen: Eine neurowissenschaftliche Untersuchung zu hirnphysiologischen Prozessen emotional-affektiver Nutzerreaktionen die für technische Assistenz relevant sind. In: Freitag, M. (Hrsg): Mensch-Technik-Interaktion in der digitalisierten Arbeitswelt (2020), S. 39-58.
[10] Mittelstand 4.0 – Kompetenzzentrum Saarbrücken: Brain- Computer-Interfaces: Einsatz von Brain-Computer-Interfaces als Mittel der Mensch-Technik-Interaktion. URL: kompetenzzentrum-saarbruecken.digital/technologieradar-brain-computer-interfaces, Abrufdatum 15.06.2020.
[11] Hochberg, L. R. u. a.: Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. In: Nature 485 (2012) 7398, S. 372-375.
[12] Myers, C.; Furqan, A.; Nebolsky, J.; Caro, K.; Zhu, J.: Patterns for How Users Overcome Obstacles in Voice User Interfaces. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems – CHI ’18 (2018), S. 1-7.
[13] Gorecky, D.; Schmitt, M.; Loskyll, M.; Zühlke D.: Human-machine-interaction in the industry 4.0 era. In: 12th IEEE International Conference on Industrial Informatics (IN-DIN) (2014), S. 289-294.
[14] Oviatt, S.: Advances in robust multimodal interface design. In: IEEE Computer Graphics and Applications 23 (2003) 5, S. 62-68.[15] Bundesministerium für Bildung und Forschung: Logistik 4.0, mit dem Exoskelett SensHand gegen Gesundheitsschäden in der Lagerhaltung. URL: www.wissenschaftsjahr. de/2018/neues-aus-den-arbeitswelten/alle-aktuellen-meldungen/april-2018/ exoskelett-senshand-bietet-sich-als-hebehilfe-in-der-lagerhaltung-an, Abrufdatum 15.06.2020.
[16] Böde, E.; Hartmann, E. A.; Lüdtke, A.; Oppenheimer, F.; Rötting, M.; Wegerich, A.: Mensch-Technik-Interaktion. In: Bundesministerium für Wirtschaft und Technologie (BMWi) Öffentlichkeitsarbeit: Autonomik – Autonome und simulationsbasierte Systeme für den Mittelstand 3. Berlin 2018.
[17] Teichmann, M.; Ullrich, A.; Wenz, J.; Gronau, N.: Herausforderungen und Handlungsempfehlungen betrieblicher Weiterbildungspraxis in Zeiten der Digitalisierung. HMD Praxis der Wirtschaftsinformatik. Faktor Mensch – Erwachsenenbildung. 2020. DOI: 10.1365/s40702-020-00614-x.

Your downloads

You might also be interested in

Serious Games as a Training Tool

Serious Games as a Training Tool

Game mechanics design to promote resilience
Annika Lange ORCID Icon, Thomas Knothe ORCID Icon
Unforeseen events are increasingly challenging manufacturing companies. Being resilient during crises is becoming a key competence. Serious games (SG) can help make resilience-building processes more transparent. This article derives specific requirements for SG from different phases of resilience and shows how these can be implemented in game mechanics in order to effectively support the training of resilience.
Industry 4.0 Science | Volume 42 | 2026 | Edition 2 | Pages 98-104
From Brownfield to Industry 4.0

From Brownfield to Industry 4.0

Learning factories as training and testing environment for digital transformation
Jakob Weber, Sven Völker ORCID Icon
To succeed in their digital transformation, manufacturing companies need engineers with in-depth knowledge of key technologies and concepts, and a profound understanding of the transition from Industry 3.0 to Industry 4.0. This article describes the concept of a learning factory that is continuously subjected to a digital transformation, thereby creating an environment for the development of transformation competencies. The concept of digital transformation is based on digital worker assistance systems and multi-agent systems for production control. These enable the incremental integration of existing resources into the digitalized factory. The learning factory is not presented to students as a completed solution. Instead, it is continuously developed further as part of student projects. This way, it contributes directly to the qualification of personnel for the implementation of Industry 4.0.
Industry 4.0 Science | Volume 42 | 2026 | Edition 2 | Pages 88-96
AI Colleagues?

AI Colleagues?

Competence requirements and training for AI use in industry
Swetlana Franken ORCID Icon
Artificial intelligence is fundamentally changing tasks, roles, and skills in (industrial) companies. Increasingly, it acts as a colleague, preparing decisions, supporting processes, and interacting with people. This article highlights key competence requirements for AI use in industry, presents an integrated competence model, and outlines practical strategies for the transfer of skills. The aim is to prepare companies and employees for humane, competence-oriented AI implementation that combines technological efficiency with human creativity and judgment.
Industry 4.0 Science | Volume 42 | 2026 | Edition 2 | Pages 78-86
Operationalizing Ethical AI with tachAId

Operationalizing Ethical AI with tachAId

Validating an interactive advisory tool in two manufacturing use cases
Pavlos Rath-Manakidis, Henry Huick, Björn Krämer ORCID Icon, Laurenz Wiskott ORCID Icon
Integrating artificial intelligence (AI) into workplace processes promises significant efficiency gains, yet organizations face numerous ethical challenges that stakeholders are often initially unaware of—from opacity in decision-making to algorithmic bias and premature automation risks. This paper presents the design and validation of tachAId, an interactive advisory tool aimed at embedding human-centered ethical considerations into the development of AI solutions. It reports on a validation study conducted across two distinct industrial AI applications with varying AI maturity. tachAId successfully directs attention to critical ethical considerations across the AI solution lifecycle that might be overlooked in technically-focused development. However, the findings also reveal a central tension: while effective in raising awareness, the tool’s non-linear design creates significant usability challenges, indicating a user preference for more structured, linear guidance, especially ...
Industry 4.0 Science | Volume 42 | 2026 | Edition 1 | Pages 50-59 | DOI 10.30844/I4SE.26.1.48
AI Implementation in Industrial Quality Control

AI Implementation in Industrial Quality Control

A design science approach bridging technical and human factors
Erdi Ünal ORCID Icon, Kathrin Nauth ORCID Icon, Pavlos Rath-Manakidis, Jens Pöppelbuß ORCID Icon, Felix Hoenig, Christian Meske ORCID Icon
Artificial intelligence (AI) offers significant potential to enhance industrial quality control, yet successful implementation requires careful consideration of ethical and human factors. This article examines how automated surface inspection systems can be deployed to augment human capabilities while ensuring ethical integration into workflows. Through design science research, twelve stakeholders from six organizations across three continents are interviewed and twelve sociotechnical design requirements are derived. These are organized into pre-implementation and implementation/operation phases, addressing human agency, employee participation, and responsible knowledge management. Key findings include the critical importance of meaningful employee participation during pre-implementation, and maintaining human agency through experiential learning, building on existing expertise. This research contributes to ethical AI workplace implementation by providing guidelines that preserve human ...
Industry 4.0 Science | Volume 42 | 2026 | Edition 1 | Pages 120-127 | DOI 10.30844/I4SE.26.1.112
Applied AI for Human-Centric Assembly Workplace Design

Applied AI for Human-Centric Assembly Workplace Design

An ethics-informed approach
Tadele Belay Tuli ORCID Icon, Michael Jonek ORCID Icon, Sascha Niethammer, Henning Vogler, Martin Manns ORCID Icon
Artificial intelligence (AI) can enhance smart assembly by predicting human motion and adapting workplace design. Using probabilistic models such as Gaussian Mixture Models (GMMs), AI systems anticipate operator actions to improve coordination with robots. However, these predictive systems raise ethical concerns related to safety, fairness, and privacy under the EU AI Act, which classifies them as high-risk. This paper presents a conceptual method integrating probabilistic motion modeling with ethical evaluation via Z-Inspection®. An industrial case study using the Smart Work Assistant (SWA) demonstrates how multimodal sensing (motion, gaze) and interpretable models enable anticipatory assistance. The approach moves from ethics evaluation to ethics-informed work design, yielding transferable principles and a configurable assessment matrix that supports compliance-by-design in collaborative assembly.
Industry 4.0 Science | Volume 42 | 2026 | Edition 1 | Pages 60-68 | DOI 10.30844/I4SE.26.1.58
Christoph Brocks