{"id":114734,"date":"2026-09-04T08:41:09","date_gmt":"2026-09-04T06:41:09","guid":{"rendered":"https:\/\/industry-science.com\/?post_type=article&p=114734"},"modified":"2026-09-07T21:31:41","modified_gmt":"2026-09-07T19:31:41","slug":"competence-assurance-for-the-verification-and-validation-of-simulation-models","status":"publish","type":"article","link":"https:\/\/industry-science.com\/en\/articles\/competence-assurance-for-the-verification-and-validation-of-simulation-models\/","title":{"rendered":"Competence Assurance for the Verification and Validation of Simulation Models"},"content":{"rendered":"\n

Simulation-based analysis of production and logistics systems <\/a>has proven its worth across various industries [1]. To ensure that the use of simulation yields reliable results for the planning and operation of production and logistics systems, it is essential to ensure the validity and credibility of the information, data, and models used.<\/p>\n\n\n\n

However, a survey [2] conducted by the working group “Simulation in Production and Logistics” of the association for simulation (ASIM \u2013 Arbeitsgemeinschaft Simulation)shows that verification and validation (V&V) are typically carried out pragmatically in practice (i.e., only to a limited extent systematically or consistently)\u2014simulation-project-related issues often require rapid insights, so ensuring validity is sometimes viewed as secondary. However, this is risky, as it can result in undetected errors in the simulation models (and thus also in the results), which can lead to incorrect decisions and must be corrected at a later stage.<\/p>\n\n\n\n

Nevertheless, it should be noted that the application of discrete-event simulation in production and logistics is also subject to change related to demographic shifts [3] as well as the areas of application within the system life cycle, which drive the development and use of assistive technologies [3]. Wenzel et al. [3] highlight the range of assistance forms (e.g., digital assistance systems) for simulation applications, which not only have the capability of providing operational support in carrying out tasks related to simulation applications but also ensure the competence of simulation experts. The authors demonstrate that the availability of such an assistance in the context of V&V has not yet reached a comparable level to that in, for example, model development or the experimentation phase (see [4] for the phases of a simulation study).<\/p>\n\n\n\n

The selection and application of suitable V&V techniques (for an overview of these techniques, see [5]) as well as the assessment of the validity of information, data, and models require comprehensive domain-specific expertise. This paper addresses this issue by analyzing the requirements for an AI-supported assistance for the V&V of simulation models in production and logistics, based on literature-derived and empirically evaluated user stories. It investigates which functions are necessary to provide operational support for the intrinsic V&V of a simulation model (i.e., the verification of the model\u2019s internal consistency, completeness, and plausibility) as well as for V&V against the task specification (i.e., the suitability of the model for its intended purpose).<\/p>\n\n\n\n

Beyond preserving simulation experts\u2019 existing methodological and domain-specific expertise through AI in the context of V&V, this paper also identifies the AI-related framework conditions that serve as fundamental technological prerequisites for practical industrial application.<\/p>\n\n\n\n

Terminological classification and scientific delimitation<\/h2>\n\n\n\n

A systematic simulation procedure model [6] was established for simulation applications in production and logistics. Starting with the sponsor needs, this model encompasses the phases of task definition, system analysis, model formalization, implementation, as well as experiments and analysis. The results of the preceding phases (i.e., the information, data, and models) form the basis for the subsequent phase. V&V involves verifying all results for correctness (the subject of verification: \u201cIs the phase result right?\u201d) as well as their suitability for the defined investigation objectives (the subject of validation: \u201cIs it the right phase result?\u201d) [7, 8].<\/p>\n\n\n\n

V&V can be performed either intrinsically or across phases. Intrinsic check focuses, among other things, on the plausibility, completeness, and consistency of the result of a single phase without considering the results of other phases. Cross-phase V&V, in contrast, focuses on checking the correct transformation of a phase result derived from a previous phase result. This paper addresses exclusively the intrinsic V&V of the executable model and its V&V against the task specification. For this purpose, various V&V techniques (test methods) are available, which differ in their applicability and objectivity [5].<\/p>\n\n\n\n

This paper examines the requirements for AI-supported assistance to faciliate the execution of V&V activities. The aim is to ensure the competence of simulation experts in the field of V&V. While there is no uniform understanding of the term of competence in the relevant literature, it is discussed to some extent in specific works (e.g., in [9]). In the context of this paper, competence is understood as knowledge related to the V&V of simulation models that is extended by a situational readiness for application and internalized by the simulation expert in the form of skills and abilities.<\/p>\n\n\n\n

Situational readiness for application combines knowledge of V&V techniques with knowledge of how to apply these techniques (experience) in relation to a specific application context (assessability\/judgment). While competence development aims at the continuous expansion and systematic acquisition of new skills that were previously unavailable [10, 11], competence assurance (or competence retention) focuses on the preservation and methodical consolidation of existing competencies during ongoing operations.<\/p>\n\n\n\n

This form of competence assurance is particularly relevant in automated and digitalized work environments [12]. As operational routine tasks are increasingly taken over by software systems (e.g., automated consistency checks of simulation models), competencies may be applied less frequently and consequently risk being lost over time [12]. To counteract this gradual loss of expertise, this paper explores the conceptualization of an AI-supported assistance. According to Wenzel et al.\u2019s definition [3], assistance includes, among other things, computer-based tools designed to reduce workload. In the context of this paper, an AI-supported assistance system is defined as a computer-based tool that utilizes AI models. Accordingly, the development of such an assistance system can be regarded as a form of software engineering.<\/p>\n\n\n\n

Software engineering typically entails a requirements analysis to identify the users\u2019 objectives and needs [13]. This process initially includes a stakeholder analysis, which involves identifying the relevant individuals and their objectives. For the identified groups, the relevant functions of the software to be developed are elicited. For this purpose, use cases are defined and formulated in the form of so-called user stories as structured statements following the format: \u201c[Role] requires [function] in order to obtain [benefit].\u201d<\/p>\n\n\n\n

These user stories subsequently serve as the foundation for the technical implementation of the software. In this paper, such a requirements analysis is performed for an AI-supported assistance for the V&V of simulation models.<\/p>\n\n\n\n

Requirements analysis of AI-supported assistance<\/h2>\n\n\n\n

For the AI-supported assistance for V&V of simulation models, the primary stakeholders are experts who build, verify, and validate such models. The methodological approach for determining the functional and non-functional requirements of the AI-supported assistance follows a two-step process. In the first stage, the user stories are developed deductively by the authors based on their expertise in the field of V&V.<\/p>\n\n\n\n

The functional and methodological basis is the previously defined purpose of the AI-supported assistance, namely to faciliate both the conducting of the intrinsic V&V of the simulation model and its check against the task specification (Figure 1: E1\u2013E6), as well as competence assurance (Figure 1: K1\u2013K7). The primary focus of these requirements is to preserve the V&V competence of simulation experts rather than to promote progressive competence development or the acquisition of entirely new V&V competencies. However, a strict separation between competence assurance and development is considered impractical and is therefore not enforced within the user stories.<\/p>\n\n\n\n

The user stories pertain to the application of V&V techniques; however, they are not sufficient on their own, as the use of AI involves inherent system-related requirements. These are represented by so-called user stories describing system-related framework conditions (Figure 1: R1\u2013R3), which include technical and legal requirements necessary for productive use in an industrial environment. To intentionally leave the specific user role open, the user stories are formulated in a modified first-person structure (“I want [function] in order to achieve [benefit]”) while preserving the general user story format. In the second stage, the user stories are evaluated by simulation experts. Based on the respondents’ self-assessment, different experience groups can be established, which may reveal statistically significant differences in response behavior.<\/p>\n\n\n\n

User stories for V&V applications<\/h2>\n\n\n\n

This area focuses on providing technical and methodological support for conducting V&V. The assistance supports users in selecting a V&V technique suitable for their respective task specification (E1). By automatically checking the consistency and plausibility of the simulation model (E3), as well as its suitability for the intended purpose (E2), the system reduces the likelihood of errors and relieves experts of routine tasks. The relevance of this category lies in improving the execution of simulation studies and extending the methodological application of V&V by recommending rarely used techniques (E4 and E5) and by providing automated, standards-compliant documentation (E6).<\/p>\n\n\n\n

\"Figure
Figure 1: Overview of user stories.<\/em><\/figcaption><\/figure>\n\n\n\n

User stories for competence assurance<\/h2>\n\n\n\n

User stories in this category focus on the didactic interaction between humans and AI. The aim is to ensure that simulation experts remain actively involved in the problem-solving process (K3) rather than accepting AI recommendations without critical reflection. Sustainable competence assurance is promoted through interactive, model-specific questioning (K1), suggested simulation-model-related prompts (K2), and corrective measures (K5), as well as the provision of explanations based on standards and manuals (K4, K6, and K7). The relevance of this category stems from the need to prevent blind trust in AI systems and to preserve the judgment of simulation experts in the long term by ensuring the explainability of AI-generated recommendations.<\/p>\n\n\n\n

User Stories for system-related framework conditions<\/h2>\n\n\n\n

This category defines the necessary technical and legal prerequisites for the productive use of AI assistance in industrial environments. It focuses on the protection of trade secrets and adherence to compliance guidelines through strict data protection for sensitive model-related and personal data (R1). In addition, the user stories address requirements for scalability (i.e., maintaining responsive performance even when processing large simulation models; R2) and tool independence (R3). These framework conditions are essential for user acceptance, as only a high-performing, secure, and tool-independent system is likely to be successfully adopted in practice.<\/p>\n\n\n\n

Evaluation of the user stories<\/h2>\n\n\n\n

The aim of the evaluation is to ensure the theoretically derived user stories in terms of their technical accuracy and their relevance in an industrial context. To this end, an anonymous expert survey is conducted online using LimeSurvey among members of the ASIM working group “Simulation in Production and Logistics” [2].<\/p>\n\n\n\n

The survey consists of four parts. In the first part, respondents characterizes themselves based on their professional experience and their current field of work (i.e., services, research, or industry). They are also asked to assess whether V&V is typically performed in a pragmatic manner in practice and whether the use of AI-supported assistance for the V&V of simulation models in production and logistics would be beneficial. Responses are collected using a four-point Likert scale (“strongly agree,” “agree,” “disagree,” and “strongly disagree”), supplemented by a “don’t know” option.<\/p>\n\n\n\n

The second part consists of a dual matrix question designed to evaluate the 16 user stories in terms of both the suitability of the proposed functionality and its benefit for maintaining the respondents’ own V&V competence. Each user story is assessed using the same four-point Likert scale, with an additional “don’t know” response option available for every item. This design enables the identification of potential discrepancies between theoretically desirable functionalities and those perceived as practically valuable.<\/p>\n\n\n\n

In addition to the matrix questions, an open-ended text field is provided at the end of this part for qualitative feedback. In the third part, respondents are asked to select the five user stories they considered most relevant. Finally, participants are invited to suggest additional user stories for an AI-supported assistance for the V&V of simulation models.<\/p>\n\n\n\n

Discussion of the user stories<\/h2>\n\n\n\n

The survey results (see [2]) indicate that the respondents (N = 27) are predominantly highly experienced, with 52% reporting more than ten years of professional experience. Most participants work in research (51.9%), followed by industry (29.6%) and services (18.5%). Overall, the experts confirm that V&V is typically carried out pragmatically in practice and regarded AI-supported assistance for the V&V of simulation models as beneficial. The following discussion is organized according to the prioritized relevance of the user stories (Figure 2<\/strong>) and the three functional categories (E, K, and R), complemented by the evaluation of each user story regarding the suitability of its functionality and its benefit for competence assurance (Figure 3<\/strong>).<\/p>\n\n\n\n

\"Figure
Figure 2: Frequency distribution of user story selections (five mentions; N = 27).<\/em><\/figcaption><\/figure>\n\n\n\n

Within the V&V application category (E1\u2013E6), respondents express strong support for AI-assisted operational functions (Figure 3). The user story describing the AI-based generation of a standards-compliant report (E6) receives the highest priority (Figure 2<\/strong>) and is also rated highly in terms of functional suitability, with approximately 70% of respondents selecting “strongly agree.” Similarly, the automated assessment of model consistency and plausibility (E3) and the recommendation of appropriate V&V techniques based on the task specification (E1) are considered highly relevant.<\/p>\n\n\n\n

These findings indicate that respondents view AI-supported assistance not only as a means of preventing errors and selecting suitable V&V techniques but also as valuable support for methodological decision-making. This is also evident when considering the benefit for competence assurance. In particular, user stories E2 (assessment of model suitability) and E6 (automated report generation) receive substantially lower ratings in this regard.<\/p>\n\n\n\n

Although automated report generation (E6) is regarded as highly suitable, only approximately 26% of respondents strongly agree that it contributes to maintaining their own competence. For the automated assessment of model suitability (E2), negative responses even outweigh the positive ones (Figure 3). These findings empirically demonstrate that respondents primarily seek relief from time-consuming V&V activities, whereas their own learning is considered secondary. Consistent with this interpretation, providing supplementary information on unfamiliar or rarely used V&V techniques (E5) is among the least frequently selected user stories.<\/p>\n\n\n\n

Interestingly, for the learning-oriented functions E4 (recommendation of rarely used V&V techniques) and E5 (supplementary information on V&V techniques), the value attributed to them in terms of competence assurance is significantly higher than their purely operational utility (Figure 3). The experts thus clearly recognize the didactic potential of these functions.<\/p>\n\n\n\n

When examining the specific user stories in the context of competence assurance (K1 through K7), the high relevance of detailed explanations and explainability (K4 and K7) is noteworthy. Explainability of AI recommendations (K7) receives approximately 70% “strongly agree” responses regarding functional suitability and about 67% regarding competence assurance. Likewise, detailed explanations of AI recommendations (K4) achieve approximately 63% overall agreement concerning their benefit for competence assurance. These findings suggest that respondents are generally unwilling to invest substantial additional learning time\u2014for example, by consulting standards or technical manuals\u2014but nevertheless have a strong desire to understand the rationale behind AI-generated recommendations. Their primary motivation appears to be building trust in the AI-supported assistance while maintaining ultimate responsibility for decision-making.<\/p>\n\n\n\n

Functions supporting error handling (K3 and K5) are also perceived as highly relevant. In particular, involving users in the problem-solving process when errors occur (K3) is considered by more than 55% of respondents to make a substantial benefit for maintaining professional competence. In contrast, user stories describing AI-generated simulation model-specific prompts (K2) and those featuring recommendations explicitly linked to identifiable sources such as standards, guidelines, manuals, or model logic (K6) are among the least frequently prioritized.<\/p>\n\n\n\n

The recommendation of simulation model-specific prompts (K2) receives the lowest rating for competence assurance, with only 15% of respondents selecting “strongly agree.” Instead, respondents express a clear preference for remaining actively involved in problem solving and for receiving suggestions for corrective actions. Overall, these findings reinforce the importance of transparent and explainable AI over fully automated execution of V&V activities. Rather than delegating responsibility to AI, respondents wish to learn from errors and avoid repeating them in the future.<\/p>\n\n\n\n

Furthermore, the fundamental role of system-related framework conditions (R1\u2013R3) is confirmed by the predominantly positive assessment of the functions\u2019 practicality (Figure 3<\/strong>). In addition, the importance of data protection (R1) is prioritized (Figure 2<\/strong>). Thus, adherence to compliance guidelines and the protection of sensitive model data represent an absolute prerequisite for the acceptance of AI-supported assistance in industrial environments. Accordingly, the protection of sensitive data (R1) and the processing of large simulation models (R2) receive the highest number of \u201cstrongly agree\u201d responses regarding their practicality across all user stories (Figure 3<\/strong>).<\/p>\n\n\n\n

This demonstrates that data protection and model-based scalability are not optional requirements but essential in practice. As expected, the ratings regarding competence assurance are lower across all three system-related framework conditions, since these represent a fundamental prerequisite for the implementation of AI-supported assistance. While they are necessary for the physical deployment and acceptance in industrial practice, their benefit for maintaining simulation experts’ competence is largely indirect.<\/p>\n\n\n\n

\"Figure
Figure 3: Evaluation of the 16 user stories regarding the suitability of their functionality and their benefit for competence assurance (percentages; N = 27).<\/em><\/figcaption><\/figure>\n\n\n\n

In addition to the 16 evaluated user stories, respondents formulate further relevant user stories that address the technological, methodological, and organizational capabilities of AI-supported assistance. Among the technological and methodological functions, respondents emphasize AI-supported parameter selection and parameter variation within task-specific constraints to ensure an adequate testing range. To facilitate interdisciplinary collaboration, participants also propose generating different views of the model logic to provide target group specific discussion material during V&V\u2014for example, for process owners or management.<\/p>\n\n\n\n

In this context, the configurability of AI-powered assistance could be suitable. For example, a flexible filtering mechanism for different user groups could dynamically switch between \u201cverified knowledge\u201d (based on standards, guidelines, and manuals) and \u201copen exploratory search.\u201d While the former minimizes potential errors for less experienced users, the latter allows experienced users to expand the solution space and utilize innovative V&V techniques. From an infrastructure perspective, the necessity of on-premise hosting is highlighted as a critical requirement to ensure control over sensitive corporate data. Respondents also propose functionality for capturing users’ reasoning during the V&V process and for automatically comparing the simulation model documentation with its actual implementation.<\/p>\n\n\n\n

Summary and open research topics<\/h2>\n\n\n\n

Based on a methodological requirements analysis and a subsequent expert survey, this paper demonstrates that AI-supported assistance holds significant potential for ensuring competence in the V&V of simulation models in production and logistics. The evaluation confirms broad acceptance of AI-supported functionalities, particularly those that provide decision support, explainable recommendations, and robust data protection. In addition, respondents identify valuable requirements and practical needs that provide important directions for future research.<\/p>\n\n\n\n

Further analysis of the user stories is necessary to support software engineering process. In addition to taking other stakeholders and their needs into account, the goal is to prioritize requirements\u2014for example, using the MoSCoW method (\u201cMust-have,\u201d \u201cShould-have,\u201d \u201cCould-have,\u201d \u201cWon\u2019t-have\u201d). This should be combined with a detailed cost-benefit analysis.<\/p>\n\n\n\n

Thus, for each user story, it must be assessed whether the benefits achieved justify the development effort. Further research is needed to incorporate the knowledge of simulation experts into AI-supported assistance and to determine the objective information requirements for the AI models\u2014in addition to the actual simulation model\u2014in order to achieve satisfactory support. Furthermore, it is necessary to investigate which AI approaches can be used for the various V&V activities, for example, in parameter selection and automated testing.<\/p>\n\n\n\n

The requirements analysis presented in this contribution forms the foundation for the development of an AI-assistance that is explicitly intended not to replace humans but to empower them in performing complex V&V activities. In this context, the successful implementation of an AI-supported assistance in simulation-based applications therefore depends not only on technological feasibility but, more importantly, on sound methodological implementation, transparent and explainable decision-making, and careful consideration of industrial requirements and constraints.<\/p>\n

Bibliography <\/h2>[1] Sutherland, R.; \u00d6zkul, F.; Wenzel, S.: Studie zum Einsatz der ereignisdiskreten Simulation in Produktion und Logistik. In: Rose, O.; Uhlig, T. (Eds.): ASIM SST 2024 Tagungsband Langbeitr\u00e4ge. Neubiberg 2024.\r
[2] Sutherland, R.; \u00d6zkul, F.; Wenzel, S.: Datensatz: Expertenbefragung zu User-Stories f\u00fcr eine KI-gest\u00fctzte Assistenz zur Verifikation und Validierung (V&V) von Simulationsmodellen in Produktion und Logistik. URL: https:\/\/doi.org\/10.48662\/daks-517, accessed 11.06.2026.\r
[3] Wenzel, S.; \u00d6zkul, F.; Sutherland, R.: 2025. Assistenz f\u00fcr die Simulation in Produktion und Logistik \u2013 Eine literaturbasierte Einordnung. Industry 4.0 Science 41 (2025) 5, pp. 66-76.\r
[4] Gutenschwager, K.; Rabe, M. u. a.: Simulation in Produktion und Logistik: Grundlagen und Anwendungen. Berlin, Heidelberg 2017.\r
[5] Rabe, M.; Spieckermann, S.; Wenzel, S.: Verifikation und Validierung fu\u0308r die Simulation in Produktion und Logistik \u2013 Vorgehensmodelle und Techniken. Berlin, Heidelberg 2008.\r
[6] VDI: VDI-3633 – Part 1: Simulation of systems in materials handling, logistics and production: Fundamentals. Berlin 2014.\r
[7] Balci, O.: Validation, Verification, and Certification of Modeling and Simulation Applications. In: Chick, S.; Sanchez, P.J. u. a. (Eds.): Proceedings of the 2003 Winter Simulation Conference. New Orleans (USA) 2003.\r
[8] Sargent, R. G.: Verification and Validation of Simulation Models. In: Johansson, B.; Jain, S. u. a. (Hrsg.): Proceedings of the 2010 Winter Simulation Conference. Baltimore (USA) 2010.\r
[9] B\u00fcnning, F.; Hortsch, H.: Kompetenz versus competence \u2013 Etymologische Untersuchung und Bedeutungsanalyse des Kompetenzbegriffs im Deutschen und im Englischen. IPTB Preprint Journal (Online Working Papers der Professur f\u00fcr Ingenieurp\u00e4dagogik und Didaktik der technischen Bildung) 2 (2022) 4, pp. 1-13.\r
[10] North, K.; Reinhardt, K.; Sieber-Suter, B.: Kompetenzmanagement in der Praxis \u2013 Mitarbeiterkompetenzen systematisch identifizieren, nutzen und entwickeln. Mit vielen Praxisbeispielen. Wiesbaden 2018.\r
[11] Kauffeld, S.; Grote, S.; Frieling, E.: Handbuch Kompetenzentwicklung. M\u00fcnchen 2009.\r
[12] Conein, S.; Felkl, T.: Kompetenzerhalt fu\u0308r Nicht-Routine-Situationen an hochautomatisierten Arbeitspl\u00e4tzen der chemischen und pharmazeutischen Produktion. Zeitschrift f\u00fcr Arbeitswissenschaften 77 (2023) 2, pp. 230-242. \r
[13] Kleuker, S.: Grundkurs Software-Engineering mit UML \u2013 Der pragmatische Weg zu erfolgreichen Softwareprojekten. Wiesbaden 2025.\r
<\/div>

Your downloads<\/h2>