Despite technological advances, challenges remain in implementing LLMs for preserving tacit knowledge in an industrial context. These include, among others:<\/p>\n\n\n\n
- \n
- a lack of validation mechanisms, as LLMs do not have inherent logic checking [17]<\/li>\n\n\n\n
- data protection and security issues when integrating company-specific content [19]<\/li>\n\n\n\n
- acceptance problems at employee level due to skepticism or technical overload [20]<\/li>\n\n\n\n
- unclear responsibilities for maintaining and using the knowledge base [16]<\/li>\n<\/ul>\n\n\n\n
For the successful use of LLMs to collect tacit knowledge, it is vital to design these systems so that they not only function flawlessly in technological terms, but are also easy to use and designed with the user in mind [3].<\/p>\n\n\n\n
During this study, the reliability and quality of knowledge elicitation will be assessed in advance using an LLM as an example. The research question of the extent to which LLMs can support the above-mentioned technical and organizational challenges will therefore be answered. The study does not yet explicitly include verification with other knowledge sources or a discussion of the necessary technical and social framework conditions, such as data security.<\/p>\n\n\n\n
Related study<\/h2>\n\n\n\n
Van den Bent et al. [21] also investigates whether LLMs are suitable for knowledge elicitation. In the study, the knowledge elicitation process consists of an unstructured interview and subsequent ontology creation. An ontology is a method for conceptualizing knowledge and consists of classes, relations, rules, and instances [22]. The duration of the interview and the behavior of the LLM during the interview, as well as the results of ontology creation, are compared with human experts\u2019 results. The study concludes that interviews using OpenAI’s LLM GPT-4 are more structured than interviews conducted by real people and therefore can yield efficiency gains.<\/p>\n\n\n\n
However, the study identifies poorer results in terms of ontology creation. For example, it was found that during ontology creation, the LLM supplements information that is not mentioned in the interviews. Some of this information is factually correct and therefore presumably originates from the LLM’s training data [21]. The present study uses a different methodology than that of van den Bent et al. [21].<\/p>\n\n\n\n
Knowledge elicitation through interviews and summarization<\/h2>\n\n\n\n
During this initial study, a personalized chatbot was developed using OpenAI\u2019s ChatGPT-5 (released in August 2025) that can conduct interviews on any topic with real people. A more recent LLM from the same provider as the related study described above is used.<\/p>\n\n\n\n
The aim of the interviews is to gather tacit knowledge from the interviewee. The interviewees should be able to steer the interview thematically but without deviating from the topic and answer specific questions for further clarification. For this reason, semi-structured interviews are chosen for the study. In contrast to the related study, in which an ontology is created, the results of the interview in this study are output in a structured text file after the interview. This summary could then be integrated as a document straight into a knowledge database.<\/p>\n\n\n\n
When creating the chatbot, the rules of prompt engineering and the procedure for semi-structured interviews are considered. For this purpose, the role, goal, background information, procedure, and response format are specified. In-depth questions are also integrated into the system prompt as examples. To optimize the system prompt and thus the interview behavior, 15 interviews are conducted, and the system prompt is adjusted after each interview using another chatbot (based on ChatGPT-5). For this purpose, the interviews are reviewed in terms of conversation management and summarization, and if the behavior is inadequate, the correct procedure is specified in more detail in the system prompt.<\/p>\n\n\n\n
For the evaluation, three experts are interviewed about nine different topics within the area of production-related innovations and processes. The experts, who are all engineers and experienced with LLMs and production, select the topics. The interview begins with the expert naming the topic, which is then explored in greater detail using questions that the chatbot has tailored to the topic. In contrast, the related study only includes one expert and one topic area. As with [21], a distinction should be made between the evaluation of the interview and the final result, i.e., the summary. The evaluation criteria, including explanations, are shown in Figure 1. <\/p>\n\n\n\n
![\"Figure](\"https:\/\/industry-science.com\/wp-content\/uploads\/2025\/11\/Kuks_I4S-25-6_Figure-1.jpeg\")
Figure 1: Evaluation criteria<\/em>.<\/figcaption><\/figure>\n\n\n\n To ensure that the duration of the interviews is comparable, all experts use ChatGPT’s \u201cVoice Mode.\u201d<\/p>\n\n\n\n
In the study [21], it was noticed that all ontologies created by the LLM contain hallucinated information. The criterion of input fidelity is intended to verify this finding. Information in the summary that does not originate from the interview is considered a negative rating for input fidelity. The criterion of misinformation is introduced as an extension. For this purpose, the interviewee deliberately make a false statement in each interview. The chatbot aims to collect tacit knowledge without verification from other sources of knowledge, so that false statements should also be included. The reason for this is to potentially uncover new ideas and concepts by individual employees that are described in existing sources of knowledge using a different approach.<\/p>\n\n\n\n
After their own interview, each expert evaluates it and the summary based on the criteria described. The experts then assess the clarity of the other candidates’ summaries. The conversation atmosphere is evaluated by the expert after all their own interviews. The criteria of breadth, depth, atmosphere of the conversation, completeness, relevance, input fidelity, and clarity are evaluated subjectively, i.e., the expert chooses between \u201cfully applies,\u201d \u201crather applies,\u201d \u201crather does not apply,\u201d and \u201cdoes not apply at all.\u201d For duration, the time is measured; for redundancy, the number of duplicate entries is counted; and for misinformation, it is checked whether the false statement is included in the summary.<\/p>\n\n\n\n
An interview achieves better results than a summary<\/h2>\n\n\n\n
The results of the study are shown in Figure 2, and the average values of all criteria with the same rating (one to four) are shown in a bar chart in Figure 3 for clear comparison.<\/p>\n\n\n\n
![\"Figure](\"https:\/\/industry-science.com\/wp-content\/uploads\/2025\/11\/Kuks_Figure-2-1024x738.jpeg\")
Figure 2: Results table.<\/em><\/figcaption><\/figure>\n\n\n\n ![\"Figure](\"https:\/\/industry-science.com\/wp-content\/uploads\/2025\/11\/Kuks_I4S-25-6_Figure-3-1024x676.jpeg\")
Figure 3: Bar chart with average values.<\/em><\/figcaption><\/figure>\n\n\n\n The results of the study show that LLMs have plenty of potential when it comes to interviews. It’s worth noting that the chatbot is better at asking in-depth questions than broad ones. Interviews last about ten minutes on average, so they don’t take up too much time. The atmosphere surrounding, the conversation also scores well. This helps counteract the employee acceptance issues mentioned above.<\/p>\n\n\n\n
The values are more volatile in the summary. Completeness, in particular, only scores 2.89 points. As a result, some information from the interview is not included in the summary. Since the goal of the chatbot is to gather implicit knowledge, a complete summary is essential. The other criteria for the summary achieve better results. The LLM seems to be able to filter out unimportant information to a large extent (relevance), and the summary is formulated in a way that is easy to understand\u2014even for people who do not have the expert knowledge (clarity).<\/p>\n\n\n\n
However, as it is being incomplete, the result should be viewed in a more nuanced way. The summary contains hardly any irrelevant information, but some important information is also missing. Accordingly, it can be concluded that the chatbot is most likely unable to reliably distinguish between relevant and irrelevant information. On average, each summary contains 2.33 duplicate pieces of information (redundancy). However, total elimination of duplicate information is desirable.<\/p>\n\n\n\n
Compared to the results from [21], the results for input fidelity and misinformation are more positive. One possible explanation for the improvement may lie in the use of a more advanced model and the different way in which results were output and presented. This, in turn, suggests that a summary constitutes a more suitable representation for collecting implicit knowledge with an LLM than an ontology. In most summaries, the LLM uses only information from the interview, and even factually incorrect information is included in six out of nine interviews.<\/p>\n\n\n\n
Potentials and premises<\/h2>\n\n\n\n
This study was conducted approximately six weeks after the release of the ChatGPT-5 model. The interviewees described the conversation atmosphere as pleasant. In the evaluation, as the focus in creating the chatbot was on optimizing its interview behavior, the interview performed better than the summary. This result can be improved by increasing the number of preliminary interviews to optimize the system prompt. Moreover, the better performance regarding input fidelity and misinformation compared to the hallucination in the related study is also noteworthy. However, as mentioned in the previous chapter, these results should not be seen as universal. Especially in practical applications, the results for completeness in particular show potential for optimization.<\/p>\n\n\n\n
Nevertheless, as this initial study demonstrates, LLMs, and ChatGPT-5 in particular, have great potential in the field of tacit knowledge elicitation. One approach may be the use of a human expert to check for consistency in the summarization process. This would also address the challenge mentioned at the outset, namely that LLMs lack an inherent logic check.<\/p>\n\n\n\n
Perspectives for further research<\/h2>\n\n\n\n
This study provides research institutions with a starting point for further investigation into the subject and encourages companies to consider using LLMs when capturing implicit knowledge. In the future, we plan to conduct numerous additional interviews with external experts and to perform technology benchmarking with other LLMs. Interviews with experts who are not adept at using LLMs can help obtain more nuanced results. The results of our preliminary study fundamentally support a more advanced and programmatically more sophisticated prototype for preserving tacit knowledge in the production environment.<\/p>\n\n\n\n
This prototype will be tested, validated, and specifically optimized in various manufacturing companies while focusing on improving the interview output, i.e., the summary. Its aim is to preserve knowledge that would otherwise be lost due to demographic change and, at the same time, to strengthen the manufacturing industry\u2019s competitiveness in the long term. Besides being verified against other sources of knowledge, further studies will examine critical considerations such as data security and ethical issues. This initial study is limited to the technical and organizational challenges of capturing tacit knowledge.<\/p>\n\n\n\n