AI applications<\/a> are being successfully utilized in various different areas to achieve these goals, whilst also being the object of intensive research. AI solutions are making a particularly critical impact on production planning and control (see [10, 4, 9]). <\/p>\n\n\n\nWithin the HaLiMo framework, a range of use cases for AI applications are identified, with the potential to contribute to achieving holistic automation of production planning and control through a multi-agent system approach. In principle, each of the eleven primary production planning and control tasks (see Fig. 1<\/strong>) can be automated using an appropriate AI approach. The multi-agent system can then be implemented so that the individual AI solutions used to automate production planning and control are linked (central AutoPPC) according to their information and material flows (see Fig. 1<\/strong>). <\/p>\n\n\n\nThe added value for both research and industry lies in the fact that implementing a future-proof data infrastructure provides a holistic basis for intelligent, adaptive, and automated production planning and control, incorporating both real-time and historical data, and can be seamlessly integrated into existing systems. In the extant literature (see [9, 10]), the optimization of individual production planning and control subtasks with the assistance of AI is the current norm. However, there is little holistic linkage like that offered by a framework model such as HaLiMo, which is the key contribution that this article makes.<\/p>\n\n\n\n
The subsequent discussion focuses on automating production control tasks using intelligent Reinforcement Learning (RL) agents. To achieve this, the fundamentals of RL are outlined, followed by a demonstration of how companies can develop intelligent agents to automate discrete production control tasks and integrate them comprehensively within their organization.<\/p>\n\n\n\n
RL is a subfield of machine learning in which an agent learns to optimize its actions by interacting with an environment. The agent operates in a defined environment characterized by distinct states. By executing actions, the agent changes the current state and receives feedback as a reward. This reward indicates the extent to which the selected action contributes to achieving the long-term goal [11, 9]. The context and schematic structure are illustrated in Figure 2<\/strong>.<\/p>\n\n\n\n\n
\n
Figure 2: Schematic structure of a Reinforcement Learning agent (following [11]).<\/em><\/figcaption><\/figure>\n<\/div>\n\n\n\n\n
The objective is to devise a strategy that maximizes the cumulative reward over numerous interactions. The existing literature thoroughly outlines the fundamental concepts and methodologies of this approach [11]. Preliminary approaches already demonstrate RL\u2019s potential to automate individual production planning and control tasks and to cope with complexity [12].<\/p>\n<\/div>\n<\/div>\n\n\n\n
For instance, it has been demonstrated that RL agents can outperform established industry standards in time management [3].<\/p>\n\n\n\n
Data infrastructure requirements<\/h2>\n\n\n\n The performance of RL models depends on the quality and variety of the underlying data and information. In production environments, this data can be categorized into three main types:<\/p>\n\n\n\n
\nReal-time data<\/strong>: Production plants continuously generate data streams reflecting the current state of machines, material flow, and production progress. This is a consequence of integrating sensors, IoT devices, and MES [13].<\/li>\n\n\n\nHistorical data<\/strong>: Extensive archives of past production processes are necessary to train and validate RL models. Historical data primarily includes production logs with information on planned values, processing times, and capacities [11, 14].<\/li>\n\n\n\nContext and metadata<\/strong>: Beyond numerical data, information regarding production plans, job priorities, resource availability, and other organizational parameters is crucial. This contextual data, often derived from ERP systems, provides the framework for interpreting production data [15].<\/li>\n<\/ol>\n\n\n\nThe performance of an RL agent is closely linked to the quality of the available data. Incomplete or inconsistent data sets can lead to misinterpretation and faulty learning processes. Therefore, quality assurance processes must be implemented to check the decisions of the RL agent for plausibility and completeness [11, 9]. This necessitates the utilization of contemporary streaming technologies and data pipelines capable of processing substantial data volumes expeditiously [16]. The diversity in data formats from disparate sources poses a significant challenge. Data lakes and ETL (Extract, Transform, Load) processes facilitate efficient aggregation and pre-processing [3, 11].<\/p>\n\n\n\n
Integration of the RL agent into existing systems<\/h2>\n\n\n\n Integrating the RL agent into the existing IT landscape is a prerequisite for realizing synergies between traditional systems and modern algorithms. The primary focus in this regard is on MES and ERP systems. MES\u2019s significant responsibility is the real-time control and monitoring of production processes [17]. The RL agent must be able to access the data provided by MES directly [17, 11].<\/p>\n\n\n\n
In addition, the agent must be able to provide feedback to the system through control commands. ERP systems are designed to manage company-wide data that extends beyond the confines of the production process [17]. To ensure optimal support for the RL agent, it is imperative to integrate and contextualize data from this system. This often necessitates the development of specialized interfaces that facilitate bidirectional communication between the systems, fostering seamless integration and effective collaboration [18].<\/p>\n\n\n\n
Implementation roadmap\u2014from concept to practice<\/h2>\n\n\n\n The successful introduction of an RL agent in practical production planning and control requires a systematic approach that considers both technical and organizational aspects. The following delineates a detailed and practical roadmap that companies can use to guide them from conceptual planning to training, integration, and continuous optimization. The approach is grounded in the widely recognized Cross Industry Standard Process for Data Mining, a standardized procedure for conducting data mining projects [19]. The schematic structure and procedure are illustrated in Figure 3<\/strong>.<\/p>\n\n\n\n\n
\n
Figure 3: Implementation roadmap for integrating a Reinforcement Learning agent in production planning and control.<\/em><\/figcaption><\/figure>\n<\/div>\n\n\n\n\n
Before developing a client-specific RL agent, companies must perform a thorough inventory of their existing IT and production systems. This encompasses documenting prevailing data sources (for example real-time and contextual data) and the interfaces utilized for MES and ERP systems. Furthermore, it is imperative to establish quantifiable key performance indicators (KPIs) that can effectively gauge the efficacy of the RL agent [9]. <\/p>\n<\/div>\n<\/div>\n\n\n\n
The subsequent stage in the process is consolidating the requisite data sources. Ensuring the quality and consistency of the data is paramount and should be facilitated through automated testing and data cleansing processes [11, 14]. Integrating the RL agent into production is contingent upon creating a digital twin or simulation model of the production system. This facilitates risk-free testing and training of the agent under simulated production conditions. Concurrently, a simulation platform should be developed that maps various scenarios, including extreme or rare production situations. This establishes a realistic environment where the RL agent can learn iteratively and refine its decision-making strategies [4, 11].<\/p>\n\n\n\n
The initial training in the simulation is initiated at this stage. The RL agent is first exposed to historical and simulated real-time data, thereby facilitating the development of fundamental decision strategies. Through systematic evaluations and refinements, the model undergoes continuous enhancement. Following the successful completion of preliminary training sessions, a gradual integration of real-time data is initiated. Transitions to a productive environment are emulated through incremental assessments, and the model is fortified against contingencies. This interoperability enables the agent to implement decisions based on real-time feedback while simultaneously feeding relevant data back to the higher-level system [17]. <\/p>\n\n\n\n
Before comprehensive implementation, the RL agent should be deployed in a pilot phase in a controlled area. This pilot phase validates the system\u2019s functionality under real-world conditions while concurrently monitoring and assessing security, data protection, and performance aspects in actual operational settings. After integration, continuous monitoring is imperative. This entails observing the performance of the RL agent, its decisions, and the repercussions of these decisions on production processes. <\/p>\n\n\n\n
Actual \u2018realization\u2019 can be initiated after completing the six steps above. In this context, realization is the practical implementation of the production planning and control tasks, which the RL agent subsequently automates. It is imperative to note that the schematic sequence of the seven steps, as illustrated in Figure 3<\/strong>, should not be interpreted as a one-time process. Instead, it is a process that must be repeatedly executed and adapted to new conditions.<\/p>\n\n\n\nThe system should be designed to continuously acquire knowledge from new data and feedback. Feedback loops between production, IT, and the developers of the RL agent enable the execution of adjustments and the continuous optimization of the model. Implementing an RL agent is known to induce modifications in work processes and necessitate acquiring new skills.<\/p>\n\n\n\n
Consequently, it is incumbent upon companies to promptly allocate resources toward training and continuing education programs. The objective of such initiatives is to facilitate the acquisition of competencies in utilizing novel technology while concurrently addressing concerns and inhibitions that may arise when adopting these new methodologies. This approach fosters not only acceptance but also the attainment of long-term success.<\/p>\n\n\n\n
This roadmap provides companies with a structured approach to successfully transitioning from the conceptual planning stage to the productive use of an RL agent in production planning and control. Companies can minimize risks and achieve sustainable competitive advantages by systematically considering technical, organizational, and security-related aspects.<\/p>\n\n\n\n
Challenges and prospects<\/h2>\n\n\n\n Implementing a holistic data infrastructure for RL in production planning and control poses numerous challenges, including the fact that relevant data is often stored in isolated systems that do not communicate effectively. To overcome these silos, it is necessary to implement not only technical but also organizational measures to achieve company-wide data integration.<\/p>\n\n\n\n
In addition to data integration, the scalability and performance of the RL agent have been known to present challenges. High data volumes and varying loads characterize production environments. Therefore, the data infrastructure must be designed to scale flexibly and handle peak loads. This challenge can be met through distributed database systems and cloud solutions that guarantee high availability and performance. <\/p>\n\n\n\n
A further challenge is the need for interdisciplinary collaboration to successfully implement and integrate such projects into a company\u2019s production process. The success of RL projects in production planning and control is contingent upon close collaboration between data scientists, IT specialists, production logisticians, and security officers.<\/p>\n\n\n\n
It is only through an interdisciplinary approach that the complex challenges of data integration and security can be overcome. Implementing an RL agent should be regarded as a continuous process rather than a one-time undertaking, as it involves the ongoing allocation of resources. The data infrastructure must be flexible to accommodate future requirements, technological advancements, and evolving production processes. Consequently, regular evaluations and updates are imperative.<\/p>\n\n\n\n
To conclude, it can be stated that the implementation of RL in production planning and control necessitates a robust and future-proof data infrastructure. Essential prerequisites include providing high-quality real-time and historical data and seamless integration into existing MES and ERP systems. By leveraging modern technologies such as digital twins and advanced data preprocessing, companies can achieve sustainable enhancement of their production systems\u2019 performance and secure long-term competitive advantages. Interdisciplinary collaboration and continuous data infrastructure development are pivotal to success in an increasingly digitized and networked industrial environment. <\/p>\n\n\n\n
HaLiMo is proposed as a good method for achieving this objective in a structured manner, due to its holistic character and its consideration of interactions, information and material flows in production logistics. Visions for the future could involve automating production planning and control in its entirety. This could be realized through the automation and integration of all production planning and control tasks.<\/p>\n\n\n\n
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) \u2013 SFB 1153 \u2013 252662854.<\/em><\/p>\n\n\n\nThe original German version of this article can be accessed via DOI: 10.30844\/I4SD.25.5.86<\/a><\/strong><\/p>\n
![\"Figure](\"https:\/\/industry-science.com\/wp-content\/uploads\/2025\/09\/Schneider_I4S-25-5_Figure-1-1024x801.jpeg\")
![\"Figure](\"https:\/\/industry-science.com\/wp-content\/uploads\/2025\/09\/Schneider_I4S-25-5_Figure-2-1024x788.jpeg\")
![\"Figure](\"https:\/\/industry-science.com\/wp-content\/uploads\/2025\/09\/Schneider_I4S-25-5_Figure-3-1024x900.jpg\")