When toddlers learn to stack blocks, they learn by trial and error — often with immediate feedback from a parent or other adult. It is a model-free learning process, or reinforcement learning, that does not require them to learn Newton’s equation to figure out how to stack the blocks.
For Dr. Hiroaki Kanokogi, Yokogawa Digital Corporation’s President and CEO, reinforcement learning (RL) in artificial intelligence (AI) has direct uses in a manufacturing environment. In fact, as Kanokogi shared during the Manufacturing Leadership Council’s Manufacturing in 2030 Project: Let’s Talk About AI event, RL was a building block when Yokogawa used AI to autonomously control a Japanese chemical plant for 35 days earlier this year.
In his presentation at Let’s Talk About AI, Kanokogi shared that there are serious challenges to applying RL to real world manufacturing. First, he said, traditional RL takes 1 million to 1 billion trials to go beyond human learning, and second, manufacturers must include safety assurances.
To overcome the first of these challenges, Yokogawa and the Nara Institute of Science and Technology developed scalable reinforcement learning called Factorial Kernel Dynamic Policy Programming (FKDPP) specifically for plant control. FKDPP allows for faster learning (typically in about 30 trials) and robust protection against disturbances. Yokogawa was able to demonstrate that FKDPP can autonomously stabilize water levels in a fundamental three tank level control experiment significantly quicker than traditional proportional-integral-derivative (PID) control.
At Let’s Talk About AI, Kanokogi shared four videos that chronicled FKDPP’s iterative attempts to stabilize the water. In the first iteration, AI does not know anything yet, so when the valve is opened the water level goes all the way up. In the 20th iteration, AI can control the water in a somewhat stable manner, but it varies and resembles a human’s performance on the task. For the 25th iteration, AI learns how to regulate the variation. By the 30th iteration, the FKDPP perfects the process. Kanokogi pointed out that this final iteration demonstrated that once AI finds a good way, optimization of this process is AI’s strength.
For the second challenge around safety assurance, Yokogawa was able to prove AI can satisfy this need during this year’s 35-day autonomous factory operation. The company first built a good simulation model by using domain knowledge in a digital twin so the AI could learn. Step two called for simulation and evaluation using both past and live data. Finally, the company ensured safety and control in the actual plant using Yokogawa’s integrated process control system, CENTUM™ VP DCS.
For Yokogawa and its autonomous operation, Kanokogi reported that he and his team continue to look at problems in the factory where AI can be applied. While the first-of-its-kind, 35-day automation demonstration is truly impressive, he sees manufacturing working in an autonomous plan-do-check-act (PDCA) loop by 2030. This loop will run continuously, and AI will help the plant improve itself. While there is no need for human intervention during this loop, Kanokogi pointed out that AI cannot add new sensors or integrate new technologies, so human experts will maintain a defined role in manufacturing.
Like a toddler with blocks, autonomous factory operation might be in its nascent years, but with the help of AI and Yokogawa’s FKDPP technology, maturation by 2030 is possible.
Manufacturing in 2030 Project: Let’s Talk About AI was held Dec. 7-8, 2022 in Nashville, Tenn. The event was part of MLC’s Manufacturing in 2030 Project.