Digital transformation has arrived for the first time in frontline work. Augmented reality technology has the power to boost frontline worker productivity by as much as 50% and reduce human errors by up to a whopping 90%. Transforming frontline work is in its early stages, and rapid learning is underway. But the effect on overall productivity may prove to have the greatest digital impact so far.
By Michael E. Porter and James E. Heppelmann

 

 

The coronavirus crisis of 2020 has changed the way we work, perhaps indefinitely. Travel bans, lockdowns, and social distancing policies disrupted business around the world, forcing companies to find new ways to work. The trend toward a more distributed, mobile, and agile workforce was already unmistakable, but the crisis has dramatically accelerated this shift and brought the use of digital technology to a new level. This is especially true for knowledge workers with the skills to operate in the virtual world. The even bigger news coming out of the crisis, however, is the change in frontline work. The far broader frontline workforce, who work in the physical world, have become for the first time part of the digital transformation in a meaningful way.

Knowledge workers, typically in roles such as management, finance, marketing, scheduling, or product development, trade in information that is readily captured in digital files and databases. Their work is easily exchanged via emails. Virtual meetings to collaborate with peers around the globe are a few clicks away. When the work is digital, location doesn’t really matter. When knowledge workers around the globe were forced into a massive work-from-home experiment with little planning, the results proved better than many thought possible. Not only did productive work continue, but additional time and cost savings were realized from reduced commuting time and business travel.

The work-from-home Genie probably won’t go back into the bottle, even when the crisis passes. This way of working will become part of the new normal because of its inherent advantages. But for frontline work, a new revolution is in the making.

How Frontline Work is Different 

Not all workers spend their day behind a desk, engaging the virtual world via their computers. Approximately 75% of the global workforce consists of frontline workers whose jobs are done in the real world and involve physical work.

While knowledge workers transitioned easily to work-from-home, frontline workers cannot do their jobs from home. In fields like retail and hospitality, many were furloughed because their businesses were closed, sometimes by mandate. Yet work at factories, farms, repair shops, distribution centers, and other types of physical worksites continued, deemed essential to the economy. The Brookings Institution estimates that between 49 million and 62 million frontline workers in the U.S. alone were asked to continue reporting to work.

“AR enables front-line worker solutions that parallel the well-established digital technologies that came to the rescue for knowledge workers.”

But frontline workers also need knowledge and training in order to do their work. Rather than depend on electronic files and video calls, such knowledge and training has traditionally come in the form of personal experience, paper documentation, training classes, over-the-shoulder mentoring, and face-to-face troubleshooting. None of these modalities have proven ideal even in the best of circumstances. But they have been seriously impaired by having to work remotely, travel bans, and social distancing. In addition, the coronavirus has also exacerbated an already serious problem for many companies beyond health risks – the increasing shortage of trained frontline workers.

Bringing Digital Technology to the Frontline 

Technologies like Microsoft Office allow knowledge workers to capture and share information. Videoconferencing tools like Zoom or WebEx work great for knowledge worker collaboration with computers at each node of the information exchange. Yet we have lacked powerful tools to facilitate collaboration between frontline employees, such as a factory assembly worker, and an expert knowledge worker such as a manufacturing engineer or a service technician. Traditionally, such collaboration has required the engineer to travel to the factory. But suddenly, the crisis made this not possible. How can frontline workers collaborate when one is a veteran technician who needs to explain a laboratory process to a newly-hired technician? Traditionally, this takes place through mentoring or job shadowing, but this is costly and social distancing makes this difficult.

Augmented reality (AR), the next generation of digital technology, makes this productive and efficient by enabling collaboration and knowledge transfer across disparate, frontline workers. This emerging technology supports and significantly enhances the productivity of physical work throughout the economy. We have discussed the concept and impact of AR technologies in our earlier Harvard Business Review article titled, “A Manager’s Guide to Augmented Reality.” In the simplest terms, AR allows actual, relevant digital information to be transmitted and displayed in the context in which it is used in the physical world.

AR has allowed Toyota to save time and money by helping experts eliminate an average of four trips per month to production plants.

AR: A Better Zoom for Frontline Workers 

AR enables frontline worker solutions that parallel the well-established digital technologies used by knowledge workers. However, instead of a Zoom video call, AR technology allows remote experts to see the companion worker’s physical world on video and annotate physical objects with information and instructions during the call. Rather than a flat PDF document or a Web page, AR can map instructional content directly onto the 3D physical environment in which the work takes place. Instead of a YouTube how-to video on a computer screen, AR can capture, in detail, the work of the frontline expert using a wearable device that shows every step. This creates an interactive and step-by-step guide mapped onto the work environment that other workers can follow using a wearable device.

The power of AR is its ability to deliver needed digital content and expert guidance into the context of the physical environment where frontline work actually takes place. This substantially reduces the cognitive distance¹ that inevitably gets in the way when a worker must translate digital information from a screen and decide how to apply it to the real world. Eliminating this gap substantially increases worker productivity while reducing errors.

Expert Collaboration and Remote Support 

AR is a powerful real-time tool for collaboration between a remote expert and a frontline worker who needs input. The frontline worker, using an AR app on a mobile phone, tablet, or AR headset, can request help from an expert at another work site, the office, or even from home. The remote expert can see and digitally annotate what to do on the on-site worker’s physical work environment using a finger pen or a mouse, with these annotations on the physical objects involved moving as the worker’s perspective moves.

AR does for frontline workers what video conferencing does for knowledge workers — allowing expertise to be shared electronically. But unlike video conferencing, which is solely digital and displays information on flat computer screens, AR has one foot in the digital world with the other foot in the physical world, serving as a bridge between them. This application of AR is akin to sportscaster John Madden’s famous “chalk talks”, where he diagrammed NFL football plays on the TV screen to improve viewers’ ability to follow. With AR, a real-time chalk talk by an expert is overlaid onto a worker’s remote work environment.

AR Markups Overlaid on a Video Call. This factory worker is being guided through a machinery troubleshooting process using AR markups by a remote expert overlaid on the worker’s physical objects and environment during a video call.

The use of AR technology for collaboration and remote support of frontline workers has soared in the COVID crisis, enabling experts to be much more productive in helping to debug problems and resolve production issues remotely. Toyota, for example, has used AR video calls to save time and money by allowing experts to eliminate an average of four trips per month to production plants because they can oversee plant modification tasks and monitor plant worker safety remotely. Using an AR-powered tool, non-expert workers such as outside subcontractors are able to tackle complex or unfamiliar challenges with the help of a Toyota expert where and when they need one.

Frontline Worker Training and Learning  

AR is also a breakthrough technology for training frontline workers. AR enables the creation of next-generation instructional content such as training materials and step-by-step or standard operating procedures (SOP) for work. AR can deliver pre-defined training (and avoid the need for real-time collaboration and remote support) that maps the content directly onto the 3D physical environment in which the work will take place. This results in a step-function improvement in comprehension and retention versus traditional training using PDF or Web-based documents that explain procedures via text or 2D pages. AR creates a 3D web where instructional content is delivered via a 3D experience that mirrors an object or a workspace, rather than on the traditional flat web page.

Augmented Reality in the Industrial World. Industrial machinery in the physical world is decorated with an AR digital display showing status and key operating parameters of oil system, gearbox, air system, and main motor.

To access AR work instructions, a frontline worker launches a web browser app on an AR headset, a smartphone, or a tablet. AI-based computer vision technology identifies the real-world machine and workspaces of interest using the AR device’s video camera. AR delivers the relevant content to the frontline worker, superimposed on the appropriate physical object or workspace. The AR device can “see” the real world the way the worker does, while projecting digital information to guide a worker in performing tasks, such as a repair procedure, more efficiently.

A striking example of this capability is Volvo’s use of AR work instructions to improve the productivity and accuracy of its final quality checks on newly manufactured truck engines. This application provides visual guidance to the quality assurance inspector such as where to stand relative to the engine, what part of the engine the inspector should look at first, and so on. The AR app shows the inspector what she or he should expect to see in each view, compared to the actual parts. This is repeated for each step in the quality control process, while recording the whole process to verify it and measure any non-conformance.

AR work instructions have a striking productivity impact. The training time for Volvo QA inspectors has been reduced by 60%, or about three weeks per inspector. Superimposing digital information onto the physical objects in the actual work environment using AR avoids the cognitive distance problem that complicates all training – the need to translate abstract 2D information to the 3D real world.

AR, then, is rapidly emerging as a powerful tool for accelerating how frontline workers learn, and how they can conduct their work at substantially higher productivity levels. Technology analysts at ABI Research estimate that AR and VR-based training will be a $6 billion industry by 2022.

AR-Enabled Welding Training. Seeing welds digitally through an AR-enabled helmet, welding trainees can improve their skill in welding straight and strong joints while eliminating the costly use of consumables such as welding rods.

Capturing Expertise and Optimizing Frontline Human Work

In addition to enabling collaboration, training, and mentoring across distance and time, AR technology solves three other important challenges many companies face with frontline workers. First, how to capture the expertise of retiring workers as they depart. Second, how to take advantage of knowledge of experienced workers without the need to travel. And third, how to enable all workers performing a given process to do so in the same way as the best, most experienced worker.

As best practices evolve, AR can also digitize process improvements and disseminate them to all workers, bringing the whole team along.

Wearable AR headsets are not just tools to deliver pre-defined digital AR content to the physical work environment. Central to AR’s value proposition is the ability of the headset to capture, digitize, and store human work processes and associated expertise in a specific physical environment for training or real-time guidance, delivered later when needed.

Experienced technicians and engineers can use a wearable device to capture a task or perform the SOP in real-time as they perform it. This replaces the need for Frederick Taylor’s mapping of process steps, and the need for stopwatches to measure physical work. AR content can then be turned into a step-by-step guide with instructions for other workers to follow, accessed through the same type of wearable device in a training setting or in the field.

Using AR to capture and replay work processes is a huge step forward from YouTube how-to videos. When AR maps the interactive digital content directly onto the actual work environment, comprehension by the recipient is dramatically greater. Academic studies in recent years have shown with statistically significant results that AR and VR training and teaching methods improve learning comprehension, reduce cognitive load, and improve achievement outcomes.²

AR also allows the digital knowledge of retired experts to be used not just to capture their own work, but to digitally shadow and coach new hires in training. And, the captured work processes of all the workers performing the same function can be analyzed and compared in order to further optimize the process.

AR Playback of a Previously Captured Work Process. A new worker watches visual content captured from an expert showing what operations she should perform at this machine, with relevant tips anchored in the physical environment and each step in the procedure.

A good example of expertise capture and exchange is Smiths Medical, a medical device manufacturer participating in the Ventilator Challenge UK to transfer knowledge on how to manufacture scarce ventilators in response to the COVID crisis. Smiths used AR to capture frontline manufacturing process knowledge and procedures for ventilator production, and deliver those instructions to production workers at GKN, an automotive and aerospace manufacturer. Using AR, Smiths is able to digitally guide workers in other companies, and even in different industries that have no medical device expertise, in how to make the ventilators properly and efficiently. The Ventilator Challenge has been heralded as a success, producing 14,000 ventilators in just three months. There are many such examples of companies that used AR during the COVID emergency and were impressed by its power.

The Emerging Platform for Frontline Work 

AR is a major productivity driver for frontline work, in both good times and bad. According to the Manufacturing Leadership Council’s 2020 Factories of the Future survey, gathered just before the COVID-19 crisis, 48% of manufacturing companies had already adopted AR or planned to deploy it by 2022. During the crisis, thousands of additional companies adopted AR to sustain and optimize operations irrespective of travel restrictions and social distancing.

Capturing and digitizing the expertise of experienced workers is also a foundation for optimizing the overall productivity of frontline workers. Case studies suggest that AR can reduce worker training time by 50%, increase worker productivity on physical work processes by 30-50%, and reduce human errors by 60-90%.

One of the positive outcomes of the coronavirus crisis has been to bring to light how AR technology enables major productivity improvement in distributed and mobile workforces across multiple settings. AR provides frontline workers with learning and productivity that parallels the conventional digital technologies that have come to the rescue for knowledge workers. In a world where a remote workforce will be the new normal, AR is a technology whose time has come.  M

Considerations for Deploying Augmented Reality
While advancements in AR technology and adoption are accelerating quickly, three common challenges could derail an AR initiative as it gets underway. It’s therefore important to keep these in mind from the outset:

• Prioritization: AR is a broadly applicable technology. Dozens of opportunities exist across the value chain, from engineering, to marketing and sales, manufacturing, and field service, which requires disciplined use case prioritization to maximize business value capture. Start by defining measurable challenges that have financial consequences for your organization, documenting the as-is state to quantify the opportunity and later measure ROI, and then prioritize the AR use cases that best achieve that business value.
• Hardware: Different use cases call for different devices, so each AR solution should be designed with a device type in mind to ensure a good user experience. Start by deciding whether head-mounted or hand-held is best suited for your use case. While a standard mobile phone or tablet work well for many use cases today, hands-free operation will require head-mounted devices from Microsoft, RealWear, Vuzix, or others. While that technology is improving quickly, companies should develop an AR strategy that is compatible with many devices, to avoid getting locked-in.
• Content: AR solutions are only as good as the quality of the content and data that companies use. Getting access to the data or creating new content, and combining it in a meaningful way, can be challenging. Leading companies today reuse 3D content from CAD systems and leverage data from IoT systems to accelerate this process. Companies would also benefit from a digital thread— a single source of data truth by real-time synchronization of systems — to link interrelated data across all the processes, products, and people in the value chain.

FOOTNOTES
1 Cognitive distance is the gap between the form in which information is presented (e.g., a computer screen), and the context in which it is applied (e.g., the real world). Cognitive distance increases the “cognitive load,” or the mental capacity required to interpret the information, hold it in the working memory and translate it into implications in the physical environment.

2 For examples of academic studies examining the comprehension benefits of AR on teaching and training, see: Gonzales, Alexis A., et al, “Augmented reality-based learning for the comprehension of cardiac physiology in undergraduate biomedical students,” Advances in Physiology Education, June 22, 2020, or Küçük S, Kapakin S, and Göktaş Y, “Learning anatomy via mobile augmented reality: effects on achievement and cognitive load,” Anatomical Sciences Education 9: 411–421, 2016.