Cleaning Up The Systems Mess: A Phased Approach To Implementing Real Automation And EBRS

The pharmaceutical industry uses the most sophisticated microbiological and genomic technology, and we experience waves of advancements in therapies, all driven by our computing and communications technology. In addition, we as a culture are nearly physically glued to our smartphones/devices that manage our business and personal lives. We bitterly complain when we enter a facility with “poor” Wi-Fi reception and downloads take 10 seconds instead of happening in the customary flash.

So, when we go to work (laptops in hand) in pharmaceutical production, why are we drowning in 100-page paper batch records, books of SOPs on the plant floor, and paper documents being handled by operators and operations support personnel throughout the buildings? Why do we cheer and celebrate when we can get a paper batch record approved and out the door in 40 days rather than the average of 65 days? Why do we track the number of deviations (mostly clerical/document oriented) on the batch records as a measure of success and efficiency? How will this paper-laden environment work as personalized therapies/precision medicine become a bigger part of our industry?

The following are some hard truths about the pharmaceutical industry that need to be addressed if we are to continue to cost-effectively and compliantly provide the life-saving drugs that patients require:

• The efficiency of our plants, which are manufacturing time-sensitive critical care therapies, is abysmal.
• The time to market for each actual production batch is glacial.
• We accept it all and blame the FDA and regulations, the operators and their skills, and the cost.
• The FDA paved the way for electronic batch record systems (EBRSs) and manufacturing execution systems (MESs) with 21 CFR Part 11 in 1997 – 23 years ago, and while many have implemented the modern systems, others are still stuck in the 1980s.
• Outside of the workplace, the operators work with the same technology as everyone else, and they generally object to working primitively, as we have forced them to.

The cost of implementing an EBRS must be weighed against the incredible efficiency and capacity gains it enables. This article will discuss the cash flow implications and the sequencing of converting a plant from manual, paper-based systems to an automated data capture system. We believe EBRS and MES are within the grasp of every plant.

What Are MES And EBRS?

An MES focuses on providing an automated and electronic work instruction system to facilitate the batch execution, collect operator input, and assure a compliant sequence of operation.

An EBRS is an overarching system that provides the integration of the MES data (batch record) with the raw material data, equipment status, product quality data, and inspection data. This system then reviews all the basic data for correctness and completeness. Being electronic, it facilitates the final routing and approval system, which should expedite release.

There are major system vendors that provide both MESs and EBRSs, along with process control systems. These suppliers can be broken down into essentially the following categories:

• Distributed control systems (DCS) — e.g., Emerson and Siemens
• Programmable logic controller (PLC) systems — e.g., Rockwell
• Software systems — e.g., Werum
• Process database systems — e.g., OSI Pi and Wonderware

Each of these suppliers can provide part of the complete solution or most of the solution, but the key is that you can start incrementally and immediately.  

Where Are We Now?

In the last five years, we have been in five major pharmaceutical plants and about a dozen smaller bio/pharma plants that had paper batch records and hand-recorded data for process variables (including each step, time, date, and initials). Access to SOPs was not in the production suites but in paper binders or on the computer screens at the operations supervisor’s desk. Arguments would erupt over various operators’ recall of the various SOPs.

As an example, these facilities typically have paper batch records that range in size from 40 pages to 200 pages. These facilities typically take two to four weeks for the operations supervisor to review and correct the records. Then they have a batch record review department that spends another two to three weeks reviewing each record for accuracy and integrating all the material records, environmental monitoring (EM), and lab results. From there, the batch record is moved to the QA organization for reconciliation of deviations, accuracy checks, and final approval.

We as an industry are adrift in a sea of paper and we imagine our businesses are not functional without the wasted mounds of paper. We have to wake up to the fact this is not 1986; it is 2020, and we are hiding behind our proprietary technology.

Where Should We Be?

In all pharmaceutical operations, we should be examining the following key areas that need to be in place. As you review this list, if there are areas of your process that are not automated, they need to be implemented in the following priority:

• Data collection (temperature, pressure, level, flow, pH, etc.) from a control system and stored in a database, to relieve the operator’s activities and reduce potential mistakes
• SOPs available to the operator electronically in the production suite, where they need it
• Environmental data (temperature, relative humidity, differential pressure) collected and stored in a database automatically and without human interaction
• Weigh/dispense information captured and stored with the operator(s) ID, weight, scale ID, and material pedigree/history
• Equipment tracking (ID, clean/dirty status, preventive maintenance status) through bar code or RFI systems
• QC lab database that integrates to a product data base
• Operation sequencing
• Data integration of the various batch record subsections into one cohesive relational database.

With the above as the main priorities from a system viewpoint, we must make sure that what we are implementing achieves success and a payback. Efficiency in our operation will either increase capacity or reduce costs, so streamlining the manufacturing core is the place to start. 

How Can We Make An Impact?

The most critical concept is to provide primary and focused support to the operators by providing them with all the tools they need to make their jobs easier. This can be done in a stepwise fashion by:

• Asking the operators what they need, what takes the most time, and where they make the most mistakes;
• Identifying every task the operator does that causes the process or execution of the process to stop, especially ones where information is written or exchanged via paper;  
• Simultaneously, looking at errors observed throughout the batch record; and
• Analyzing all the physical transactions across the execution of a batch – samples, obtaining labels, material receipts, etc.

So much of what we are looking at is directed at data and the effort of gathering it. One of the most important benefits of EBRS/MES/automation is the reduction and elimination of paper transactions throughout the classified space and across the classified/non-classified spaces. Very few of us discuss how handling paper, including the hurried transit of paper and the exchange of pages of documents, spreads bioburden, particulates, cross contaminants, and product. Remember, all the paper in manufacturing ultimately passes through the hands of the operators, directly on the process or line.

How Do We Start?

Some basics need to get underway immediately to use as leverage in the long term:

• We need a high-capacity network of WAPs (wireless access points) to allow multiple systems to connect wirelessly and support wireless IO. This starts with a fiber optic backbone and strategic (or suite by suite) location of the WAP’s devices. This can be easily retrofitted, even in older facilities. The WAPs must be planned to cover all operating areas, storage areas, and potential IO locations.
   
• Select a plant standard for PLC or DCS applications and make sure all systems that are brought in-house only use that system. This is necessary for long-term compatibility. From this standard, migrate your disparate systems and legacy “islands” based upon future purchases (lines, autoclaves, parts washers, etc.) and active migration of legacy systems as their life cycle nears the end.
   
• Select a standard database repository using a Part 11 compliant process database (e.g., OSI Pi, Wonderware, etc.). This often is part of the control system standard, but it must support detailed and independent investigations as well as the core data repository for the batch record.
   
• An approved document repository needs to be provided in a secure system that permits quick access by operators and excludes all other modes (edit/review/etc.).

Implementation Steps

From the “impact” study done as described above, the key areas should be obvious, with the most benefit derived from assisting the operators by offloading their paperwork and their paper-chasing. There are many applications that can be done, one at a time over an implementation period, given the starting points mentioned above. Here is a series of win-win applications that will start the legacy facility (or any, for that matter) down a successful path:

• Operator-focused assistance and integration usually starts with data capture and archiving from the process, data acquisition and concentration from a skid, and data acquisition and concentration from a filling line system. Skid makers and line integrators generally provide an open digital architecture, and here is where the connectivity will pay off. Process analysis is best served when all the data from the process, filling line, and environmental equipment (e.g., HVAC) can be superimposed for analysis and the batch record compilation and archiving.

The skid, HVAC, and filling line system data acquisition is primarily an IT function, taking the data from the filling line and routing it to your database of choice (OSI Pi, Wonderware, etc.), and it can be done with relatively little process interruption. The process data acquisition is yet another level of involvement and presents a level of complication within an existing building, especially one without a low-voltage or instrument chase. With process pressures, temperatures, levels, flows, and pH, among other measurements, the primary instruments will need to be changed from a local analog device to either an electronic analog or digital device (4-10 mA or 1-5 v output) that is wired to a marshalling panel and routed back to the PLC or DCS via Ethernet (assuming remote IO capability). This typically presents logistic, facility, and cost burdens; however, today’s instruments can be converted to digital Wi-Fi compatibility and thus use the WAP/fiber backbone installed in the early preparation stages. This can make a conversion from a legacy facility to a modern facility relatively easy.

• One of an operator’s most frustrating experiences is the lack of access to a paper SOP, especially the paper SOP that contains the critical sign-off sheets that must be acquired, signed, and attached (physically) to the paper batch record.  The solution is to provide the SOPs via a cleanroom tablet. Tablets today are easy to use, have touch screens, and are Wi-Fi enabled. This will provide the operator with a portable tool they are familiar with and that is GMP cleanable and useful directly in the suite. If they need a form from the SOP, the tablet can call up the form and the operator can go into the .pdf edit mode, fill it out, and save it under the batch number. If the form needs to take the place of a label, it can be printed out via a Wi-Fi enabled printer in a secure enclosure in the process suite and used directly. Providing the SOPs on a tablet to assist the operators is a major step in improving operational excellence and compliance.
   
• Weigh/dispense and line-level material transactions are another time-consuming activity that can be easily automated with PC-based systems that are uploaded to the main database via the WAPs. The system can use Wi-Fi enabled bar code devices (Zebra, Symbol Tech, etc.) to scan, calculate, and report the results via Wi-Fi to the database. In this application, the operator, scale, and material each have a bar code ID for tracking.
   
• Equipment tracking is another problem bound by paper tags. The same bar code technology and Wi-Fi enabled capability used in weigh/dispense can be used to track equipment. All equipment (tanks, skids, filter housings, etc.) gets a polyester bar code label fixed on the unit. The operator then can scan the bar code to update the use, process, and return for cleaning for each piece of equipment. This data can also be added to the master database. The use of wireless IO allows you to move the equipment and skid anywhere in the plant to execute a process, clean, or activate/remove from service.
   
• Material tracking can be added in sequence after the equipment by having an operational/warehouse/transactional universal bar code for each entity touching that material. In most facilities, an enterprise resource planning (ERP) system does have material control and, if that is the case, the ERP system has a “gateway” from its relational database to external systems – another non-invasive application with benefit to the operators.
   
• At the end of a multiyear effort implementing the above aspects, the master data base contains all the production-related data and transactions. At each step, an incremental investment is made, and the facility moves closer to an EBRS. With integration of the vendor database, QC lab data, and a QA approval, the plant will be on the threshold of an electronic batch record system.

Maximizing ROI And Phasing In A Solution

“Where is the highest return?” is always the first question most folks ask when discussing this topic. The key is to focus on the impact on the operators. They will show you how to improve the production process and how to minimize errors. They have the most unique insight. You also cannot avoid building the Wi-Fi network first and selecting the basic building blocks of a database and a control platform.

As discussed above, there are many steps in the process, and the main constraints are those created by sorting out the key areas above, removing the systems that you currently have in place, and beginning installation of the infrastructure. Whether transitioning an existing facility or coping with the need to deliver lifesaving medicines as soon as possible in a new facility, we can build out systems in phases with some planning as outlined below:

1. Initial
   1. Tablet-based operator recording: no intelligence, just reduce the paper in the cleanroom
   2. Embrace packaged automation from all skid suppliers
   3. Ethernet or Wi-Fi connectivity for all equipment
   4. Focus on wireless everywhere
   5. Open platform communications (OPC) server to collect disparate data from multiple future systems
2. Follow on
   1. Encapsulated bar code printer in each suite
   2. Batch record on tablet leading to an EBR
   3. Real-time release via process analytics
3. Long-term vision
   1. Digital twin for assets and preventive maintenance
   2. Augmented reality for operator training
   3. Predictive maintenance via real-time vibration monitoring of drives
   4. MES
   5. Automate dirty, dangerous, or repetitive tasks 

Conclusion

The efforts outlined above provide the basics for an EBRS system and require further integration and record production. However, if implemented correctly, they can achieve a high level of return on investment by:

• reducing batch review time dramatically
• reducing batch cycle time and increasing capacity
• improving compliance
• improving capacity and reducing costs by avoiding shutdowns and accidents
• improving “right first time”
• facilitating a lean execution

Implementing an MES can improve upon these successes to further control execution and compliance. While complete implementations of EBRSs and MESs are not as common in the pharmaceutical industry as in others, components can be implemented incrementally if a plan is put in place and the primary source of direction and implementation is the operators that work the process and line operations.

About The Authors:

Herman Bozenhardt has 44 years of experience in pharmaceutical, biotechnology, and medical device manufacturing, engineering, and compliance. He is a recognized expert in the area of aseptic filling facilities and systems and has extensive experience in the manufacture of therapeutic biologicals and vaccines. His current consulting work focuses on the areas of aseptic systems, biological manufacturing, and automation/computer systems. He has a B.S. in chemical engineering and an M.S. in system engineering, both from the Polytechnic Institute of Brooklyn.

Erich Bozenhardt, PE, is the process manager for IPS-Integrated Project Services’ process group in Raleigh, NC. He has 14 years of experience in the biotechnology and aseptic processing business and has led several biological manufacturing projects, including cell therapies, mammalian cell culture, and novel delivery systems. He has a B.S. in chemical engineering and an MBA, both from the University of Delaware.