Citation: Wardell S, Thacker C, “Automating Previously Impossible Manual Inspections Using Machine Learning & Concurrent Validation”. ONdrugDelivery Magazine, Issue 99 (Aug 2019), pp 40-44.
Steve Wardell and Catherine Thacker explore how machine learning technology is enabling the automation of challenging, or previously impossible, manual inspections.
Every manufacturing process has one – the difficult manual visual inspection process that seemingly cannot be replaced with an automated inspection process. The two major hurdles are finding a technical solution that duplicates what human inspectors accomplish naturally – and proving the equivalency of machine-based inspection with human inspection.
“ATS Automation and its vision group have continued to develop inspection capabilities leveraging machine learning techniques and are now enabling the move away from manual processing to automated processing.”
Advancements in the field of machine learning now make the automation of many of these challenging inspections possible. ATS Automation and its vision group have continued to develop inspection capabilities leveraging machine learning techniques and are now enabling the move away from manual processing toward automated processing.
THE CRITICALITY AND SENSITIVITY OF RELIABLE VISUAL INSPECTION
Visual inspection plays a critical role in assuring the quality and repeatability of a manufactured product. Whether we inspect to make sure the product is within specification or we inspect to verify compliance with industry standards, these inspections are a necessary part of any manufacturing process (see Table 1).
The nature, complexity and criticality of visual inspections influence the inspection approach, namely manual, automated or a combination. The scale can range anywhere from periodic sampling, or auditing, to 100%, or complete, inspection of all parts produced. Where there are multiple process steps, there can be in-process visual inspections to flag defects early and avoid subsequent value-add processing. The reliance on operators for these inspections comes with a high price tag.
| Inspection Category | Examples |
| Finished product quality checks | Cosmetic defects, colour, fill level, seal quality (contamination, homogeneity), particulate |
| In-process quality checks | Presence/absence, orientation or position, colour, glue pattern |
| Counting and verification of counts | Reconciliation of components, e.g. labels, plastic parts, verification of reject action success |
| Regulatory requirements and compliance | Batch number and expiration dating information, serialisation |
| Customer assurance | Final functional / performance checks |
| Safety requirements | Package integrity, leak detection |
| Process improvement and diagnostics |
Machine monitoring, operator assistance and supervision |
| Metrology and measurements | Insertion depth, placement position accuracy, volume dispense, seal width |
Table 1: Typical inspection categories in manufacturing.
But how do we move toward automated inspection? In applications where a human inspector examines a component with their eyes and checks for defects, an analogous automated system has to duplicate not only the gathering of the image of the part butalso the analysis of that image to decide whether or not there are defects present. In some situations, this can be very difficult due to the nature of the defects and/or the parts themselves (Figure 1).
Figure 1: Challenging inspection.
A good example is the manufacture of certain types of pharmacological products. Within the pharma industry, the assurance that the sellable product (medicines, tablets, vaccines, etc.) is without defects is of paramount importance. Therefore, manufacturers put a strong emphasis on the deployment of inspection systems throughout their manufacturing process. Historically, many of these inspections have been manual because the automated solutions’ reliability was unable to match that of the operators trained for the job – or because a strong validation case could not be made. In this industry, all automated systems need to demonstrate that they are at least as capable as a standard trained operator under well-defined conditions.
Human inspectors have the innate ability to manipulate a part, view a scene, process the information and arrive quickly at a conclusion, e.g. pass or fail. Even when the scene is chaotic, it is relatively easy to train people to pick out the defects that lead to a good or bad determination. For automated systems, this type of inherent analytical capability does not come naturally. Machine developers need to itemise the human inspection thought process and then mimic it through programming code and part presentation. To do that, there must be a sufficiently detailed and nuanced description of acceptable and unacceptable product. This in itself is a challenge. The creation of an unacceptable product is the exception rather than the norm, so the availability of samples that represent every possible defect or variant of a defect is very low.
ATS has been working on a solution for these difficult applications – a solution that leverages the skill and experience of the trained operator with current vision technology and machine learning.
The ATS M+ solution is a combination of technology and phased implementation. It begins with automated image gathering and manual image classification and then uses this information to eventually teach a system to think like a human inspector. Over time, there is sufficient comparative data collected to validate the performance of the automated inspection solution without ever jeopardising the confidence in the quality of the product released to market.
In order to illustrate how the ATS M+ solution is applied, we will provide an example. For confidentiality purposes, the following example is from a fictional customer, ACME Syringes. ACME is investigating automating their manual inspection process.
ACME SYRINGES
Hypothetical company ACME Syringes produces disposable syringes for hospital, office, laboratory and home use. It has a reputation for a high-quality single-use syringe product family. Key to its quality success are its master-certified inspectors who visually examine every syringe for needle insertion depth, bond inspection, needle tip inspection (shape and orientation) and barrel text (legibility and position), as well as other quality attributes.
“In this industry, all automated systems need to demonstrate that they are at least as capable as a standard trained operator under well-defined conditions.”
As word of the superior quality and performance gets out, ACME experiences a growing demand for its syringes. This puts a strain on the team of master-certified inspectors, the human resources recruiters, ACME’s training department and production’s operating budget. ACME’s quality group is becoming concerned about the risk of releasing and shipping sub-standard product. Can the reliance on the master-certified inspectors be reduced? Can the inspection process be automated?
A few years ago, and in anticipation of just such a situation, ACME’s general manager investigated the feasibility of automating the inspection processes. He talked with several automation providers and imaging experts, only to find that no one would commit to attaining the quality requirements the master-certified inspectors were delivering. They cited as their reason the difficulty of mimicking the decision-making capabilities of the inspectors with the imaging tools available at the time.
Now in a challenging situation, ACME asks the automation experts to revisit the feasibility of automating the inspection processes. The response? Machine learning capabilities in the field of machine vision solutions have taken a major leap forward and now offer some possibilities for automating the inspection process. However, they need 5k–10k images of all different defect types, along with the master-certified inspectors’ grading for each. Once in receipt of the images, the automation experts could go away and “teach” an automated solution to do the same inspections and make the same decisions.
This is impossible for ACME to accommodate for two reasons:
- The current quality level is quite high so it will take months to produce the requested number of images of a defective product.
- There is no image capture technology on the production lines so there is no image library. The master-certified inspectors look at each syringe directly to make the pass/fail decision and they do not record the defect type when it does occur.
With no image library available, many of the solution providers decline the opportunity. Those that could provide the image capture capability still need ACME to classify the images before they re-engage. It seems that ACME is no closer to an automated inspection solution.
Figure 2: Phased approach.
ATS M+ SOLUTION AND IMPLEMENTATION
ACME contacts ATS and ATS immediately identifies its M+ solution as the perfect fit for ACME’s needs. No images or samples are necessary as they will be acquired during the phased implementation of the ATS M+ solution. ACME and ATS set to work (Figure 2).
1. Automated image capture
To begin the project and build understanding, ATS works with ACME to determine what it is that the master-certified inspectors look for when manually examining the syringes. In addition, ATS asks about inspection aids like lighting and whether these help to highlight certain syringe features or defect types that matter most when determining quality. Once the inspection needs are clearly defined, ATS integrates the M+ automated camera system with controlled lighting and syringe presentation and orientation capability to ensure the images gathered contain the information needed to show the syringe quality.
2. Manual image classification
ACME is now collecting real images and presenting them in real time to the master-certified inspectors via display screen stations located remotely. The remote location provides a couple of advantages – increased availability of premium floor space and decreased distraction from the production area. The inspectors continue to review each syringe image to determine the pass/fail status. The inspectors’ disposition decisions are entered through the station and automatically attached to the images, recorded and passed back to the production line so that the actual syringes are appropriately dispositioned – either a reject bin or packaging and labelling.
In addition, and in anticipation of the next phase, the inspectors classify the images of defective product by the various defect types. The human interpretation of each image is still required in order to build up a library of images that comprise a specific defect type or types.
3. Machine learning from classified images
At this point in the implementation, we have a truly hybrid inspection system in full operation. Automatic image capture combines with human assessment for defect detection and classification. However, we have yet to meet the ultimate goal of replacing the human inspector with a fully automated system.
The key to reaching the final goal is the employment of the latest advancements in the field of machine learning. We will not go into detail here about artificial intelligence and the algorithms of the machine learning discipline. However, it is important to understand the essence – to “teach” a machine to make the same decisions a human does through the processing of sufficient amounts of pre-classified data such that the machine can make the same classification decisions with the same accuracy as the human on any new unclassified data that it is presented with. In other words, if we can develop the right model using enough quality data, we can develop an algorithm to do what the human does now.
For ACME, ATS’ M+ technology is building an image library classified by the master-certified inspectors. ATS is concurrently developing a reliable machine learning algorithm – the other part of the ATS M+ solution. With the image library, it is not necessary to wait for 5k–10k reject images to be processed. ATS creates the model in the background using the data as it becomes available and recycling the library images. With unique approaches and proper algorithm development, it is possible to produce a reliable model with less data.
4. Parallel validation
The challenge now for both ACME and ATS is to prove that the automated inspection system with the machine learning algorithm is as robust and reliable as the current manual inspection. If ACME sells a defective syringe, the end customer is likely to complain and, worst case, a regulatory agency may order a product recall – a very expensive proposition.
The concept of parallel validation involves concurrently running the existing system and the desired future-state system and comparing the results – in this example, the existing manual inspection by the master-certified inspector and the proposed ATS M+ solution. At this point in the phased approach, the inspectors are still classifying all the images collected on the line and dispositioning the syringes accordingly – ACME has not relieved them of that duty, and will not be doing so until the verification that the algorithm is just as good as human judgement is complete (Figure 3).
Figure 3: Phased approach and parallel validation in syringe manufacturing.
Once the capability of the machine learning model is established through test cases and verification processes, it is set into operation in parallel with the inspection team. Now, both the inspectors and the ATS M+ solution process every syringe image. The inspectors’ and the algorithm’s determinations are compared, although the inspectors make the final disposition decision. Further, ACME chooses to pass the same images of defective parts past different inspectors to determine if all inspectors will make the same disposition decision and whether refinement of the definition or alterations to the machine learning model are required. ACME retains full decision-making capability based on these results as to when and if the transition is made to fully automatic inspection. ACME can test the model on the images captured since the beginning of the project. It can continue to test the model with concurrent validation until it is satisfied with the performance. This data-based approach to validating the equivalency and capability of the automated solution provides ACME with the confidence to support a switch to machine inspection.
5. Full automation
After only three months, ACME has gained confidence in the ATS M+ solution’s capabilities and reliability and gathered sufficient validation evidence to complete the switch to fully automated inspection. Many of the master-certified inspectors are relieved of the duties of manual syringe inspection and reassigned to other quality-related roles. Some inspection processes were transitioned more quickly than others as the validation evidence was gathered sooner. The machine learning model development and concurrent validation required less data and yet achieved higher accuracy than was originally estimated by some of the other solution providers. Without the ATS M+ solution and the phased implementation, this would not have been attempted, let alone accomplished.
| Phase | Description of Enhancements | Benefit |
| Automated Image Capture | Automatic acquisition and archiving of all images | Robust image archive – remove inspection subjectivity, helps in inspector training |
| Images archived with associated product data | Robust image archive – increases quality review opportunities and reduces recall exposure | |
| Image Classification | Rapid disposition of obvious defects | Reduce inspection time; reduce inspector burden; increased throughput |
| System provides recommended disposition with rationale to inspector |
||
| Parallel Validation | Queue of images for inspection changes to dynamically balance workload |
Reduce bottlenecks; increased throughput |
| Inspection performance (timing and accuracy) is tracked and inspectors who require training are identified | Increase inspection accuracy and throughput; increased confidence in fully automated solution |
|
| Machine Learning | Self-learning and self-improving algorithms for continual refinement |
Increase robustness of inspection, increased likelihood of success for highly complex inspections |
| Fully Automated | Complete deployment of automated inspection | Increase inspection consistency, accuracy and throughput; reduced reliance on labour |
| Reduce production area footprint |
Table 2: ATS M+ solution and phased implementation benefits.
CONCLUSION
In summary, the ATS M+ solution facilitates the timely transformation from 100% manual-based visual inspections to 100% automation-based visual inspections. The automated capturing of inspection images, combined with the classification of these same images by trained operators, sets the stage for developing machine learning algorithms. Operating the models in parallel with the manual inspection process builds confidence and data-based evidence of the equivalency and reliability of the automated process. The disciplined execution of a phased implementation plan provides opportunities to assess results and adjust course as necessary. Thus, the risk associated with transitioning difficult inspection tasks to automated inspection is mitigated and benefits can be realised (Table 2).
Machine learning is a powerful new tool in the field of machine vision processing. Inspections that were previously thought to be extremely difficult to automate and validate may now be automation candidates with the proper knowledge, tools and implementation plan. The ATS M+ solution is such a tool. A phased implementation, including concurrent validation, improves