Researchers at Carnegie Mellon University’s (CMU) Center for Iron and Steelmaking Research are bringing computer-vision and machine-learning techniques to the study of inclusions, hoping to increase the efficiency of inclusion analysis and gain new insights.
What are inclusions?
Inclusions – nonmetallic compounds and precipitates that form in steel and alloys during processing – are by-products of steelmaking, arising from different chemistries and processes. Endogenous inclusions occur within the liquid steel, precipitating out during cooling and solidification. They are typically more uniformly distributed than exogenous inclusions, which are entrapments of materials from refractory interfaces, slag or other materials in contact with the melt. Inclusions can vary widely in size and composition, giving rise to a corresponding wide range of effects and mandating sophisticated analytical equipment for characterization.
“One of the big focus points in steelmaking research has been on these nonmetallic inclusion particles. They’re these little particles floating around in the steel that are always there,” said Bryan Webler, a professor of Materials Science & Engineering at CMU. “They affect the final performance of the steel, which is why we care about them so much.”
Solid-phase inclusions can clump together to clog nozzles and other flow control systems that mediate the flow of liquid steel, posing a threat to process operations. Some inclusion chemistries reduce ductility, resistance to fatigue or overall toughness in steels. The absence of inclusions poses issues as well because “clean steels” can be harder to machine, decreasing the lifespan of cutting tools and requiring higher power consumption for machining. Understanding their nature is of critical importance in steelmaking operations, and “inclusion engineering” is both an ongoing area of academic research and an operational focus of some steel plants.
Fig. 2. SEM images of different, representative nonmetallic inclusion populations showing different chemistries analyzed in Webler and Holm’s computer-vision work
Inclusion Analysis: Current Practices
Inclusion analysis, even in clean-steel operations, is a routine part of process and quality control, typically using scanning electron microscopy (SEM) coupled with energy-dispersive spectroscopy (SEM/EDS). SEM uses backscattered electrons (BSE), which provide quantitative number and size distributions as well as shape and location information. Spectra from EDS identify the chemical composition of inclusions. These batch methods often require a trained metallurgist to operate equipment and analyze the output data and are thus both labor- and time-intensive.
Consider calcium treatments that are used in clean-steel operations to reduce alumina formation to illustrate the feedback cycle between inclusion analysis and process control. Calcium is added to the melt in order to form calcium-aluminates, which convert alumina inclusions to a liquid phase. Liquid inclusions reduce the risk for clogging nozzles and other flow-control hardware that solid inclusions present. However, the addition of excess calcium can lead to solid calcium-sulfide inclusions.
Fast inclusion analysis would help identify any undesirable inclusions, informing whether the correct process-control decisions were made. Are there CaS inclusions? Turn down the calcium. Too much Al2O3? Add more calcium. The speed at which inclusion analysis can inform the right level of calcium addition, in this case, impacts the bottom line of a clean-steel operation both in material costs and process performance.
Fig. 3. Schematic of convolutional neural-net processing of nonmetallic inclusion images
Computer Vision and Machine-Learning Promise
Researchers at CMU are attempting to improve inclusion analysis through the use of computer vision (CV), which relies on advanced image processing and machine-learning (ML) classification.
Materials science is rife with images. This fact makes it an ideal target for applying CV. Liz Holm, also a professor of Materials Science & Engineering at CMU, has previously used CV to classify carbon nanotubes, predict stress hotspots and characterize additive manufacturing (AM) powder feedstocks.
Having observed CV wending its way into adjacent fields, Webler realized that inclusion analysis may lend itself toward CV, eventually leading to a collaboration with Holm. With a legacy both in steelmaking research and machine learning, CMU is an unsurprising place for these two fields to converge.
Giving computers “vision” first requires numerical representation of an image, and Webler and Holm employ a popular CV approach to do so: convolutional neural networks (CNNs). CNNs create multiscale representations of images by passing them through various “filters.” These filters are learned from the images themselves and then classified. Classification, too, is learned from the images and does not require any pre-selected considerations, though it does require a training dataset to establish ground truths.
Fig. 4. Representative SEM images of features classified as (a) inclusion or (b) “not inclusion”
In the case of character recognition of texts, a training dataset tells the computer what a “letter A” looks like across many printed sources. For steel inclusions, training data details whether or not a feature is an inclusion and what its chemical composition is, providing a baseline against which the computer makes sense of new data going forward.
How might CV improve the current state of inclusion analysis? First, there are always non-inclusions (e.g., dust, holes, scratches) in images of polished steel that can be detected as inclusions (false positives). Even when classified correctly, analyzing these non-inclusions is wasted time for the EDS system, and filtering them out can improve efficiency. Further, were CV techniques able to determine composition information from SEM images alone, EDS could be eliminated entirely, reducing both equipment and labor costs.
Even more appealing to Webler, though, is whether CV could elucidate information beyond the scope of current inclusion analysis. In other fields, CV has been able to produce insights invisible to human interpretation. One example of this is a CV system that could “see” traits such as age, gender and even cardiological risk factors in retinal images. No ophthalmologist had previously uncovered any of those traits by looking at human retina. By turning CV loose on inclusion images, the team hopes that unknown insights may be generated, learning through the intricate differences in the images what might be hidden within inclusion populations.
Fig. 5. Confusion matrix of CV model predictions from the test dataset
Applying CV to Inclusions Shows Initial Promise
While the research is still in the beginning phases, initial results from Webler’s and Holm’s research shows promise that CV can be useful for analyzing SEM inclusion images.
The team, presenting its initial findings at the 2019 Association for Iron & Steel Technology conference, was able to determine with 98% accuracy whether a feature on an SEM image was an inclusion or not. A previous study that applied CNNs to inclusion SEMs was only able to achieve 72% accuracy. Integrating CV into the current systems to filter out non-inclusions already shows promise for saving time in EDS scans. They were able to differentiate between inclusions and non-inclusions in 70 ms. Making that same determination for a feature using EDS takes over 14 times as long, at roughly 1,000 ms.
Interestingly, Holm said it was not clear to her by looking at the SEM images why their CV tool classified features in one group or the other. Yet, that the CV analysis did so with a high degree of accuracy holds promise that more information is likely still lurking in the SEM images that CV may be able to tease out, though this was limited in the current dataset due to image-to-image variability. The team hopes that refining setup parameters of the SEM may address this issue. The next step, they say, is to “classify inclusions by chemical composition based only on BSE images.”
Though he acknowledges that the work is still in its early phases, Webler is optimistic about his team’s approach and how it could eventually have a positive impact on the industry.
“We hear lots and lots about big data, industry 4.0, all of these things, but it’s still opaque in a lot of ways,” Webler said. “But this is one example, at least, where I can see how machine learning could be useful when applied in this specific way.”
