For a video version of this blog post, check out Praneet's presentation from Perceive 2021 on the same topic.
Imagine you’ve decided to go to the grocery store on the way home from work. Since this wasn’t a planned shopping trip, you didn’t have the chance to check the fridge before heading out, and now you find yourself standing in the dairy aisle trying to remember if you have enough eggs to last until Thursday.
What might help you out in this situation? Let’s suppose you call your neighbor and somehow get them to snap a photo of the inside of your fridge. This changes everything; now, you have an idea of what your current inventory is. You happily begin shopping until you realize that while the photo of your fridge shows that you have milk, the photo has no information about the age or condition of the milk. The image alone isn’t enough.
This is where predictive maintenance – the use of computer vision, data science, and predictive analytics – would be useful. It combines computer vision with analytics to make judgment calls on when something needs to be replaced or has a poor current condition. While our fridge example would save some time and annoyance, when applied to serious applications such as aircraft engines, the technology can generate huge savings by estimating when a component needs corrective maintenance before it fails. The equipment’s lifespan can be fully optimized by scheduling maintenance at the most convenient and cost-efficient moment.
Predictive Maintenance starts with understanding your equipment. Equipment varies widely from pipes to turbines, and all of it is out in the real world under environmental stress, and there are consequences to not knowing it. Equipment of all types can suffer from degradation such as corrosion and cracking, and complex machinery is made up of many parts; it's tough managing and tracking it all.
Human maintainers spend considerable time and money. Repairs are repetitive and reactive, and human maintainers can make mistakes, incurring high costs. Because your equipment is essential, the time spent not being used is a compounding cost. Safety is also crucial; failure can be disastrous!
Aircraft: Aircraft engines are a complex set of parts that go through extreme environmental conditions, and the aircraft’s structure like the wings and horizontal and vertical stabilizers.
Turbines: Wind turbines are huge, expensive, crucial infrastructure that is hard to replace due to massive transportation costs.
Pipelines: Vast infrastructure crossing massive areas of land that can cause major damage if not maintained properly through leakage or oil spills. They’re also crucial infrastructure.
Predictive maintenance is most effectively used with crucial infrastructure and equipment, where a high cost is associated with failure and human maintainers who can’t just stare at something time and time again. This is where we can leverage automation and precision with finely-tuned computer vision models that outperform humans and improve as more data comes in.
We can also create a shared data environment to maintain and manage equipment data and train models. A big problem with getting feedback from equipment is that all the data is often in siloed datasets. Once you introduce a visual element, you can use these models to synthesize this data and share them in a data lake. We can also deploy these models in multiple environments to enhance detection and collaboration ability. We can use computer vision models that are interoperable and function with various sensory domains.
Downtime: reduce downtime of equipment. As mentioned previously, equipment that just sits around broken costs money due to reduced output.
Cost: manage degradation and reduce the cost of repair or replacement. By understanding your equipment better and sooner, you can reduce this cost.
Risk: maintain safe equipment for operators and consumers. When you find those cracks or corrosion, you reduce the risk for your people.
Let’s discuss a real-world use case using aircraft. These machines are very complex in every way, from the engines to the superstructure, which comprises many moving parts. They’re also frequently at high altitudes, where they experience extreme environmental conditions. One of the newest use case requests from a major airline is for automated detection or lightning strike damage.
With AI, highly accurate, consistent models and workflows rapidly accelerate detection of defects, and relieved human operators can quickly verify and improve the detection without the burden of manually inspecting everything. As the data grows, you can use the model’s history to examine how something is changing over time. The precision is there, and as the data is structured, we can compare it temporarily.
The benefits of this technology are numerous. You can greatly reduce depot time for critical aircraft, where some planes can take two years to clear after repairs. Maintenance costs and human labor costs are reduced. Supply chain issues, which are extremely prevalent at the present time, can be avoided, as aircraft parts present a complex and expensive demand for replacement parts.
Tracking potential issues and repairs as they evolve is another enormous benefit. You can create accurate predictions and projections of essential equipment, reduce risk to flight operators and maintainers, and prevent catastrophic damage or loss of life.
Actual people in the field can use a smartphone app to ingest this imagery data. They take pictures of panels, fastener areas, and engines to capture input data. Clarifai has a flexible gRPC API that can ingest this high-resolution imagery and send it to the Clarifai Platform for indexing. Within the Clarifai platform, a structured database (Postgres) can create time-series data of issues using appropriate codes for the use case. This creates a catalog that shows how fast each fastener is corroding within each panel. Finally, an RPA application (Robotics Process Automation) service can set thresholds and send alerts to notify maintainers of potential issues.
With Clarifai being a critical component, this end-to-end example adds structure to this unstructured data, which is typically the most challenging part of all this. Right out of the box, we’ve built Clarifai models with extremely high accuracy in detecting these typical anomalies on aircraft.
Above is an example of a Clarifai model detecting fasteners and detecting corrosion in fasteners on a side panel (left side). You can see the confidence levels indicating how likely the corrosion is present (right side), and on the bottom, you can see a model detecting paint peeling on another aircraft.
The first step in training is to detect the fasteners. Once found, the regions are then cropped to reduce the region where we want to classify minute corrosion. Ample data is already labeled in Clarifai to train detectors and classifiers. We then build a workflow to run the detection, reduce the region through cropping, and then accurately classify the fasteners as being corroded or not.
In the inference stage, operators can capture an entire surface area and label each component such as “side-panel” or “underside.” The component metadata and timestamp are sent with the image for inferencing. Clarifai multi-model workflows (detector to classifier) can find parts such as panel connections and fasteners and score anomalies such as corrosion or leakage. At this stage, the structured data is sent for “scoring” of issues and can be compared across time to assess the need for priority maintenance.
