A Cloud-Based MATLAB Visual Inspection System
Hear about a use case study describing a repurposable cloud-based visual inspection system powered by image processing and computer vision algorithms running on MATLAB Production Server™. See how to use mobile device–based workflows to visually catalog and inspect objects of interest via streaming video and image capture and implement a DevOps workflow.
Published: 7 May 2023
[MUSIC PLAYING]
Hello, everyone. My name is Brett Shoelson. I'm a principal application engineer with MathWorks. I've been with the company for 18 years, focusing almost exclusively on image and video processing and computer vision. I'm here today to talk about a cloud-based MATLAB visual inspection system with my colleague, Arvind.
And hello. I'm Arvind Hosagrahara. I've been with MathWorks for 21 years. I work as a solution architect, putting together systems like the visual inspection system that we're going to talk about.
So before we get started, let's just take a brief moment to talk about what we mean by automated visual inspection. This term is used interactively for our purposes with automated defect detection, optical inspection, machine vision, or automated inspection-- basically, any time you want to detect failures or evaluate quality defects, typically in the manufacturing processes.
So when we were building out our cloud-based visual inspection system, we had some requirements in mind. We wanted the system to be secure as if you were doing banking online. We wanted it to run at scale so that it could support 10 users or 10,000 users, and we wanted it to be repurposable so that you could substitute different algorithms for different tasks as you use the system.
We operationalized our MATLAB AI solution on the cloud using microservices built to modern standards and best practices for scalability and security, and we used DevOps processes for agility and development and for deployment of our AI and vision algorithms.
So we've got a sample problem here. Assume that you are on the assembly line for a Raspberry Pi board, for example, and your job is to find defects or to determine that a board is free of defects. Defects could include misaligned components, bad assemblies, other damage, missing solder, labeling mistakes, just about anything. So how do you go about solving this problem?
Well, let's show you how we did it with an actual demonstration. We're going to show you the MATLAB visual inspection using our cloud-based system.
So here you see our iPad. We're pointing it at-- we're going to point it at a QR code, which we'll talk about briefly in a moment. This QR code triggers the analysis of Raspberry Pi boards. And we open up to our portal where we can open up and drop or drag photos or take precaptured photos. Here we've got a Raspberry Pi board.
And as soon as we capture it, we send it up to the cloud where it calls the MATLAB algorithm behind the scenes, indicating that it's received the image and it's processing it. And in just a moment, we get an annotated image that tells us that it's found three defects in this board. We'll talk about those defects in just a moment. You see three different defects.
Let's try that again and show you that this system can be robust even if the boards are skewed or at different angles. So here we take another image of a Raspberry Pi board.
Again, snap the picture, send it up to the cloud, let MATLAB do its magic. And again, in a matter of a few moments, MATLAB will analyze that skewed board, the skewed image, and again tell us that it's found three defects on that same board.
It's desirable to constrain the capture of this. This actually allows users to position their handheld devices better, to capture the right perspective of the board in this case. To do this, I just enable an overlay. And by doing so, it gives me a little bit of a stencil that I can actually position over the board or position the board within the stencil, very similar to how you would deposit a check-- say, for example, with your banking application.
Triggering the shutter on analysis like this takes the image, captures it, and sends it over to the cloud for storage and for subsequent analysis, like in the previous cases. It takes a few seconds to actually reanalyze the image, but we've seen what that looks like. It's not necessarily always to use the camera. You can actually preprocess existing images that you have stored in the past.
In this case, we're going to take an undamaged board-- take a good board, a board with no defects in it, and send it over to the server. And within a few seconds, MATLAB would-- the MATLAB image processing algorithms in the back end would actually kick in, analyze the board, and identify that there are no damages. This is something that a human would find difficult to do with this kind of speed and efficiency.
The system can be repurposed for different applications. To try something completely different, we have another algorithm that analyzes the ingredients on food packaging. Everybody has food at home. And off the ingredient list, we can send images over to a different piece of analysis to identify allergens. In this case, I'm looking for the word "lemon." And I can tell that this particular tea has a lemon flavoring to it just by having MATLAB analyze the content.
The start of this process is the QR code that Brett talked about just a few minutes ago, and it's really simple to create these QR codes to point to certain analysis, to point to certain workflows for the person capturing the images. To create one such analysis, filling out a form or some of this can be populated or attached up to your other systems. It would lead to a particular analysis on the application. And once you add the workflow, what the system will do is generate a QR code for you.
This QR code can be printed off if necessary, placed near the analysis. But essentially, it becomes a very quick way for you to place this by the manufacturing desk so that you can actually trigger the analysis. Here I have a few different QR codes for different kinds of entry points into the application. Taking a step back and putting on a different hat like an IT administrator of the system on the cloud,
I wanted to spend a few seconds showing you what the back end of the system looks like. This is typically what an IT organization does. They put together DevOps pipelines and any changes to the scale of the system that is infrastructure expressed as code, to any changes to the application structure or the system as a whole-- changes to the database, changes to the web application, changes to the MATLAB code. DevOps pipelines kick in that would first configure the cloud, run the tests to make sure that what we are deploying is working, will containerize or dockerize all the applications, update the Docker repositories, and use Kubernetes in this particular case to deploy to the live system.
What you're seeing here is a mechanism driven by DevOps processes for continuous delivery, continuous delivery of applications, of system functionality on a continuous basis as required. In this case, on successful execution of a pipeline pass, it can even produce release bundles of software.
A system like this needs to be monitored and managed. In this particular case, we use OpenTelemetry. And if you took a look at the performance, what we do is-- say, for example, you were interested in how long MATLAB Production Server took to run these. For each call, I can actually see where MATLAB is spending its time, how long the requests take, where I should spend my optimization dollar to make the system better. The system as a whole can be scaled. It's designed to be scaled.
It's sitting on Kubernetes, which makes it very simple for you to scale or even auto-scale if required. So if you were to pick up, say, for example, a particular service, it is almost trivial to be able to scale that up. We use Kubernetes native storage for storage of the CDF, so the compiled MATLAB artifacts that get delivered as services through MATLAB Production Server.
And finally, the back-end storage systems are all cloud based, which means to say , my data is going into buckets that are sitting on the cloud. MATLAB has latent capabilities of actually connecting to this. For example, we're using S3 in this case. You can configure MATLAB to pick up these images as they're being collected. This actually drives training and other workflows in MATLAB for developing your algorithms which you can stick on the DevOps pipeline and continuously update the kind of algorithms available on the system.
So to take a step back and take a look at the system architecture and overview, let's zoom out to 10,000 foot. What we have is an application that can be built using web app server, products that we produce, or web applications of your choice that send images, videos, and video streams, webRTC streams over to a back-ed system that are being analyzed by MATLAB. Think of it as having MATLAB on the other end of a video teleconference, at which point the images are computed and returned to the user. All of this is built on top of Kubernetes, which sits on top of cloud infrastructure.
At the highest level, this enables access from any mobile device with a standards-compliant browser. You don't need to install any software on your device. It becomes accessible for people to actually use their device, their iPads, iPhones, Android, whatever it is, or regular PCs to connect it to cameras that actually have fixed apparatuses on the manufacturing floors to send data up to the cloud.
DevOps processes for MATLAB-- MATLAB users like Brett can continuously deliver their algorithms to a system like this where it gets operationalized automatically. And IT managers can actually-- IT administrators can actually manage a system like this using DevOps for updating other parts of the functionality for scaling the system up and down.
So as Arvind mentioned, I am a MATLAB user. I'm not an IT person. The point I want to get across here is that if you're building your algorithms for visual inspection, you have all the tools and all the functionality for all the image processing, computer vision, deep learning, statistics, machine learning, whatever toolboxes that you've installed on your system. You can build out essentially anything you want to build out using these tools.
In addition, though, we have add-ins. So if you went to our homepage and went to our add-ins box, you would find a way to automatically install the Computer Vision Toolbox Automated Visual Inspection Library that's going to extend the capabilities. And I strongly encourage you, if you're doing visual inspection, to go grab this package.
In particular, this Algorithm Development page is going to show you some nice ways of getting started with nice examples. It's just really a good way to jumpstart your efforts. And if you dig in, you can see the functionality. You can see that we have state-of-the-art fast flow, patch core. Whatever kind of analyzes you need, whatever kind of deep learning approaches you want to take, we can help you support. And remember that we're always here to help as well.
So I'm not going to talk too much about how I approach the problem here. Basically, at a high level, I created a template. And with the computer vision approach, I found the template, and I used it to orient the Raspberry Pi board. And then having oriented it, I did a detection for the components that we were interested in looking at.
And then I compared it from a good board to a bad board, and I found that three of these components are poor. One is rotated 100 degrees, 180 degrees. One is missing, and one is just misaligned. So we can easily detect and classify errors of this nature.
And as we built this out, we built it again to be triggered by a QR code. We configured it to support live constrained capture on your iPad or your iPhone or even your Android device. And we haven't talked about it, but as you're using the system, you can actually build up a collection of ground truth with which you can build better and more accurate models. And we built it to be scalable and to return the annotated imagery and the reports that you need to assess your problems.
So putting my IT administrator hat on, looking at it from a solution architecture perspective, in a sense, we are gathering data from users of the system that are using their mobile devices in this case that send it over to a backend system where it is persisted on the cloud. This enables capture from any environment, be it a fixed camera on top of an operating-- on top of an assembly line process or from handheld devices like we just showed you. All it requires is a web-compliant front-end, and this would capture the data and store it into cloud-based storage systems.
That means that you've got practically unlimited amount of storage in the back-end. Scalable microservices can be orchestrated. The MathWorks publishes reference architectures to help you very quickly put together products like Production Server on the cloud. You should take a look at it on our GitHub repositories, and we certainly can point you to that or help you with that.
The end output is containerized deployment of a Kubernetes-based system that will align with your IT organization's best practices for scalability and security. You can build a system like this, as Brett put it, to be as secure as your banking website. And finally, enabling DevOps processes means that a system like this can continuously evolve with your requirements, all of this running on cloud-based storage, compute, and network infrastructure. Zooming out even further, the implication of what we're discussing is a modern DevOps-based automation for continuous deployment of MATLAB applications to a system that actually allows you to operationalize it.
A little look under the curtain of what the system looks like. We have systems that actually will allow users to manage their Kubernetes better and scale it up and down, sometimes as simple as actually clicking a button up and down based on the capacity that is desired, based on the performance, based on the desired quality of service.
At the outcome, we wanted to conclude this talk with just takeaways from what we are actually describing or what we described, the takeaway being that MathWorks products, both toolboxes block sets as well as our server products, along with published support packages and published reference architectures, can be leveraged to put together production-grade solutions, in this case for visual inspection, but production-grade solutions on the cloud.
These systems can be built to be secure, scalable, and agile. It will dovetail with modern DevOps processes, and they can be built to best practices. And finally, MathWorks as a company has a huge array of domain-specific toolboxes and support packages. This allows people to write less and do more and allows MATLAB users to go from their prototypes that they can build very quickly into production. That concludes our talk. We open up for questions and answers. Thank you. Thank you very much.
[AUDIO LOGO]