How Python and MATLAB Can Work Together | MathWorks Energy Conference 2022 - MATLAB
Video Player is loading.
Current Time 0:00
Duration 29:42
Loaded: 1.43%
Stream Type LIVE
Remaining Time 29:42
 
1x
  • Chapters
  • descriptions off, selected
  • captions off, selected
      Video length is 29:41

      How Python and MATLAB Can Work Together | MathWorks Energy Conference 2022

      From the series: MathWorks Energy Conference 2022

      While most people start by asking whether they should choose MATLAB® or Python®, the right answer is often to use MATLAB with Python. Combining MATLAB and Python, you can take advantage of the best capabilities of each environment. This enables you to reuse existing code, incorporate functionality, and collaborate with colleagues working in another language.

      Published: 22 Mar 2023

      Hello, and welcome to my presentation on using MATLAB with Python. My name is Yann Debray, and I am part of the MATLAB product team. And I specialize on all aspects related to interfacing with open source and Python in particular.

      So today, I'd like to lead the conversation by addressing the elephant in the room. Some of you may think about MATLAB versus open source and versus Python in particular, but as this was presented two years ago at the last energy conference, Gustavo Sanchez said himself, there are things that I hear about that open source can only do this. MATLAB cannot do that. And at the end of the day, tools are tools.

      So since MATLAB integrates with open source and Python in particular, the way he does is he use both. And that is what I'd like to share with you today is this idea of using MATLAB with Python. So if you want to know more about this presentation from Gustavo Sanchez in 2019, you can go look it up on our website.

      Let us start by some of the motivations for using MATLAB with Python. First, you may have a climate data API, like in this example, that is expressed in Python. And you just simply want to reuse it. You don't want to be redeveloping the code. So you can just simply grab the Python code and call it from MATLAB.

      Another example, in speech command recognition, is leveraging some of the deep learning frameworks that exist in Python like TensorFlow and PyTorch but still using MATLAB for some of the domain-specific signal preprocessing. And in this example that you can find on our GitHub repo, there's some of this preprocessing happening in the Python training loop. So calling a method from Python in this case.

      And finally, if you want to go to deployment all the way until embedded devices, you have examples where there is the battery state of charges estimator as it was initially developed with simply computers, so traditional signal processing techniques. But my colleague has developed a TensorFlow version, and I simply want to integrate it into the broader system simulation. And go all the way to code generation with that.

      So those examples cover three types of use cases. The first one would be reusing some existing code either from colleagues or from the community and using functionalities that are-- may come from other languages. We mentioned signal processing in MATLAB and deep learning in Python but also, simply collaborating between people working in different languages.

      So for today's presentation, what I aim on doing is developing this simple application that is a web app doing some prediction about the air quality based on the current weather. And for this, I'll invite you to go and get the quote directly from the repo from my colleague Heather. So you can find it with this QR code directly on this URL.

      The walkthrough will be as follows. So I've developed a simple Python snippet of code that is a module with a few functions in it to get data from a weather web service and to preprocess it. And so the first step for me will be to hand over this code to Heather so that she can use it from MATLAB and develop some basic modeling and prediction capabilities in MATLAB.

      Once she's done that, she'll hand it over back to me, so I can call the math code from my Python environment and start to go into integration into my Python web app. So eventually, what it looked like is that instead of having the Python module, I'll also be able to replace this MATLAB algorithm as a microservice that will be embedded into the final production application that you can see here.

      Before we go further, let me share with you here. So this is the web service that we will be using for today's example, openweathermap.org. This is the application that we aim at developing. So when I click on Submit, then it's going to request in the backend to my MATLAB Web service. And it's going to return the quality of the air, in this case, good and some prediction of the temperature.

      Finally, this is the code that I'll be using. And you can directly go and get it yourself. So let's start directly with calling Python from MATLAB. And for this, I'll simply bring up my MATLAB environment, as you can see here, then I will retrieve it including the repository from Heather.

      So I'm going to simply start by opening up this live script, and going to make it slightly bigger, but not just as big like this. And just to make sure that you can follow along without seeing the answer already, I'm just going to clear every output.

      So the first thing that I want to do in order to set up the connection between MATLAB and Python, is to call this function pyenv. And if I have a version of Python that is installed on the machine, it's going to link to it. Now, I have two versions that are installed, and so I'm going to pass in with a bang command where Python, in order two here retrieves the two versions of Python that I installed. So let me copy the output. And I'm going to change the version to be this one instead.

      So I'm using this new convention of name equal value. I have passing in MATLAB, and I'm defining the version to be 3.10. So I can change this version before it's loaded. And in order to simply call a Python command inside of MATLAB, I'm going to use this PI dot notation, with the name of the module. In this case, the module. And the name of the function. In this case, the square root function. And then you go you get directly the output that is wired to MATLAB.

      So now let's try something slightly more difficult with here another module called text wrap, and the function, perhaps, that is taking a string at the end, another input argument width. So I'm going to execute it.

      And it's giving me an error. And in this case, the error lays on the Python side, as I can see here because the index must be an integer. And by default, the arguments in MATLAB are not integer but double. So I will comment this out, and here execute this command line over here that specifically specifies the input to be an integral of eight digits.

      The output is a Python list. And you have a few options to convert it into a MATLAB array. The more logical one, in this case, would be to convert it in a string array. So now you directly have a MATLAB data type that is a string array.

      Let's move on to our weather forecasting example. For this example, I'll be using the sample's API. That's why I don't need an API key. I can simply use the one provided by the openweathermap.org demo. But you can also go and get your own API on the website and replace it here.

      So I'm first going to call this function, get_current_weather. And in order to see how I need to be able to call it, I can simply open the one with the py file and look at how the function is defined with name of the city, name of country, API key, and here's some optional arguments like API, in this case, the sample's API called demo.

      So let me just execute this. And it's going to give me a JSON data type. That is a nested dictionary in Python. And in order to preprocess it before turning it over to MATLAB, I'm going to use another function called parse_current_json. So I don't transform the data into metadata type quite directly. I simply pass it the Python dictionary to retrieve something that I can work with with MATLAB.

      Now this dictionary can be represented as a MATLAB struct. So I simply use the struct constructor in order to get a MATLAB struct from the Python dictionary. Now, as you can see, there are a few remaining Python types that haven't been translated automatically, like the city. And I can simply do like this, convert it into to a string. I've developed a simple function, convert_data that is going to be doing all of this automatically for me and send it back to me as a table.

      Now, this was for dictionaries and for dictionaries of scalars. Let's see how we can gather the forecast for this week in Munich and have it as a dictionary or convert it into a struct of Python lists. I can look into the data that have been passed inside of this struct, forecast struct, and in this case, as you can see, it's a Python list.

      So here, I have the option to convert it into a double. And since those data are in Kelvin, I also will do the conversion to get it in Celsius. Now, let's do the same for the current time, and it's a list of strings. So in this case, I will convert it consequently into a string, and then into a datetime data type. Once I've done that, I can plot simply the result. Now, here is my prediction for the weather in Munich.

      Now let's play around in MATLAB since we have the data inside of our environment. I'll be using a slightly more evolved datatime, which is a timetable. And I'll run with what we call here Live Task. So Live Task are available into the live data here. And you can simply do some preprocessing with low code, especially here in this case, we timetable.

      So I'm simply going to be here retiming the data, since I only get data every three hours. And I'd like to have it's data smoothed every hour. Now, I'm going to use this new table. And do some smoothing of the data. That will prompt me here a plot, as you can see, with the smoothed data. And if you want to replicate the piece, there's also the code that is generated for you directly from tonight's task.

      OK, so now that we've been playing around a little bit with data, with timetables in MATLAB, let's go to the core of our problem, which is predicting the air quality. And for this, I'll be first pre-processing the data with the function prep_data, OK, and loading a model that has been pre-trained by my colleague, Ezra, with around one terabyte of historical with data.

      And in this case, it's a simple classification tree that we'll use in order to do the prediction. So let's see what it looks like. Right now, the air quality is rather good. So this is, essentially, the predict function. That we will embed into the script, predictAirQuality that I'm going to show you just now. That will be used later on to call from Python.

      So let's summarize what we've seen so far. First, those are the basics of the Python interface on how to call Python from MATLAB. So configuration works with this function pyenv, and you can give it arguments like the location of the version that you want to use.

      For simple call to Python function, you would use this py dot notation with the name of the module, and the name of the function. And something we haven't seen in this example is pyrun and pyrunfile. Those are two new functions introduced in 2021 that enables you to simply execute commands in scripts.

      So now, let's switch to the other side, which is calling a MATLAB from Python. And for this, I'll be using my favorite environment for Python programming, VS Code.

      So I'm opening up this project here inside of Visual Studio Code, and I will here, use this format of a Jupyter Notebook in order to go through the code. So first, what I'll do is I'll also clear all of the outputs. And like in user modules in Python, I will import the matlab.engine module in order to start a connection with MATLAB.

      Next, I need to initiate a connection with the MATLAB engine with this function start_matlab, which will be stored into a variable m. And I won't, by default, show the desktop. This can be enabled with this option -desktop.

      So for now, let's just do a basic thing, which is call the MATLAB square root function. And here is the result, which you can print like this. So remember, by default, the behavior of Python is not to be printing the results. You need to specifically print it out.

      So now, if you call a function like the get common divisor function, there are several options in terms of the number of output arguments. In order to specify it explicitly, you can simply give an a gout the name of the function that could be written. And for some function, you need to also return specifically zero so that it doesn't error. So let's retrieve x1. As you can see, indexing starts at 0. So what we retrieve here from this container of data, is the second argument from survival x.

      Now let's do some exchange of data between the two workspaces. Here, we will define variable a inside of the MATLAB workspace. And in order to show it, we can simply reveal the desktop by saying m.desktop.

      So we are opening up a new session of MATLAB, which is different from the one we had working in previously. As you can see, it's only containing a. And here, I can type b equal 48 and retrieve it here by saying, OK, what is the value of b? It's 48. So this is your way of working back and forth between the MATLAB workspace and the Python workspace.

      Now let's go into our example of predicting some air quality. So we can import the weather py module that has been developed with a few functions in it. So you're getting the current weather and passing it. And now we will call the predict_air_quality function in order to assess the quality of the air. right now.

      So it's good, and right now, I'm going to show you something more involved, which is debugging if you have an error on the MATLAB side. So let me use this function here, and let's say, I'm going to put a breakpoint over here.

      And what I want to do here, is to execute this function, but here, it's going to step into each of the elements of this in order for you to debug and see what is passed as information as a struct to MATLAB. And what is here returned, and you can continue in step into the regular debugging workflow.

      So let's press it further. Just going to close this for now. And do some processing of time series once again, but this time, from the lens of the Python user. So first thing that I'm going to do is, not going to use a signal analyzer for this, but instead, I will use the data cleanup.

      So I'm going to store here the temperature from the data retrieved from here, this dictionary. I'm setting it up as a variable t in my workspace. And I'm going to evaluate a string to make it into a timetable onto my website.

      I could also, as an alternative, create a CSV file on disk, and remove it once again from the webhubsite. I'm not going to do that for now. I will directly import the timetable from the workspace.

      So here I am starting an app directly from Python. And this app this going to import from the workspace from the timetables that I just generated. So let me select it here. Let's go make it bigger.

      And that will return the timetable just as I did previously from the lens of the MATLAB user. Let's resample it every hour. And as you can see now, it has been resampled every hour. Now what I can do is smooth the data. And we simply use a moving average by default.

      So once I've done all of this, I can generate a function, which I will call directly from my Python code. Let's be original and code it .

      So in order for me to be able to call this function, I need to get this timetable on the Python side. And I will call this function and m.preprocess with this timetable. And I will store it into variable TT2, which I will then create a parquet file from. And in order for you to see the result on the Python side, you can simply read the parquet file with pandas and plot it directly. And to exit the Mat origin, simply end the thing.

      So what we've seen right now is a semi-interactive workflow with data cleaner app code from Python and a few basic commands. In order for you to install the MATLAB engine API for Python, you can simply do pip install matlab.engine. And in order to start to automate the process, you import matlab.engine. And you store the location or the instance of the MATLAB session into a variable m, for instance. Then you call, subsequently, all of the functions from MATLAB with m dot name of the function.

      So a few additional options that you may want to know about. For instance, how to use Deep Learning between MATLAB and Python. The first option would be to simply call PyTorch or TensorFlow as in user module from Python through the co-execution steps we've just shown in the first part.

      Another way for you to do import or export of models directly, is to use some of our direct importers and exporters with TensorFlow, our new importer for Python, or some extension format like onnx that is an open network extension format for neural networks. But finally, you can also directly go into our MATLAB deep learning model hub and see for yourself. Some of the available models can be used directly for transfer learning.

      Now, another example where you want to be using MATLAB with Python is simply to exchange data. And the basic numeric format that you can use is MAT file. With the support of MAT files into the site by module in Python, or as I just mentioned earlier, the parquet file, which is a tabular data format that can exchange data directly between MATLAB tables and pandas at the frame, for instance.

      So finally, we've seen all of the steps of the development. Let's now go into deployment. The first thing that I want to do is to generate the Python library from my MATLAB function. So let me show you how to do that. I developed this function predict_air_quality. And what I will do is use the REAPER library compiler here, giving it the name AirQual, selecting Python package. We replace existing projects in that package my function as a Python package.

      Now that I've done this, I can go inside of the testing repo and open my command prompt to install the new module with this common Python setup.pi. And I can go back to this other notebook that goes into a similar kind of workflow, where I import the weather module.

      We'll get the current data. And this time, instead of importing MATLAB, I will import the AirQuality module that I just generated. First step is to initialize it and store as an alias aq for subsequent call. And now, instead of calling m.predict_air_qual, what I'll do is do aq.predict_air_call. End result is still good.

      So this is a case when you want to deploy Python packages. But if you want to deploy your full MATLAB plus Python application in production, what you may want to be using is what is called the Docker microservice. And we have the ability since 22B to generate those, as well, directly from your Windows MATLAB. So this would be a really good way for you to generate this microservice that can then be integrated into our app.

      So as a summary, what we've seen inside of this presentation, is how to call Python from MATLAB with this py dot name of the library dot name of the function. We've seen how to call MATLAB from Python with the m dot notation as well. And we've seen additional options for Deep Learning and Data Exchange as well as deploying a MATLAB package as a MATLAB function as a Python package or as a microservice. Thank you.