Formula Student Driver-in-the-Loop Simulator Using Simulink and Unreal Engine

Hi, everyone. This is Veer Alakshendra from the MathWorks student programs team. And today, we're again going to welcome one of the student from the Dynamis PRC student team. And we are going to talk about a new topic that how the team created the driver-in-the-loop simulator using Simulink and Unreal Engine.

So again, welcome all of you to the MATLAB and Simulink Racing Lounge another video. And I am handing over to Daniele. So Daniele is going to introduce himself, his team members, and he is also going to mention few of the name of the people, those who have contributed in this project. So over to you, Daniele.

OK. So hello, everyone. Thank you, Veer, for this amazing opportunity. I'm glad to guide this video about this amazing formula student project we have been developing.

Also, Francesco has been the main contributor to this project. He couldn't be present here, but I wanted to acknowledge him, as well as Ivan Ferrari, Emanuele Motta, and Andrea Fontana for the double-track model, but also Mattia Triccoli for the telemetry model, for the roadrunner scenery, Alessandro Nusiner, and from the custom vehicle mesh, Vincenzo Di Martino. So thank you, all of you, and I'm glad to have this opportunity to present our driver-in-the-loop simulator.

Awesome. Daniele, thanks. And yeah, I can see there are a lot of people involved in this project. And I can say that it's a whole team work. So over to you again to talk about your achievements and your team members.

So our team in Dynamis PRC is composed of around 130 people. And this year has been the most winning year of the electric category of our team. So we have been able to reach some overall podiums, as you can see from the slide, in Switzerland but also in Czech Republic, and also a third overall place in driverless category at FSG.

So this has been a year full of accomplishments, and this could not be possible without a great track work but also a great simulation work. That is why we are now today presenting a tool that involves a lot of simulation, which is exactly a driver in the loop simulator.

I'm really excited in what you're going to talk about today. And at the same time, it was really great to meet all your team members at the formula student Germany. And now let us go ahead and show us what you are going to demonstrate in today's video.

So we are going to see this video. I hope that everyone likes it. As you can see, there is literally a driver driving our car in a formula student track. So we have been developing the physics of the model, the graphics, the cones position, the custom scenery, the sounds, the custom HUD.

Everything is custom and done by the team. This is very useful in order to simulate the car, to literally drive and feel the car while it is still being built or while some development are required. In addition, we also added a live telemetry model so that we can stream data to an external laptop in real time and be very effective and very quick in decision making.

OK, great visuals. And at the same time, I love the way that you have also added the collision effect in the Unreal Engine. To all the viewers, these are the topics that we are going to cover. So basically, we will focus on the whole methodology, how Daniele and his team members built the whole simulation and the driver-in-the-loop loop simulator.

And at the same time, we also talking about, what are the model capabilities and how are the results that they obtain? And once the session is over, we'll also, again, summarize things. And we'll also share some of the resources that can help you build something like this. So over to you, Daniele. Let's talk about that. I mean, one thing which I'm curious about is that this driver-in-the-loop simulator, you guys have built it, but why is it important for formula student teams to build driver-in-the-loop simulator?

Yes, Veer, because the driver-in-the-loop simulator is the perfect way to use a full car model, for example. A lot of teams already have a vehicle model, and you could literally drive your car in a simulation. Furthermore, you could evaluate different setups in few seconds, and you could train drivers to specific formula student dynamic events. You could train them to memorize a specific formula student track, but also, the drivers themselves could learn how to drive in a skidpad, for example.

You have free time. It would cost no money while booking tracks. Testing on track can be expensive, and you have limitless time. You can just play and run the model every time you want. The analysis that you do on a driver-in-the-loop simulator, lastly, are useful not only to vehicle dynamics, for example, but also to every department. So there are a lot of useful information, a lot of useful data that everyone in the team can use to their advantage.

Coming to the last point, when you say that useful analysis for several departments. So what I understand is that, for example, once you have built this driver-in-the-loop simulator and you have the setup, so anyone from different departments, for example, let's take the example, someone who is working in the powertrain department, and they want to drive it and collect some data for their analysis. So they can easily come and drive it. So they don't need any kind of knowledge in what's going inside the software, right?

Yes. That's how we want to do to build this simulator. So we have also created some intuitive menus and graphic interfaces so that everyone in the team can launch a simulation easily. The driver will be driving possibly, but everyone could launch a simulation autonomously and get some data according to what is the field of interest. So this is a very good point, and I'm happy that you raised it, Veer.

Yeah, Daniele. So another question is that, what is the importance of creating a custom simulator over here?

Yes, Veer, because we decided to build everything in MATLAB and Simulink so to have the maximum customization freedom possible. You can have a lot of modeling capabilities. You can modify and explore your code as you want. You can import and export data quickly and easily use it with MATLAB, for example, for post-processing.

You can customize your visual, your sound effects. You can create your own vehicle mesh, and you can easily make your system interact with other Simulink subsystems. So the whole integration is very easy.

And the last but not the least, as you may have seen in the demonstration video, you could also perform a real-time telemetry, since TCP communication that we'll see later on is possible. Thanks to MATLAB and Simulink TCP block sets, for example.

I mean, those blocks are really helpful. OK. This gives a good understanding again, that if you have a full control on the simulator that you have built, it's easy also to modify and focus on all these points that you have mentioned. So until now, we have understood why a simulator is important in formula student competitions and also what is the importance of having a custom simulator. So now let us talk about what is the methodology or the workflow that you and your team members have utilized?

So this is a very precious piece of information that we managed to have. Also, thanks to the incredible support of the whole MathWorks team that I'm really grateful to. So this is the architecture of our driver-in-the-loop simulator. This is due to one main necessity, which is reaching a real time target. So making the whole architecture run in real time, since the driver has to respond quickly to the inputs that he does.

So there are three models, the physical model, the graphical model, and the telemetry model. The physical and the graphical model run on the same computer but in two different MATLAB instances. In this way, you could run multi-thread simulation, and this results in a way faster simulation. The physical model includes the driver model, the vehicle model, control systems, the battery model, data logging, and so on.

This model sends data to the graphical model through a TCP connection, which exploits the loopback port since we are in the same computer. The 3D vehicle position the forces, the talks, the camera position. The sounds are sent to the Unreal executable file, which is a file that contains a RoadRunner scenery, a custom vehicle mesh, a custom HUD, the colliding cones, and so on.

Through the simulation 3D message set, instead, these are some very useful block sets which allow you to send some signals to the Unreal scene. In this way, we can modulate the volume and the pitch of the sounds, for example, and we can set a custom camera position, which is crucial to have an on-board camera for the driver. So this is happening on the same computer.

On an external PC, there is the telemetry model running. It receives data from the physical model, always with a TCP connection but this time with Wi-Fi since it is wireless. The telemetry model receives signals as the driver inputs, control systems, parameters, what is happening in the battery, the forces of the model, the lap time, the lap number, and so on.

You can stream them with the Data Inspector. In this way, you can know quickly what is happening, whether a setup change, for example, has been effective, and you can save a lot of time. So this is the whole architecture. And I suggest everyone to split the whole driver in the loop simulator in these three models so to have the best simulation experience possible.

Yeah, this is like-- even though it's one slide, but it is having a whole lot of content. So just to summarize what I understood or maybe what viewers must be understanding is that, in one PC, you have two models, and in one model, you have your-- which you are calling it as physical model.

So that is comprising of, for example, the vehicle dynamics model, the powertrain model, or the control system. These are all in one model, which you are calling it as physical model. And then, you have another model, graphical model, which is basically communicating with the Unreal Engine.

So basically, from physical model to your graphical model, you are sending these position orientation and other things like forces and torques. And regarding the 3D environment, you are sending camera position, orientation, lap time, and other piece of information. And I know, we are going to, again, see the demo where you will be showing different blocks.

And then, on the other PC, you have the telemetry model, again, through which-- I mean, again, in that what you're doing is you are sending the data from the physical model to the telemetry model. Thanks, Daniele. Thanks for the explanation. It is pretty much clear how things are working on in your model. So I have a question that since you have two models running, so how are you making sure that these two models are running in real time and there is no latency in these two models?

Yes, Veer. This is a big problem that we are going to address maybe even more in detail later, but the best tip I can give is to use a fixed step solver because the driver-in-the-loop simulator requires a fixed step with an explicit solver like ode8, for example.

And we ensure a stable connection, a stable simulation by choosing very carefully the integration step, which can be different between the physical and the graphical and the telemetry model but in a way that computational efficiency is reached. So an integration model, which is little enough short to have a proper simulation but not too short. Otherwise it would require too much computational effort. So this is how we ensure that we are always faster than real time, and then we paste the simulation to the real time.

That's a good approach that you all have implemented. I would also suggest another alternative. If teams know about Simulink Real-Time time so they can also check out some of the resources that we already have. We already have a one video where we are showing the driver-in-the-loop simulator using Simulink Real-Time and the real-time target machine like Speedgoat.

So we'll put that link in the video if you all want to explore what's going on and what's the workflow to do to build a driver-in-the-loop simulator so they can explore. Thanks, Daniele. Thanks for answering this question. So Daniele, let's quickly talk about your model capabilities

So as introduced before, we have a user friendly interface so that everyone can launch a simulation properly and so that setting up the simulation is very quick but also, the Simulink models themselves are rapid to execute. We have introduced also an automated conversion script so that log data can be exported into data packages for third-party analysis tools.

Through subsystem references, we can easily replace submodels. And this is something that saves a lot of time. And we can simulate different formula student events but also different formula student tracks.

Thanks for sharing the model capabilities, Daniele. Another thing which I'm curious about is that we have seen the simulation workflow. You just shared the model capabilities. I'm curious to see what is the setup look like.

Yes, because this could be also very interesting for all the other teams. So what Dynamis PRC has done is to take the DP8, so an old combustion prototype of the team. The internal combustion engine has been removed and has been replaced with a PC with a proper computer. Three flat screens have been positioned at the front and also, a Fanatec steering wheel always at the front.

The steering wheel itself instead is custom, as well as the carbon fiber pedals. And this allows us to have an immersive setup and also something we are very proud of because we are giving a second life to an old prototype of our team.

Awesome. This is a smart move utilizing your previous vehicle. Great. So I'm really excited to see the live demo. So if you can start with the demos and open the models that you have built, that would be really great. Thanks, Daniele. So let's move to the most exciting section. That is the demonstration of your model. So over to you, Daniele.

Thanks, Veer. So this is the menu we have prepared so that everyone can easily select the track, switch among the Fanatec interface, for example, or an open loop maneuver. Everyone can select easily the setup of the car and choose among aerodynamics, suspension, control systems, kinematics and tires, and so on. And you can also open the several models of the architecture I have already presented.

Let's start from the physical model. So this is the physical model that regards everything about the car itself and models, the physics of the car.

So there is a driver model, the control systems, an input interface for proper conversion, the whole set of parameters, for example, the setup parameters, the vehicle dynamics subsystem, an output interface, and then the audio subsystem, the lap timing, the logging subsystem, and then two subsystems that allow the interface with the communication with the graphical model and the telemetry model.

To start with order, let's go inside, for example, the driver model. This is something very easy to do, but probably a lot of teams do not know. You can easily connect your steering wheel with this joystick input block. This is something very easy to do. It will automatically recognize your steering wheel.

You can give, as input, the force feedback that comes from your vehicle dynamics model. And then, you can easily get the steer of the car, the steer of the steering wheel that the driver is applying, the throttle, and the brake input. You could also add some buttons, for example, the DRS button in our case.

Moving on. These are the control systems, the vehicle dynamics blockset, but I wanted to stress the attention on the audio, for example. This is something simple in theory. We are just assigning to a signal-- a specific variable.

For example, for modulating the wind-- so in order to have a wind noise that becomes higher and higher when the car increases its speed, we are just using the velocity of the car that comes from the vehicle model. And we are using this variable as a gain to modulate the intensity of the wind sound.

The same applies for the tire, so for the tire slip, both in longitudinal and lateral, and for the sound of the electric engine. For this, we are using instead the rotational velocity of the wheels. Going on, we have added a lap timing subsystem, which basically consists in a chart.

So this is a chart which cuts and refreshes the lap count and the lap time every time the car crosses the starting line. Indeed, every time we are under a certain threshold, so a certain distance from the starting line, the lap count is updated, so is increased.

And whether the lap time is the best of the session, we are saving that lap time as a best lap so that the driver can know his best lap time. In this way, we can cut the different laps and we can also save. We can also know the specific lap times that the driver has reached.

Then, we have also added a logging subsystem, which is very, very easy. So the whole struct of signals is saved. Thanks to a two-workspace block. This is very easy. And we can post-process signals in MATLAB at this point.

So Daniele, just a quick question. So now, I'm curious to that how are you sending the signals that you're getting from your physical model to the Unreal Engine?

So we are sending data both to the graphic and to the telemetry. As far as the graphic is concerned, there is this subsystem. Basically, we are packing all the useful data, as you can see here, the sound but also the position and the rotation of the tires of the chassis and so on, through a byte packing subsystem. So we pack all of this data, and then we send it with those two very precious subsystem.

We require TCP server and then a TCP send block, which literally sends all this data to the graphical model. In this case, we are using the loopback port since, as I said previously, we are in the same computer. Physical and graphical model are in the same computer.

Yeah. It makes completely sense actually.

And the same is true for the telemetry. So we are packing all of this data and sending it through the same subsystems. This time instead, we are using a Wi-Fi connection. So the EP address will be just different.

Perfect. Completely understood. Basically that the ID that you have to mention, that needs to be changed, depending upon where are we sending, whether we are sending to the same PC model or we are sending to the telemetry model, right?

Yes.

I would also like to ask Daniele one question. That last year, we recorded one more video with the Dynamis PRC team, where we invited Niccolo, and he built the whole model in multi-body for the formula student competition. So how are you utilizing that model over here?

Yes, the multi-body model is very useful because we are taking the kinematics curves generating from the multi-body model in order to have the proper inclination angle of the tires in the double track model so we know the correct dynamic camber and toe position, for example. Thanks to the curves to the lookup tables, which are three dimensional. Thanks to the multi-body model.

Thanks, Daniele. And now let's move to the graphical model.

Sure, Veer. So here we select graphical model, and here we are. Basically, we are now featuring the same subsystems we have seen before, which are TCP block sets, but this time, we want to receive data. So we use a TCP client to connect to the server and then a TCP receive block.

The data is packed, as we have seen in the physical model. So we are unpacking the whole data. And then we are sending the several signals to other subsystems. First of all, we are creating the translation and the rotational matrices. These are useful for the Unreal Engine visualization. Let's go inside it.

So there are the translation matrix and rotation matrix, which are necessary for the vehicle visualization. The scale is just set to 1. And then, the scene is created. Thanks to this block. On the other hand, we have a custom field of view and a custom camera position, which is assigned-- which is sent to the Unreal scene. Thanks to these blocks, which are also very useful. They are called simulation 3D message set.

So basically, these blocks are used to send the messages to the Unreal Engine?

Yes.

On one hand, on the simulation 3D vehicle block, you can also select a custom vehicle mesh. Also, the formula student vehicle mesh as a default is provided, but you could also include a custom mesh. And then, you can connect to the Unreal Engine executable file through this simulation 3D scene configuration block, and you just have to select Unreal executable and then select the exact executable file. Now let's go inside the Unreal Editor scene just to see how it has been developed, how it has been built.

Meanwhile, the model is opening. I can just mention that the block that you used for adding the vehicle, the custom vehicle, so there is a video also that we have recorded that can guide you to import your custom mesh. Because we know that most of our teams, they have their own files, 3D files for their car so they can import that watching that video or there's also documentation present. So that will be linking in the video description.

So this is the scene in Unreal Engine. Basically, it has a lot of custom features. For instance, there is a RoadRunner scenery, which is the main scene you can see in here, which allows to model the proper track.

So those who are new, so RoadRunner is a MathWorks tool, which is used for designing 3D scenes for automated driving simulation.

Yes. Thanks, Veer. And this has been very useful to generate the actual track so the driver can feel where he is driving in the actual track. We have also replaced the normal clouds and the normal weather with a 360 image that we can get from Google Maps, for example. And then, other custom features have been added, like the cones that you can see here. They can collide, so they can fall down, and they replace automatically after a specific amount of seconds.

Another very cool feature is the HUD. So on the screen, the driver can see the speed but also, for example, the lap time and the lap number and a bunch of control system parameters. How can we assign a specific number in this HUD? We are using the simulation 3D message send block that I have mentioned previously.

So from the Simulink environment, we are assigning the exact number of the lap, for example, the exact number of the lap time, as well as the correct speed number so that the driver can have a visual interface of what is happening.

Then, in addition, there are different sounds, as you can see from this right window, that we use to simulate the engine, the tire slip, and so on, but we have created also other sceneries. Let me just show you, for example, FSG, that a lot of you may know. And so in this case, we have also recreated the FSG scenery with the iconic grandstand on the back.

So basically, you have multiple scenes added in your Unreal project, and then you can go from one scene to another, right?

Exactly. So according to the executable file you choose in Simulink, you can test, you can drive in a different racetrack.

Perfect, Daniele. This kind of clarifies that how you have built the scene in Unreal Engine.

Thanks, Veer.

Perfect, Daniele. So can we just move to the telemetry part?

Sure. So if we open instead the telemetry model, you'll see it has a very simple concept. It is basically very similar to the graphical model since it has to receive data with a TCP connection, this time with the Wi-Fi EP address. Data will be unpacked, and then we use this to stream these channels with the Data Inspector. We are using, for example, these channels as the velocities, the steering angle, the throttle pedal, the brake pedal that I can just show you now.

So once opening the Data Inspector, you can see we could stream, for example, the velocity of the car, the steering angle, the brake pedal, the throttle pedal. This is a recorded simulation, but however, you can do this live. So you can understand live what is happening to the car. And this is a very precious piece of information for everyone, for every department. And this can save a lot of time.

Thanks, Daniele. So we have seen the workflow. We have also seen the major parts of the physical and the graphical model. And you just spoke about the importance of Data Inspector. So now what we are going to do is that we are going to play the video, since it was difficult to record the video and at the same time run the simulation in real time and running the simulator. So what Daniele has done is that Daniele has recorded the video and shared with us. So this is what we will see now.

So thanks, Daniele, again for showing the driver-in-the-loop simulator in action. And now, I'm curious to know that, how are you validating the result or what's the outcome of the driver-in-the-loop simulator that you have built?

Yes, Veer. As you mentioned, validation is a crucial thing for any type of simulation and/or simulator. We are comparing the lateral acceleration through a sign steer, while in the bottom picture, there is the longitudinal acceleration of a straight acceleration.

In the top case, we are comparing this with the multi-body model. Since it has a higher order of accuracy, we want to compare the double track model, which is present in the simulator, with something more precise to see if we are going in the right direction, while in the bottom, we are just comparing it with the actual acceleration logged with the sensors in the real car.

So you have a one multi-body as a reference model, and that you are comparing with the double track model? And then, that model you are comparing with the log data?

Yes.

OK. As a follow up question, can you talk about the challenges and how did you address them? The main challenge is balancing the accuracy of the model with the computational efficiency. So please, everyone listening to this, if you are interested into making driver-in-the-loop simulator, make sure that your model, every time you add something, every time that you make the whole simulator more complex, please make sure that it will run in real time because this is crucial.

So focus on accuracy, but at the same time on computational efficiency. On the other hand, you have to be aware that a certain familiarity with Unreal Engine and RoadRunner must exist. These are two new software, and you might know how to use them. This could not be so direct.

They are pretty different for some traits to MATLAB and Simulink, but the best solution I can suggest to everyone is splitting the architecture into two or more models maybe, as we have said before in the architecture overview, and definitely just watch the MathWorks video. There are a lot of tutorials on YouTube. They can guide you on how to export an executable file in Unreal Engine, how to make a custom mesh, and so on. There is a lot of resources. Please use them. They are very useful.

Thanks again, Daniele, for sharing the challenges and how did you address that. So just to add over here that most of the teams, they are familiar of the videos that we published on the Racing Lounge. So now, these are some additional skills that you would require, for example, your familiarity with Unreal Engine and then, as I mentioned, this RoadRunner tool for 3D autonomous driving simulation, for which we have a full video series.

So at the end of the video, we'll be sharing all these resources, and we'll also link them in the video details so that you can explore those and learn and build your own simulator. So Daniele, we have seen the workflow. We have seen the models, and you just spoke about the challenges and how you address them. So we know that we are at the end of the video. Can you quickly summarize and just share some of the key takeaways?

Surely, Veer. So the main point of the driver-in-the-loop simulator is that you can test in a controlled environment a lot of different setups quickly. You have limitless time. You spend no money. You have no fear of crashing your car virtually. So this is a very good point. And you also have the possibility to perform accurate and efficient simulations as long as you want, with all the customization you want. And this is also another very good point for performing analysis of any type of any kind.

But as I just said, always keep in mind the computational efficiency, which is something very crucial because every time you make the model of the car or the control systems more complex, you may not be able to run real time. So please just consider always your efficiency in terms of computation.

Thanks, Daniele. So just throwing some lights on the computational efficiency. So I think the key over there is that what teams can do is that they can have various subsystems.

So based on what their goal is, whether they want to build the model for the driver-in-the-loop simulator or they want to build a complete full body, multi-body simulation, which requires high fidelity, or if you're a new team who just want to start with a simple model, so in that case, you can use various subsystems. And accordingly, you can switch based on your requirements. So once again, I would like to thank Daniele for the clear explanation of the workflow and demonstrating the model to all of us.

Thank you, Veer.

And once again, if you all have any queries, you can reach out to us at racinglounge@mathworks.com. And we have also started a new Instagram page where we'll be sharing multiple resources on a weekly basis. So feel free to join. And these are the other resources. And additional resources, we'll be just sharing with you now on the screen.

Thanks again, Daniele. I think this piece of information will motivate multiple teams across the world, and they will be motivated enough to get started and build their custom formula student driver-in-the-loop simulator. Thanks again.

Thank you, Veer, for the amazing opportunity. Thanks to the whole MathWorks team for the amazing support. And I'm eager to develop even more exciting tools together. And I hope that this could be helpful also for other formula student teams.

Sure. Thank you. Thanks, Daniele.

Thank you, Veer