Self-Driving Ferries | MATLAB Day for Marine Robotics & Autonomous Systems, Part 3

Hello, everybody. And thank you for having me. Welcome to my presentation. My name is Martin Kosch. And I am currently working as a research assistant at the Institute of Automatic Control at the RWTH Aachen University in Germany. And my current PhD studies are dealing with self-driving ferries. And this has been a research topic that has been addressed by many projects in the recent years, most prominently probably there was a project called SVAN, which was conducted by Rolls Royce and Finferries.

I think they finished the project in 2018. And they were basically the first ones to actually manage to create a more or less fully autonomous ferry that could travel long distances and also do docking processes automatically. Then there was a project from Wartsila & Norled in Norway, I think. And they did almost at the same time an autodocking feature for this vessel you can see in the center picture.

And then a bit different from that, there is a small passenger ferry from the University in Norway, NTNU in Trondheim. The ferry is called milliAmpere. And they are capable of transporting pedestrians and bicycles as well. And they are operating right in the city of Trondheim right now. And all these different projects, they face some very unique challenges in comparison to, let's say, the automation of a cargo ship.

First of all, as a fundamental part of ferries, they have to be able to do frequent departures and docking maneuvers which mostly require high accuracy. They also have to operate in very confined spaces and are usually confronted with high traffic density. And of course, they also have to do very dynamic maneuvers because of the points I said before.

And in my personal project that I'm studying during my PhD time, I have to face the same challenges as well. And as an introduction to that project, I brought the video that I will show you on the next slide. And this is actually a forward recording of the ferry crossing in my project. Let me start the video. I hope it's working. All right.

Now, you can see how my ferry just departed and crosses the large ferry area. Also, being an interaction with many other traffic participants. Now, we are right now crossing an area which is characterized by sandbanks and shallow areas, where we dock again. And then we move back. And as you might have noticed, there is some special feature about this ferry connection. Because this is not a stationary body of water. But instead, it's all happening inside of a river.

So it's actually a river ferry. And the project I am talking about is called Project AKOON. And it's for automated and coordinated navigation of inland ferries. And the objective of this project is to actually fully automate the ferry shown on the right, which is currently operating on the Rhine River close to the city of Mainz in Germany. And we are aiming at a safe and also energy efficient operation, not only during the crossing maneuvers, but also during docking and departure.

And of course, we try to get every stakeholder involved in this project. So it's not only us as a researcher, but also the ferry staff and the passengers. The project has been going on now for about three years. And it has a total duration of 3 and 1/2 years. So it will be ending this year. And it's luckily funded by the German Federal Ministry for Economic Affairs and Climate Action.

We are not alone in this project. Actually, there are three other partners from the industry. There is the company Voith, which is a multinational engineering company and in this case, the project lead. And they, among other things, produce propulsion systems for ships. Then there's us-- we are a research institute at the RWTH Aachen University. And we see ourselves as a very application-oriented institute that focuses on modeling and control of dynamic systems in various research areas.

Then there's a company called Argonav from Southern Germany. And they are very specialized in the field of radar-based navigation for inland vessels. And last but not least, there is the ferry operator itself. The company is called Rheinfahre Maul Ingelheim. And they are, of course, responsible for converting the ferry to a fully autonomous ferry, but also help us answer some very specific questions.

Let's have a short look at how the ferry operation area and the ferry itself look like. In the top right picture, you can see a picture of probably the most complicated area in the ferry operation area characterized by sandbanks that are quite hard to see from above. And also, it is a very narrow passage that we have to pass. And also, because we are operating in the river, there are quite strong and turbulent currents that we have to expect.

And of course, since we want the ferry to dock frequently, we have to reach a very high position and accuracy below 10 centimeters. And the ferry itself, you can see it right here as a tiny little dot in the picture is a rather old ferry built in 1987. But it's quite large. Actually, it is 57 meters long and has an empty weight of 255 tons and a draft of only 95 centimeters.

And it is propelled by four so-called Voith-Schneider propellers, which enable the ferry to not only go straight, but also move sideways and rotate in place, which is very helpful for these kinds of maneuvering exercises. And during our development, we decided to go for a three-stage approach, starting with local simulations on regular development PCs. This was mostly done in Simulink and is almost finished now.

Then we did a transition to a full-scale vessel simulator provided by the Voith company. And we use this simulator currently for hardware-in-the-loop tests. So we actually use Simulink to deploy some of our controllers to proper industrial controllers and also used all the final hardware interfaces that will be used on the experimental theory. And right now, we are in the transition onto the experimental ferry, where we hopefully will be able to do everything that we could show in the simulator before.

Next, I want to give you a broad overview about the system architecture and then also dive into some of these modules in a more detailed way. Going from left to right, you can see all our data sources on the left. For example, there is one software module responsible for the perception of the environment around us. For example, there is a current and wind estimation or measuring algorithm going on and also a depth estimation.

Plus, we need to know where our ferry is all the time and how it is oriented and how fast it is moving. And this is done using a so-called navigation filter that is basically a sensor fusion algorithm that is based on an inertial measurement unit and a GNSS receiver. And also, there is a module provided by the company Argonav that provides us with data on dynamic obstacles. And they are based on AIS data and radar and also LiDAR.

And all this information goes inside the trajectory planning module, which is one of my tasks. And this module is responsible for calculating safe and also energy-efficient routes that lead the vessel from its current state to the desired docking position. And this calculated trajectory then goes into a trajectory tracking controller. So it's a controller that decides which forces or thrust forces need to be acting on the ferries so that we can follow the planned trajectory despite any disturbances.

And the control forces coming out of this module then go inside a thrust location module, which is responsible for deciding which propeller should be running with which rotational velocity and with which pitch angles in order to actually provide the amount of thrust and preferably in the energy efficient way. And this goes all to the propellers.

And above everything, there is a human-machine interface. Of course, you want to be able to interact with the whole system. And this can be done mostly by two interaction possibilities once there is the traditional control lever interface. But also, we are experimenting with a 3D joystick currently. And of course, there's also a state machine, which is responsible for orchestrating all the software modules and taking care of potential errors to make sure that everybody is safe even when there are problems.

And now I would like to focus on the current estimation module for a moment. Because that's very interesting and unique. The background is that the theory is very much affected by the current in the operation area. So that's very important knowledge not only for the control and thrust allocation module, but also and especially for the trajectory planning module. Because it is very much helpful to know which current will be acting on the ferry in order to optimize the fuel consumption.

So what we actually would like to have is a map of the current field. And this is not very easy, because the current field, of course, changes over time-- for example, when the water level changes, or when the geometry of the riverbed changes. So we thought of a way to actually measure the current. And we did that in a rather unconventional way by using a so-called acoustic Doppler current profiler, short ADCP.

That's the sensor you can see right here in blue and white hidden in a metal cage to protect it from driftwood. And we mounted this sensor on one side of the ferry looking forward, looking ahead and sending out ultrasonic beams out of these black transducer surfaces. And the method in principle uses the Doppler shift and the time delay of these ultrasonic beams reflected on particles moving with the water.

And this helps us to obtain measurements of the current velocity or the component of the current velocity vector at different points on the beam directed in the direction of the measurement beam. And this sounds a bit complicated, and it actually is a bit complicated. Because for a human, these measurements are very hard to interpret. And so we thought of a very advanced way of post-processing this data and to aggregate all incoming data into one single adaptive map of the current field.

And without going too much into detail because I don't have too much time, we do that using two computation steps. First of all, we compensate the movement of the vessel itself. Because this sensor is only able to give us measurements of the relative velocity. And after compensating the movement of the vessel itself, we use a combination of so-called Kalman filtering and Gaussian process regression to give us a map that can be continuously updated as we move through the ferry operation area.

And this might sound a bit complicated, but I brought a short animation that should make clear how that principle looks like. On the right, you can see basically our ground truth in this case. This is a bird's eye view of the ferry operation area. And all the little arrows you can see are current velocity vectors that came out of a fluid simulation-- the fluid simulation.

And on the left, what you can see if I click on the simulation in the second, you can see how the little ferry moves through the operation area. There are also the two ultrasonic beams pointing ahead. And as the ferry moves through that area, it starts to explore the current. And it stitches all the data together into a single current net. And also, as you can see, we assumed no prior knowledge.

Actually, we assumed that the velocity is 0 everywhere. Let me just start this animation. There you go. Now, you see how the ferry moves through the area. And it collects data and aggregates the data inside this current map. And the nice thing is that we don't are limited to using this zero velocity prior knowledge. But we can actually use this simulated current map as a prior knowledge and starting point of our current estimation so that every time we move through the operation area, collect data and use it as an update to our map.

And this estimation is actually based on real measurement data. So I'm very happy that this idea of current estimation seems to be working in real life. And now, I would like to focus on, let's say, the heart of the automation-- so everything that is needed to actually control the ferry and make it move from one side to the other. And this is all managed starting by using the trajectory planning module.

And as I said before, this module is responsible for finding a safe and efficient route to the docking position, starting from the current position, wherever that might be. And we would like that to be fuel-efficient, but also time-efficient. So we want the ferry to stick to the scheduled time. And of course, the plan trajectory has to respect the limitations on the dynamics of the ferry. And that makes things really complicated.

And in fact, as you can imagine, this ends up in a really, really big searching space, such space that has to be explored during an optimization. And as we do need to do that in a very limited time of only a few seconds, we decided to reduce this search space using a so-called path velocity decomposition approach.

So the idea is basically that we assume that oftentimes the path, so the locations that the ferry visited during a crossing, are mostly similar. And the only thing that really changes is the velocity profile that we use to travel along this path. So what we do here is we use a predefined or actually pre-recorded trajectory that is, for example, pre-recorded using a manual steering.

And we optimize over the velocity profile so that we optimize fuel consumption and travel time. And also, we can make sure that the velocity profile is chosen in a way that the dynamic constraints are not violated. And right behind the trajectory planning, there comes a trajectory tracking controller, which is supposed to calculate generalized thrust forces. So in this case, a longitudinal and lateral thrust force and your moment in order to track the reference trajectory.

And also, to help to track the trajectory despite any disturbances. For example, when there is a modern mismatch or unexpected winds. And currently, we try to do that using a rather simple linear quadratic regulator, LQR, which you might know if you are familiar with control theory. This is rather a simple approach but also a very good one. And it is able to deal with multiple-input and multiple-output systems as we have in this case.

And as one major problem, we have a rather nonlinear system. And using a successive linearization approach, we can deal with these nonlinearities. The problem, however, is that LQR controllers are not able to deal well with control limitations or maybe delays and input rate limitations. So we are actually currently experimenting with other control approaches as well.

And after that, there is a thrust allocation module that decides which rotational velocities and which pitch angle all of the four propellers need to reach in order to efficiently realize the demand thrust. And this can be formulated as a regular thrust control education problem using a constraint quadratic optimization problem.

And on the right, I brought a short animation of these three modules playing together in a hardware-in-the-loop simulation in this case. And if I press Play in a second, we will see how the ferry, right now located here in the South, starts to move along a predefined trajectory towards the docking position in the North. And while doing the crossing maneuver, it encounters two or even three obstacles on the way and chooses the velocity profile in such a way that collisions are avoided.

Now, you can see how the ferry slowly starts to move. And now, it detects that there is a vessel coming down the river. This is the gray box in this case. And all the red rectangles only are keep-out zones in this case. And now, you can see how another unexpected vessel comes out of our sandbank area and our ferry manages to hit the brakes really hardly. And that's the third vessel passed before then proceeding to the docking position.

The current stage of our project is as follows. We have managed to upgrade the ferry completely. So we installed a multitude of new sensors on the ferry-- for example, a wind sensor, multiple Janus S antennas in use, and also the current sensor I was just talking about. And also the user interface does not look like a ferries from the 1980s anymore. But instead, it's all new and shiny and state-of-the-art and has a lot of touch screens involved.

And we also did some land surveying on the ferry and many other things. And luckily also, the propulsion system is completely upgraded to a steer-by-wire system. So we are actually able to control the ferry using our algorithms. Our local simulations are mostly finished by now and also our hardware-in-the-loop tests using the full-size vessel simulator are finished. And we are currently trying to get these algorithms onto the real ferry.

And last month in May, we had our first closed-loop testing. And this will be going on for the next few months. And I'm very excited that we managed to reach this point. Now, you might ask, what is coming next? We are actually currently planning our next project, which would be in some regard quite similar. So the ferry itself will not really much change, but we are heading towards a maritime scenario next time with a very different traffic scenario. And even though I can't really say much about that now, if you're interested in it, stay tuned. And there will be much more coming in the near future.