Robotarium: Remote Access Robotics Lab

I am excited about what I've been asked to talk about, which is the Robotarium at Georgia Tech. This is a remotely accessible swarm-robotics lab that we created in order to democratize access to a sharp, world-class research lab.

When we did that, we obviously did not know that there was such a thing as COVID-19, and one of the things that I do want to spend some time talking about is how the Robotarium has been leveraged since middle of March, when virtually every researcher on the planet went home. The Robotarium has actually continued to play an important role in all of this, in a way that we had not predicted. So I am going to pivot towards the last part of the talk to really discuss remote-access instrumentation and experimentation in this new reality that we are finding ourselves in.

But just to get started. First of all, I want to paint a little bit of a mood picture. So this is my lab at Georgia Tech a decade ago or so. These are 12 robots that are doing formation control of various forms. And these kinds of videos are always included in talks given by people like myself, meaning, look we proved some theorems and here at the end, the robots are moving.

But there's always stuff going on that are not shown in these videos. Around these robots there are people, grad students and postdocs, standing, making sure that everything is going all right. There's also a lot of infrastructure around it. In this particular case, we have a motion captioning system. We have other kinds of sensors and cameras. There's a lot of stuff not actually shown.

And to put it a little crudely, here's the stuff that's not shown. It costs a lot of resources to build, run, and maintain a world class research lab in the area of multi-agent robotics, in particular, but in robotics in general. And the problem with that is there are only a handful of labs globally that really matter, in terms of where most of the exciting experiments come out of.

And that is not because there are only a handful or so people with good ideas. Far from it. There are tons of people that are really, really doing excellent work in theoretical multi-agent robotics or simulated multi-agent robotics. But the fact that it requires so much to run and maintain a world-class research lab, it is a barrier to entry.

So what I set out to do five plus years ago is really get rid of this barrier. Try to build some kind of system that lets people with good ideas test their ideas without having to incur the costs that come with running a lab like this.

While we did that, there are other things that have been irritating me in my lab. One of the things is we're spending a lot of time duplicating efforts. We're reinventing wheels over, and over, and over, and over again. We're writing the same device drivers or the same fast planning algorithms in my lab as in Sertac's lab at MIT, just to pick on Sertac, who was on the screen just a few slides ago. So that seems like a massive waste of energy and time.

Also, all of these labs are massively underutilized. I amused myself a handful of years ago by just counting-- what percentage of the time are my robots actually moving around? And I clocked in under 5% of the time. And my lab is active. Even in a highly active lab, more than 95% of the time the robots are just sitting there being glorified, overly-expensive paperweights. So that doesn't seem right.

There are all sorts of other reasons why the way we are structuring our labs may not be the right way, not just a matter of barrier to entry. So long story short, we set out to solve this problem. And the overarching vision-- the idea was let's build an open, remotely accessible swarm-robotics testbed that people from all over the world could access, upload their code on, and basically test their ideas.

So I went to the National Science Foundation. They have a program called MRI, which stands for Major Research Instrumentation. So I went to NSF and pitched this idea, and they liked it.

This picture here that you're seeing, this is literally the picture that was in my MRI proposal. It's the Vienna Chamber Orchestra's concert hall, first of all, with some Harvard Kilobots sprinkled on top and then some random graphics that I've found that-- this is really pushing the limits of my photoshopping skills.

So anyway, we said, this is we're going to build. And we got the money, and this is what we built. Actually, this was the prototype. Here is what we actually built. We did build a, I must say, rather visually stunning remotely accessible lab. It ended up going live in August 2017. So it's been in operation almost three years.

The first thing that happened, of course, August 22nd when we went live is-- so this is a quote from Field of Dreams, which is a Kevin Costner movie that you may have seen or not. And the quote is "If you build it, they will come." In remote-access robotics, it's almost true. You just need to make one little modification, which is if you build it, they will not come.

In fact, it turns out that the history of robotics is littered with remotely accessible robots that no one ever used. So the biggest problem we had in the beginning was we had built this thing that looked really good, but no one was interested in using it. I spent a lot of time, initially, calling my friends, begging them to use it.

In the beginning, it was kind of a slow uptake. The first users were basically my friends that I called in favors from. So Jorge Cortes, at UC San Diego-- I still owe him a lot, because he was the first user of the Robotarium. And there I, at the end, basically said, please just give me some code that we can put on the Robotarium. So that's how it started.

But it slowly got more and more traction, and around month three or four, or so, we started doing a whole lot of media around it, before we really had the user base in place. In fact, we were on the front page of the Wall Street Journal. And this was quite exciting, actually, to see that the popular media got excited about the Robotarium almost before the researchers that were going to use it got excited.

But the media, me begging, the Robotarium starting to actually pan out-- all of a sudden, maybe six months after its launch, we started seeing tons of submissions. And the first thing that happened was people started breaking stuff. Because we wanted people to actually run things.

So these are code that had been submitted. In the beginning we went air and ground, and as you see, things are falling out of the air quite a bit. So lots of robots got broken in the beginning, once we launched the Robotarium.

And this was actually a problem for us. We wanted this to be broadly used. We wanted to allow anyone to upload code. We needed it to be a research instrument.

And I should say already at this point, research is different from teaching. If you're building a teaching lab and you want to teach someone about PID control, then you have three well-defined knobs, P, I, and D. Then you let people tweak those knobs, and that's it. You're over-constraining the system, and that's the only thing you allow people to do.

In research, you have to allow it to be open. You have to let it be flexible. Also, when you have an idea and you test it, finding out that that idea was not a good idea is oftentimes as useful as finding out that it was a good idea. So we have to allow for people's code to be bad. We have to allow for lots of flexibility. And this became a problem.

In fact, a lot of the problems-- there we go-- a lot of the problems had to do, of course, with things colliding-- colliding with each other, colliding with the boundary of the arena. So we needed to do some kind of rudimentary safety verification steps in there, having to do with collision avoidance.

This of course is old. Collision avoidance is as old as eponymous robotics, right? And in fact, the first thing you do when you are a new robotics student is you write a collision avoidance or an obstacle avoidance routine.

Here's, by the way, a picture of-- I still think it's my favorite autonomous robot collision. This is MIT's self-driving car in the DARPA 2007 urban driving competition colliding with Cornell's self-driving car. And Mark Campbell and John Leonard are still arguing to this day who's at fault. I don't know. I just think it's a little amusing to see.

We also had, by the way, a beautiful, beautiful car in the Urban Challenge. And we did really, really well. We made it to the semifinals. We were performing, I think, quite admirably until we didn't. So here's our car. It's a Porsche Cayenne. I don't know if you could hear that, but that's a horrifying sound of robots colliding. So point being, collisions are bad.

Now, luckily for us, if you go to Google Scholar and you look for collisions, you end up with quite a few hits. This was, maybe, four or five months ago. There are 323,000 papers or hits on collision avoidance, and 118,000 hits on obstacle avoidance, and none of them worked. We didn't try all of them. But they were inappropriate for what we needed to do in the Robotarium.

In particular, we had a special setup that made our lives different. The first thing that was fundamentally different in the Robotarium from almost any other application is the robot density is really, really, really large. What that means is that if you use a standard kind of collision avoidance routine, where you have a force that goes like 1 over the distance squared-- Lennard-Jones type binding potential ideas-- then that force becomes too large when robots get too close, and you can't pack enough robots together. So the density is high.

The other thing that we've got going for us that's different is our agents are actually allowed to collaborate. Virtually all of collision avoidance deals with the following problem-- the other robot is going to do whatever it does. What do I do to be safe? But all of a sudden, we're in a situation where we're allowing the robots to kind of help each other out to avoid collisions. So that's the second thing that's different.

The third thing that's different is we really don't know what it is that the robots are trying to do. A lot of times, you can be smart when it comes to collision avoidance if you know what the goal of the robot is. But we don't. Someone uploaded code and ran something on the Robotarium. We don't know what this is trying to do.

So what we have to do is we have to deal with high robot densities, robots that are actually able to collaborate, and where we don't know what the overarching objective is. The way we solved this was with barriers certificates.

And in particular-- so here is, instead of math, here is what we're saying that we're doing. What we're taking is-- we're looking at the actual controlling inputs generated by the user at any given point in time, and we're trying to get as close to that control input, the actual control input, subject to a do-not-collide constraint. And this do-not-collide constraint is a differential constraint expressed in terms of the control signal.

The details you can find now. It's based on these control barrier functions, that are relatives to control the output of functions. But it allows us to be minimally invasive in a very particular mathematical sense, which means make as little change to the control inputs as possible, subject to don't collide.

So with that lined up, here's what it now looks like. So going back to this cartoon. This was actually an experiment where we're asking the robots to swap positions by driving through exactly the same point, which is a car crash, right? And now, with the barrier functions, we're modifying the controller. We're not telling them how they should solve it. All we're saying is, hey, make these changes if collisions are imminent.

What I like about it is that it looks almost right. It looks natural. A lot of times, you look at robotic algorithms and you go, yeah, kind of. This feels right. And it's exciting that they're, as an ensemble, figuring out-- in this case they should rotate clockwise, in the other case they did counter-clockwise.

And it works because they are collaborative. It works because it's minimally invasive. It also works even if some agents are not collaborating. So here we have a big robot that just goes, no, I'm not moving. And the little ones have to run away to not get run over. But the beauty of this is it's still safe, even though it's no longer minimally invasive.

So we put that on the Robotarium. We also did the same thing in the air. We have these quads that we saw falling out of the air. There we put barrier functions around them. It's not just coalitions in this case.

Here in the example, we have four robots flying in a spiral or a circle, and Li Wang, one of my students, is taking one of the robots and dragging it through the spiral without the robots colliding. And again, the user is actually trying to drag the robot through the spiraling robots, and the barrier functions actually prevent them from colliding.

So this is now running in the background. So here is on the actual Robotarium, thirty-five robots, and they're asked to do this experiment where they're supposed to go through the same point in the middle of the arena. So this is a massive robotic car crash.

Here is where the barrier functions kick in, and they figure out that the right way to manage this is by this spiraling motion. We don't tell them what to do. All we're saying is here's the constraint that needs to be satisfied by the actual controller.

So this took care of a lot of our safety problems. Right now, if you upload code in the Robotarium, you're going to be running it with this, unless you can convince us, in simulation or by pestering us by email, that what you're doing is actually safe.

Uh, I'm having some issues.

Here are some of the very early experiments that were run on the Robotarium with these kinds of barrier functions. This is on the prototype version, the tabletop. One was Mark Spong doing some kind of formation controllers out of University of Texas, Dallas. Seth Hutchinson was doing fault-tolerant rendezvous, and Masayuki Fujita from Tokyo Tech was doing attitude synchronization. But the point here is more that with these little tweaks, we could allow and open up a lot of different types of experiments without having to worry about people breaking stuff.

So having said that, here is how it actually works right now. You go to the Robotarium website. The link comes in a bit. It's robotarium.gatech.edu. The first thing you do is you register as a user. Then you can download this simulator, and you run your code in the simulator, hopefully. We have no way of checking, but hopefully.

When you're happy with your algorithms, you submit a simulation script. This is a code snippet, and it's a little bit of an experiment description. We do some rudimentary checks on the code just for making sure that there are no weird runtime errors, and there are no other nebulous things happening.

Then it gets placed in a queue. It's a first-in, first-out queue. When the Robotarium is ready, we're pulling it from the queue. It's automatically being generated. The robots do their thing, they return to the charging stations, and the user that submitted a code gets the data, and the video feed, and whatever other sensor measurements they wanted back.

The reason, I think, I'm at a session sponsored by MATLAB has to do with the fact that we ended up going MATLAB in the Robotarium. In the interest of full disclosure, when we started, we thought we weren't. We thought that what we were building was something that's going to run ROS, Robotic Operating System, because that seems more natural, by tapping into what's going on.

But it turned out that a lot of our early users, they're control theoreticians. They like to program in MATLAB. We ended up having quite a few biologists studying social insects, and we had social scientists, we had applied mathematicians. They don't want to hack in C++, but they do know how to use MATLAB. So the way you upload code is MATLAB scripts.

In fact, here's what it would look like. This is the submission process. I'm not really expecting anyone to read this, but you do things like, you set the number of robots. In this case, you have an experiment with 20. You initiate the objects. You explain how long the experiment is going to be running, how many iterations.

Then, you get the positions or the poses of the robots. These are differential drive robots, so they're nonholonomic. The default positions are xy and theta. If you want to run them as single integrators, you can convert to single integrator dynamics. Then you have to have some main code, and then at the end, you basically return to velocity. It's as simple as that.

Here is an example of what the main code might look like. This is a consensus equation. All you're saying is, you know what, let's get the topology. You can specify different kinds of topology, if you want to do this graph, or a fixed topology, what that should look like. Then you basically specify the velocities. And here, this is the standard linear average consensus equation, and that's what gets, then, submitted onto the actual robots.

So having said that, like I said, in the beginning. I had to basically lie, cheat, and steal, and beg users to show up and start using them. Since August 2017, so this is now almost up to three years, we've had over a thousand distinct users. These are now people that have signed up with an account.

We have, actually, way more experiments then we have users. This is quite exciting. We have a lot of repeat customers. It also turns out that 40% of the users never actually run an experiment. We've tried to understand why that is. A lot of them only use the simulator, because we have a simulator that's, again, in MATLAB, and that's something that people enjoy getting access to. We've also learned that a lot of the users are working in teams, so 10 people sign up, but one person is the only person that actually submits code.

But anyway, across these 1,200-- it's a little bit more, actually, now-- distinct users, here's what the geography looks like. These are the registered users. These are relative, so these are fractions. North America has over 50%, the US and Canada. Asia is second and Europe is third. Maybe not overly surprising.

But then, I was very happy that we have users from South America. We have users from Australia, Oceania. We have users from Africa. In fact, we have users from every continent except Antarctica. So if there's anyone listening to this right now who happens to run a research lab in Antarctica, please go and upload code on the Robotarium so I can say we have users from every continent, which we don't right now.

Another thing that's interesting, though, talking about the users, is if you don't look at the registered users but the actual experiments that are being executed, then there are more from the US and Canada, more from North America, that are submitting things. Asia is now number three, and Europe has overtaken Asia in terms of the number of submissions.

And there, it seems like people are really using the simulator more than anything else. To me, it's a little interesting that just being able to provide a reasonably good simulator for large teams of robots has proven to be engaging.

The other thing that we've looked at are what kinds of things are people interested in. So these are the topics that are associated with the experiment. When you submit an experiment, you have to basically say, roughly, what it's about. You could put a blank space in there, that works, but people don't. People are being nice.

A lot of them are class projects. Even though we did not set out to design a teaching instrument, we set out to design a research instrument, but people are using them a lot in classes.

Then there are things like formation control, path planning, leader-follower, coverage control, consensus, locking and swarming, collision avoidance, task assignment, connectivity maintenance. These are all topics that you can find all over [INAUDIBLE]. So it turns out that the kinds of multi-agent system stuff that we as a community are interested in, are, indeed, what people are putting on the Robotarium.

The other thing that we've been quite excited about is we just hit our 100 mark in terms of peer-reviewed papers not from our group that use the Robotarium as the experiment platform. here. Is what the papers are about, if you squint. And again, could be ACC. So the fact that we've generated 100-plus peer-reviewed papers all out of the Robotarium, not with us, the Georgia Tech team, as authors, is fantastic. I'm really, really happy about that.

So having said that, there are a few things that we've learned about remote-access research along the line. One is this question of security. There is a physical security that we solved with these control barrier functions, but there's also a cybersecurity question. We're literally giving people access to physical things on Georgia Tech's campus, and Georgia Tech's OIT, Office of Information Technology-- they're in a constant state of nervousness around this, because there have been multiple cyber attacks on the Robotarium.

This goes back to this tension between research and education. If you're building an educational testbed-- this is an instructional lab-- then you can constrain it so that the constraints are rather tight, and the only thing you can really ask of this system are the things you're interested in in your educational module.

That's not how research works. You have to allow strange stuff to happen. You have to allow failed experiments. In fact, one thing we realized very quickly is that if we can imagine what people are going to do with the Robotarium, then it's probably not worthwhile. It's the stuff that we cannot imagine that really, really matters. So that's something that we're still kind of struggling with a little bit. How do we make it maximally flexible?

Right now, for instance, we have an overhead projector that allows people to project down scenes in the Robotarium. So instead of dragging out obstacles, we have a central computer that hallucinates the obstacles, and tells the robots what they would see if they actually had sensors that could pick up these obstacles.

And we've seen ant hives, we've seen road networks and traffic, we've seen a lot of urban environments with buildings, we've seen farm fields, so cornrows. We've seen quite a bit of different kinds of scenes that people are projecting into the Robotarium.

The one thing that's hard about this, that I'm constantly worrying about, is the funding model. You can find money to start something like this. And I went the MRI route. Then I got what's called the DURIP. This is instrumentation funds from ONR, the Office of Naval Research. But that's not maintenance money.

And to me this is very, very important, that the costs of using the Robotarium is free. The whole point is there should be no barriers to entry here. So it has to be free. But it's hard to run a business on free.

Over the last year, year and a half, we've discovered that companies, for-profit companies, are using the Robotarium. So we're now standing up the model where, you know what, if you're a company and you make money off what you're doing in the Robotarium, you should probably pay something.

The other thing that's happening is people are raising money. They're writing grant proposals where they're proposing to use the Robotarium as the platform. And then we're trying to make sure that there is some way of putting in Robotarium support, also, in other people's grant proposals, as a way of funding it. But this is a problem. So right now it's being funded by me building Robotarium into whatever grant proposals I write. But that's not a long-term solution to this.

It turns out that this is not unique to the Robotarium. There is out of Rutgers University a very long-established wireless networking remote-access lab. They have exactly the same problem, which is you can find the funds to build it out, but how do you maintain it over the decades? And this is something that is tricky.

OK. I said I was going to spend just a little bit of time talking about what happened during the pandemic. So this is the monthly usage numbers, starting in August 2017. One thing that was interesting is in the middle of March, when we were, at Georgia Tech, sent home, and we were sent home all over the world, we were allowed to maintain critical activities.

These were research activities focused explicitly on COVID-19 rapid response research. It was things like stuff that would blow up if you didn't have someone in there, or experiments that have been running for six months and you needed two more weeks to get the final data, things like that.

Well, the Robotarium was deemed a critical activity because it helped researchers all over the world after they went home. So what we did is we kept it running throughout the shelter-at-home phase that we're still, to some degree across the globe, in.

But anyway, if you look at the data, the first six months, like I said, were miserable. The first peak was really me begging people to use it. But then it started to pick up. It's interesting how bursted the date is. We actually went back and tried to figure out what was going on.

A lot of these peaks coincide either with IROS or ICRA deadlines, these are the two main robotics conferences, where we're seeing a huge uptick in usage. Then a few know large class projects that also correspond to peaks.

But what I particularly want to point out there is after people went home, mid-March of this year, we've seen usage grow, and grow, and grow, and grow. And this is really, really quite promising. I think, even though we set out to build something to democratize access, what we're really built is a way for people to keep running experiments, even though they have to shelter at home during a pandemic crisis. To me this is also-- this is a side effect that we certainly didn't expect, but it is, I think, very, very exciting.

Having said that, I have started to talk about making lemonade now, in the age of the coronavirus. What I mean by that is we've all been handed a big, fat, sour pandemic lemon. What are the things that we can actually do that is positive around this, to make lemonade, if you will? One thing that we've seen that I already talked about-- lots of submissions since we started the shelter-at-home phase.

Other things that I think are very positive that have not happened just during the pandemic but also earlier is we've seen labs start to federate. Here's an example, for instance, of researchers at Tokyo Tech in Tokyo, across 11,000 kilometers and a 400 milliseconds communication delay round-trip, coordinating robots in the Robotarium with their robots.

The fact that you have robots in Tokyo and robots in Atlanta doing things together in a coordinated fashion is fantastic, because it allows us to expand the reach and the footprint. If everyone has one robot, then all of a sudden we can federate them together and get to how many robots we may need, and it's running in my living room and in your living room. That's cool.

There are also a lot of lessons that we've learned that we have helped transition to other labs. These are other robotics labs. This is a picture from the Duckietown ETH in Zurich. They have a Duckietown Robotarium, which is a remotely-accessible version. It says "Robotarium watchtower camera feed." It's not our Robotarium. It's their Robotarium. There is a Robotarium Paraguay that's come on line, and then stopped being online. I hope it's coming back. There's a Robotarium-West. Kennedy Space Center is talking about standing up a Robotarium. So there are other Robotarium being built, and if we could federate them together I think that would be spectacular.

I'm also excited to see things happening in other domains. I know for a fact that Sonia Martínez, for instance, at UC San Diego, is building a distributed energy version, or at least plotting to build a distributed energy version, based on similar types of principles. There are lots of cybersecurity and cyber physical security labs now that have popped up that use these kinds of principles. This is all great.

But from my point of view, the number one thing that makes me the most happy about the Robotarium is that I think we have succeeded in democratizing access to world class robotics. The fact that the demographics look the way they do, both in terms of geographical spread but also who the people are, really is exciting.

And you can say, yeah, yeah, but you only have 10 people from Africa. But you know what, 10 is strictly greater than zero, and I am really, really happy with that. Something to build with. We haven't actually torn down the barriers to entry, but we certainly made them less tall. We've had girl scout troops run experiments.

We have had quite a few public high schools, like I said, labs in Africa and South America. Next week there's a Robotics hackathon in Nepal that's about to happen in the Robotarium. That's cool. So to me, these are things I actually want to take with me as success stories.

So with that, I want to say thank you.