Teaching drones to fly
Last time, I made a drone fly with PID controllers. Now I got it to finish the same test tracks in about half the time, with much less overshoot at the stop.
Loading recorded flight…
| PID | RL | |
|---|---|---|
| Median finish time | 26.4 s | 15.9 s |
| Median distance from the stop | 0.2 cm | 9.9 cm |
| Median maximum overshoot | 2.68 m | 24 cm |
Learning the motors
Reinforcement learning is letting agents learn a to do a task by following a policy to maximize a reward.
The reward scores what a desired action is, e.g. passing a waypoint for a flying drone, or a negative reward for crashing.
The policy is what the agent learns to do in order to maximize the reward.
Getting it right involves a lot of flying, scoring, and trying again. I train with PPO, which nudges the policy towards actions that did better than expected, while limiting how much it changes at once.
The network reads the drone's status from sensors. It gets eight frames of readings and previous motor commands, plus slow averages of position error and tilt. That gives it some memory of what its own commands did.
The network drives the motors directly. Work like Molchanov et al.'s stabilizing controller and Ferede et al.'s time-optimal flight was useful inspiration: small networks can handle both the balancing and the steering.
Fly, then park
For my game, I want the drones to track the player, and get in strategically advantageous positions which are in line-of-sight to the player.
This means that we calculate a desired position, a path to it, and then navigate while tracking (yawing toward) the player, stopping at a stable hover once we arrive.
I split the job between two policies. One flies the track. The other brakes, settles at the final point, and faces the look-at target.
This lets the agent focus on one policy at once, allowing greater behaviour with a smaller (more performant) network.
One is the parking expert, another is the race expert.
They have different rewards.
Flying required a lot of back and forth. First it cut corners, so I had to add a line penalty. Then it kept rotating unnecessarily, so I added a spin penalty. Then it wasn't fast enough... yada yada...
Parking gets its own reward as well. But it's simpler than the fly reward. It's a single reward term paid each tick, prioritizing being close to the point, still, and facing a desired direction.
Giving it enough practice
Both policies learned from scratch in a batched simulator: 256 drones fly at once, the policy updates on what they did, and the loop repeats. Over a billion simulated ticks in total.
Every episode draws its own difficulty. Easy tracks are short and nearly straight, hard ones are the full test track with turns any way at all, and half of all episodes are drawn at full difficulty. The easy tracks never leave the batch, but hard ones get added on as we perform better and better on the existing ones.
A few things made a visible difference:
- Let it see the line. Adding the offset to the track, together with a penalty for wandering, roughly halved the deviation. It reached the end sooner too.
- Starting in motion. By starting some simulations mid-run, we let the agent learn explore difficult-to-reach behaviours from the get-go. Allowing it to optimize for end-goal from the beginning.
- Test the handover. Two good policies could still crash together. The final flying policy arrives at the switch with less speed for the parking policy to absorb.
Give it a shove
Another thing I noticed, is that the learned drone is way better at handling shoves, impacts and rotations.
Below you'll see the effect of an impulse applied offset to the drone to cause it to push away, and spin.
Notice the difference in recovery ability.
Loading recorded flight…
This is without recovery specific training. If one wants to push this even further, adding impulses to the curriculum and training it to recover would make it even more robust.
Making practice cheap
Tasks like these are an iterative process. You try something, it doesn't work. You try again, works better, but still not good enough. You try something different, you try something similar. In the end, I must've explored close to a hundred different reward, curriculum and weight setups.
In my first training setup, a run took several hours and I had to run it overnight. After realizing that I'll need many more iterations, I took the time to optimize my setup.
I replaced the separate Jolt worlds with a flat batch of rigid bodies, compiled code to run directly on the GPU, and made many improvements to cut the time down.
Below you'll see the performance of the 10 rounds of optimization, making the work fun at the end.
In the end, I even began renting cloud machines to run my algorithms and explore parameter sets in parallel.
Being able to try another idea in half an hour made a huge difference. And now I have the part I wanted: a drone that gets somewhere quickly, comes to a stop, and turns to look at you.