Reward Design Is the Hard Part: Lessons From Training Autonomous Surface Vessels With Reinforcement Learning

The Setup

The challenge was an internal Leidos AI competition in which more than sixty teams built intelligent agents to control simulated unmanned surface vessels (USVs) — autonomous watercraft — escorting a high-value unit as it crossed a contested strait against attacking small craft. Leidos wrote up the event afterward, and it's worth reading for the full picture:Leidos researchers compete to develop the best unmanned surface vessel model.

What made the competition such a good teacher isn't the naval scenario. It's that every team had access to the same simulation environment and the same class of algorithms, and the results still varied enormously. That spread is the whole story: when the algorithm is held constant, the difference between a mediocre agent and a strong one comes entirely from engineering decisions that have nothing to do with which optimizer you picked.

Why USVs Are a Reinforcement Learning Problem in the First Place

An unmanned surface vessel operating in a crowded, adversarial environment can't run on a fixed rule set. The decision it faces every moment — hold position, intercept, ignore — depends on a continuous, changing world: the positions and headings of every other craft, the state of the unit it's protecting, and the difference between a genuine threat and a fishing boat that happens to be nearby.

You can't enumerate those situations by hand. This is the same wall I've hit in much more mundane systems: past a certain complexity, deterministic rules stop scaling and start becoming a liability. Reinforcement learning is attractive here because it learns a policy — a mapping from observed state to action — by interacting with a simulated version of the world thousands of times, rather than by having an engineer anticipate every case up front.

The Algorithm Is Not the Interesting Part

There's a persistent misconception that the leverage in an RL project comes from picking the right algorithm. In a competition like this, everyone can reach for the same well-understood methods, and they roughly work. The teams that pulled ahead didn't do so by inventing a novel optimizer. They did it by framing the problem better. Three framing decisions do most of the work.

Observation Space: What the Agent Is Allowed to See

The observation space is the set of information the policy receives at each step. Give the agent too little, and it's flying blind — no amount of training recovers information that was never in the input. Give it too much raw, unstructured data, and training becomes slow and brittle, because the network has to learn to ignore noise before it can learn to act. Deciding what to include, and how to represent relative positions and headings rather than absolute ones, is a design problem, not a tuning problem.

Action Space: What the Agent Is Allowed to Do

The action space defines the moves available to the policy. A coarse, discrete set of actions is easy to learn but may not be expressive enough to execute a good strategy. A fine-grained continuous action space is more capable but far harder to train. Getting this boundary right — expressive enough to win, constrained enough to converge — is one of the least discussed and most consequential calls in the whole pipeline.

Reward: The Part That Silently Decides Everything

The reward function is where projects live or die. The agent optimizes exactly what you reward, which is rarely exactly what you meant. Reward only the destruction of a threat, and you may train an agent that fires on the neutral fishing vessel because nothing taught it that a false positive is costly. The winning agent in the Leidos challenge was the one that both defeated real threatsand correctly left benign craft alone — which is to say, the team that encoded that distinction into the reward instead of hoping the agent would infer it.

This generalizes far past naval simulation. Every optimizing system does precisely what its objective function says, and the gap between the objective you wrote and the outcome you wanted is where nearly all the failures hide. Writing a good reward is a specification problem, and specification is a skill software engineers already know is hard.

How Our Team Actually Approached It

We came into the competition without deep reinforcement learning experience, which turned out to shape our strategy for the better. Rather than betting everything on one carefully engineered agent, we treated the six months as a search problem: try every idea, no matter how crazy, as a small, cheap experiment, and only scale up the approaches that actually showed signal. That discipline — fast, disposable experiments first, expensive training runs second — kept us from pouring compute into ideas that were never going to work. Two techniques did the heavy lifting.

Transfer Learning: Don't Start From Scratch

High-fidelity simulation is expensive to run, and reinforcement learning is notoriously sample-hungry. Training an agent from a blank slate directly against the full-fidelity COSMOS simulation would have burned our entire time budget on the earliest, least productive phase of learning. Instead we did the bulk of the early learning offline in a cheaper setting, then used transfer learning to carry that accumulated knowledge into the high-fidelity environment. The agent arrived at the expensive simulation already knowing the basics, so the costly compute went toward refinement rather than teaching it to move in a straight line.

Curriculum Learning: Crawl, Then Walk, Then Run

The other lever was ordering. Dropping an untrained agent straight into the full adversarial scenario — multiple attackers, benign vessels to distinguish, a unit to protect — is a reliable way to get an agent that never learns anything, because the reward signal is too sparse and the task too hard to make early progress. Curriculum learning solves this by starting with simplified versions of the problem and increasing difficulty as competence grows. Our models learned to crawl on easy scenarios, then walk as we added complexity, and eventually run on the full task. Each stage inherited what the last one learned, so the agent was never asked to solve the hard problem cold.

Simulation Is the Real Infrastructure

Underneath all of this sits the part that rarely gets credit: the simulation environment. RL agents learn by doing, and you cannot let an autonomous vessel learn by doing in the real world when the failure mode is a collision. The Leidos teams trained against a defense simulation platform wired into an RL training loop and distributed across cloud infrastructure so many agents could train in parallel.

From an engineering standpoint, that simulator is the product. Its fidelity sets a hard ceiling on how good any policy can get, and the throughput of the training infrastructure sets how fast anyone can iterate. A team with a slightly worse model and a much faster, more faithful training environment will usually beat the reverse. The lesson I keep relearning is that in machine learning, as in most systems work, the tooling and the environment quietly determine the outcome long before the clever part shows up.

The Takeaway

The autonomous-vessel framing is dramatic, but the transferable lesson is ordinary and durable. When you point an optimizer at a problem, your real job is defining the problem: what the system can see, what it can do, and what you actually want — expressed precisely enough that a machine relentlessly maximizing it produces the behavior you intended. The algorithm is a commodity. The specification is the engineering.

If you want the full account of the competition and its results, read theoriginal Leidos write-up.