Testing maritime autonomy: USVs, UUVs and the role of simulation

Why maritime autonomy is hard to test

A day at sea needs a vessel, a support boat, a crew, a weather window, and often a safety observer. The weather that matters most for testing is the weather you least want to be out in. Encounters with other vessels cannot be staged safely in large numbers, and once an undersea vehicle dives, you mostly cannot see what it is doing until it reports back or surfaces. Field testing alone produces small samples of rarely repeatable conditions.

Testing surface vessels against the rules of the road

For uncrewed surface vessels, collision avoidance is central, and the reference is the International Regulations for Preventing Collisions at Sea, known as COLREGs. Typical test encounters start with the basic geometries and grow from there:

• Head on approaches
• Crossing situations from either side
• Overtaking and being overtaken
• Several vessels with conflicting obligations
• Situations where following a rule exactly would cause a collision, so a departure is required
• Other vessels that do not follow the rules at all

Grading maneuvers objectively

Grading behavior is harder than it looks. One approach published by maritime autonomy researchers scores each maneuver against several objective measures, combines them into a weighted score, and uses a fixed core set of encounters to compare autonomy software versions and catch regressions. Where to set the threshold for safe remains an open question, which is a good reason to report the individual measures and not just a pass or fail.

Environment factors for surface autonomy

Surface vessels see the world through sensors that the sea itself works against:

• Sea state: roll and pitch move the sensors, tilting LiDAR beams and camera horizons
• Radar clutter: returns from waves and rain can hide small targets, and turning gain down to suppress clutter can hide real contacts too
• Small craft: low profile boats may appear on radar only intermittently, even in calm water
• Glare, reflections, fog, and heavy rain degrade cameras and LiDAR
• AIS gaps: small craft often carry low power AIS units that transmit less often, AIS is optional for many vessels, and some vessels transmit nothing
• Wind and current change how the vessel answers its helm, especially at low speed
• Communications: line of sight radio range and satellite link delay affect remote supervision

Closing the loop on perception

The common thread is that perception degrades exactly when the situation gets harder. Researchers have also noted that perception is often evaluated separately from estimation and control, which leaves open how a brief sensor dropout changes the vessel's actual path. Closed loop testing, where degraded perception feeds the real planner, answers that question.

Testing undersea vehicles

Underwater, there is no GNSS. A UUV typically takes a GPS fix at the surface, then navigates by dead reckoning with an inertial navigation system aided by a Doppler velocity log. Inertial error grows without bound over time, and the velocity log works only within a limited range of the seafloor, so it can lose bottom lock in deep water or return bad readings. Radio barely penetrates seawater, so communication relies on acoustic links that have little bandwidth, suffer from multipath and Doppler shift, and change over the course of a day. Because the link is so limited, the vehicle must make most decisions on its own. Useful UUV test themes include:

• Long duration navigation drift, with and without velocity aiding
• Velocity log dropouts and loss of bottom lock
• Acoustic messages that arrive late, corrupted, or not at all
• Logic for when to surface or seek a position fix
• Seafloor driven behaviors such as terrain following and obstacle avoidance
• Currents that push the vehicle off its planned track
• Integration effects, such as interference between sensors that each pass bench tests on their own

Where simulation helps

Simulation lets maritime teams do what the sea will not allow: run every encounter geometry, at every speed and visibility, as many times as needed, and repeat an exact situation after a software change. Scenario sets can be grounded in reality by drawing on AIS traffic data from real waterways and on the judgment of experienced mariners. Sensor faults, radar clutter, AIS silence, and acoustic loss can be injected on purpose. Some programs connect the real autonomy software to professional ship simulators and hydrodynamic models so experienced navigators can judge its behavior across many scenarios and recommend a safe operating envelope before live trials.

Where maritime simulation falls short

The ocean is difficult to model. Wave and hydrodynamic models simplify reality, radar clutter and small target returns are hard to reproduce faithfully, and acoustic channel models often assume conditions hold steady when they actually shift quickly. Few simulators model networked acoustic communication accurately. Treat simulation as a way to explore and to prioritize, calibrate it with logs from your own vessels, and confirm key findings in the water. Defense evaluators in particular expect models to be validated against live data before results carry weight.

Darkstar is an early stage simulator project that starts with drone and swarm teams, and USV or UUV teams who want a voice in what it tests can say so on its design partner page.

Questions