Adversarial scenario testing for autonomous systems

What adversarial testing means

The term covers two related ideas. The first is operational: the system faces an opponent or a hostile environment, such as jamming, spoofed sensors, decoys, interceptors, or other vehicles that break the rules. The second is methodological: the test process itself acts as an adversary, deliberately searching for the conditions that make the system fail rather than sampling conditions where it is likely to pass.

Both matter because autonomy fails at the edges. Defense test and evaluation researchers have noted that traditional testing mostly establishes what a system can do, while autonomy also needs evidence about what it will not do. The US Department of Defense policy on autonomy in weapon systems, DoD Directive 3000.09, calls for testing in realistic operational environments against adaptive adversaries taking realistic countermeasures. Outside defense, any customer deploying autonomy into contested or unpredictable settings will ask a version of the same question.

Anatomy of an adversarial test case

A test case should be specific enough that someone else can rerun it and judge the result without asking what you meant. Recent work on adversarial testing of AI enabled systems suggests recording the following for each case:

• The adversary's capability and what it is allowed to do
• The initial state of the system and the world
• The manipulation or adversary behavior applied
• The expected behavior and the property that must hold
• The observed behavior and whether any objective was violated
• For simulation runs, everything needed to reproduce the run exactly: scenario definition, random seeds, software and simulator versions, configuration, and the terrain or map used

Building a scenario library

Test cases like these live in a scenario library, the reusable heart of adversarial testing. A good library is not a pile of one off scripts. It is a structured collection of parameterized scenarios. A single logical scenario, such as pursuit by an interceptor during a GNSS outage, defines ranges for timing, geometry, speed, and environment, and concrete runs are sampled from those ranges. In automated driving, declarative formats like ASAM OpenSCENARIO are used so scenarios stay portable and events can be timed precisely, because random background traffic cannot reliably recreate a hazard that depends on exact timing. Useful categories for drone, swarm, maritime, and ground autonomy include:

• Navigation attacks: GNSS jamming, plus gradual and sudden spoofing
• Communication attacks: jamming, intermittent links, partitions, delayed messages
• Perception attacks: sensor spoofing, occlusion, glare, decoys, camouflage
• Reactive opponents: interceptors, pursuers, adversary swarms
• Rule breakers: vehicles or vessels that ignore right of way
• Environment: weather, lighting, sea state, terrain masking
• Field failures: every real incident, rebuilt as a scenario

Organizing and searching the library

Organize the library by mission and operating envelope so you can see what is covered and what is not. Keep a core set that runs after every software change as a regression suite. Then use search, from random sampling to optimization methods such as genetic algorithms, to probe beyond the core set for new failures. Results tend to cluster into performance modes, and the scenarios on the boundaries between success and failure are the most informative, both for engineering and for choosing which live tests to run.

Why deterministic replay matters

Deterministic replay means that running the same scenario with the same inputs produces the same result every time. It sounds like a technical detail, but most of the value of adversarial testing depends on it:

• Debugging: a failure you can reproduce on demand is a failure you can fix
• Fix verification: rerun the exact scenario to prove the fix worked, rather than hoping
• Regression: confirm that old failures stay fixed as the software changes
• Version comparison: hold everything else constant so any change in outcome comes from the code
• Trust: an evaluator can rerun your evidence instead of taking your word for it

What breaks determinism, and how to get it back

Without determinism, a flaky result is ambiguous. You cannot tell whether a change fixed the problem or simply rolled different dice. Simulators are often assumed to be repeatable, but many things break that assumption: floating point behavior that differs across hardware, multithreading and GPU scheduling, variable time steps, wall clock dependencies, message timing between processes, and unseeded randomness anywhere in the simulator or the autonomy stack. Some physics engine documentation states that results reproduce only on the same hardware and software version, and that soft body simulation is not guaranteed to be deterministic. Practical countermeasures:

• Seed every source of randomness, and seed the simulator and the autonomy stack separately
• Run on simulated time with a fixed step, not the wall clock
• Pin and record simulator, engine, driver, and software versions with every result
• Log every input that crosses into the system under test
• Compare trajectories within tolerances when runs may move across hardware

When the system itself is not deterministic

Learned components, sampling based planners, and concurrent software can vary from run to run even in a perfectly repeatable world. Separating the two sources matters. If the simulator is fixed and the result still changes, the variation belongs to the system. Fix seeds to debug a specific failure, then run many seeds to estimate how often it occurs, and report it as a rate rather than a single pass or fail. Researchers in AI security accept this kind of probabilistic reproducibility when a behavior appears consistently under defined conditions.

Also note the difference between replaying logs and replaying scenarios. Replaying recorded sensor data is open loop: it shows what the software would perceive, but not how the world would respond to different decisions. Closed loop scenario replay keeps the whole system in the loop, which is what you need to test behavior.

Darkstar is an early stage simulator project designed around deterministic replay of adversarial scenarios on real terrain, and drone and swarm teams who want to help decide what it tests first can learn more on its design partner page.

Questions