Preparing autonomy for evaluations: build evidence, not demos

What evaluators are actually asking

Different evaluators use different words, but the questions are similar. What is the system supposed to do, and under what conditions? How was it tested? Where does performance hold up and where does it fall off? What happens when things go wrong? Can I check your results myself? In US defense test and evaluation, the stated aim is to characterize where a system performs well and where deficiencies exist, so they can be fixed before fielding. Commercial customers ask the same thing in plainer language.

None of those questions is answered by a successful run. They are answered by a body of evidence that grows over time and is organized around the system's intended use.

Define the operating envelope first

Evidence only means something relative to the conditions it covers. Before planning tests, write down the operating envelope, sometimes called the operational design domain: the missions, environments, terrain, weather, communication conditions, threats, and speeds the system is meant to handle. Be equally clear about what is out of scope. A system that is honest about its envelope is easier to evaluate than one that claims to handle everything.

Write the test plan before you test

A test plan makes evidence credible because it fixes the questions and the criteria before the results are known. A practical plan covers:

• The questions the testing must answer, traced to requirements and hazards
• The measures of performance and safety, and how each is computed
• Pass and fail criteria, set in advance
• The scenarios that will exercise each part of the envelope
• Which method answers which question: analysis, simulation, hardware in the loop, range tests, or field trials
• The order of testing, so early results shape later phases
• What data will be recorded and how configurations will be controlled

Treat testing as a sequence

Testing works best as a sequence rather than a single event. Each phase refines the next, and simulation sweeps point the expensive live tests at the conditions where the outcome is least certain. Plans should expect to change as results come in, with every change recorded and the reason stated.

Show scenario coverage you can explain

No one can test every possible situation, and evaluators know it. Defense researchers have argued for years that brute force coverage of an autonomous system's full operating space is not achievable, which is why evidence has to be argued, not just accumulated. Coverage becomes persuasive when you can show it along clear dimensions such as environment, terrain, mission phase, threat, and failure mode. Show what is covered and what is not.

Include degraded and adversarial cases such as communication loss, GNSS denial or spoofing, sensor faults, and reactive opponents, because those are the cases evaluators will probe. Concentrate effort near the boundaries between success and failure, where each test teaches the most.

Keep replay logs and full traceability

Every result you present should be reproducible. For simulation, that means storing the scenario definition, random seeds, software and simulator versions, configuration, and the terrain or map used, so the exact run can be repeated, ideally in front of the evaluator. For field tests, keep raw sensor logs, telemetry, ground truth where available, weather, and test conductor notes. When something surprising happens in the field, rebuild it as a scenario so it joins the regression suite. Deterministic replay is what makes this work: if the same scenario gives the same result every time, an evaluator can verify a claim instead of trusting it.

Tie it together with an assurance argument

Many safety and assurance frameworks organize evidence as an assurance case: a set of claims about the system, the argument for why they hold, and the evidence behind each step. UL 4600, a standard for the safety evaluation of autonomous products, takes this goal based approach rather than prescribing specific engineering methods. A good case also records defeaters, the reasons a claim might be wrong, and how each was addressed. Evidence usually comes from several sources at once, including analysis, simulation, hardware in the loop testing, range testing, and field use, because no single method is sufficient for autonomy.

Simulation evidence carries weight only when the simulator's credibility is established. Expect to show how models were checked against live data and which effects they leave out. For autonomous weapon systems, DoD Directive 3000.09 calls for rigorous verification and validation, realistic developmental and operational testing, and repeated test and evaluation after changes to the system's operating state, including changes from machine learning.

Mistakes that undermine evidence

Even strong engineering teams weaken their own case in predictable ways:

• Presenting only successful runs
• Changing pass criteria after seeing the results
• Results that cannot be traced to a specific software version
• Simulation results with no evidence that the simulator matches reality
• No regression testing after updates
• Hiding known limitations instead of documenting them

Be honest about limits

The last mistake matters most. Evaluators expect autonomy to have limits. A team that documents its failures, explains the root cause, shows the fix, and proves with replay that the failure stays fixed earns more trust than a team that only shows wins.

If you are building drone or swarm autonomy and want to help shape an early stage simulator project for replaying adversarial scenarios on real terrain, the Darkstar design partner page explains how to take part.

Questions