Autonomous threat hunting proof of concept: a two-week plan

In one paragraph

A trial of an autonomous hunting system should answer three questions: what it does without being asked, whether its results hold up, and what it costs in review time and query compute. Two weeks is enough if the setup is done before the clock starts. This plan splits the work into preparation, a week of observation, a week of deliberate stress tests and a written close-out, with the checks to run at each step.

By Autonomous Threat Hunting editors · 28 September 2026

Why a plan, and why two weeks?

Vendor demos show a system on the vendor's data. A trial shows it on yours, which is the only place its claims can be checked. Without a plan, trials drift into a longer demo. Two weeks gives one week to see normal behavior and one week to test the edges, and it is short enough that people stay engaged. If the data platforms bill monthly, extend the trial so that a full billing cycle is covered, as the risks page recommends.

Before day one: set up

Agree the scope in writing: which data sources, which time ranges, which intelligence feeds, and what the system must never touch.
Create a dedicated service identity on each data platform with read-only, least-privilege access, and turn on the platform's own query logging for it.
Pick one past incident from your own records whose data is still within retention, and write down what a correct finding would contain. This is your planted test.
Agree where findings, proposed detections and staged actions will go, and who reviews each kind.
Record the approval settings as configured on day one, with a screenshot.
Ask the vendor which rung of the autonomy ladder the trial configuration represents, per task.

Week one: observe

Change nothing. Record what happens.

For every hunt: who or what started it, which intelligence it came from, which sources it read and where it stopped.
For every item that reaches a person: the time from arrival to decision, and the decision.
For every finding: whether the report states the hypothesis, the exact queries, the sources and time range, and what was not checked. The lesson on evaluating a hunt report has the six checks.
Daily: compare the data platforms' query logs for the service identity with the vendor's list of queries run.

At the end of the week, count items per day and average review time, and enter them in the review-time calculator.

Week two: stress

Now test the edges, one at a time, and note the date of each test so results can be traced.

Planted test: check whether the system surfaced the past incident, and whether its finding matches what you wrote down. If it needs prompting, note how much.
Missing source: disable one source for a day. Do findings from that day say the source was unavailable, or do they read as clean?
Approval expiry: leave one approval unanswered past its time limit and confirm nothing ran.
Permission change: try to change an approval setting from an account that should not be able to, and check it is refused and logged.
Promotion: take one proposed detection through backtest, review and deployment to a test target, then roll it back.
Export: export hunts and detections to your own repository, in a portable format such as Sigma where the vendor supports it.

Close-out: write it up

Write one page, with the evidence attached. Keep it to facts from the trial.

Rung per task, as observed, next to the rung the vendor described.
Items per day and review hours per week, measured.
Query volume from the platforms' own logs, and any gap with the vendor's report.
Results of each week-two test: passed, failed or not tested, with the reason.
Open questions for the vendor, especially anything the checklist asks that the trial did not answer.

Running more than one trial

If you are trialling more than one system, run the same plan with the same planted test and the same scope, one after the other or side by side with separate service identities. The side-by-side view shows what each vendor publishes before you start, so you can decide which claims the trial must check.

Frequently asked questions

Can a trial be shorter than two weeks?

It can, but a week of observation before any stress test is what shows normal behavior. Shortening the observation week makes the review-time numbers less reliable.

What if the system misses the planted test?

Record whether it had access to the needed sources and time range. A miss with full access is a result about the system; a miss without access is a result about scope.

Sources

MITRE ATT&CK, Enterprise tactics attack.mitre.org
SigmaHQ, Sigma repository github.com