Run an attack simulation from setup to report
This walkthrough connects Operator and its managed Attack Harness to the mock customer-support agent in Interceptor. You will prepare the agent's world, verify a normal interaction, submit your investigation objectives, execute a bounded campaign, and review its retained evidence.
The attack simulation workflow also supports operational simulations that evaluate functional effectiveness. Keep security objectives for adversarial investigations; choose task completion, consistency, or reliability objectives for operational evaluation. Both use the preparation, execution, and evidence steps below.
For an assistant-led setup, use the quick start. This guide makes the individual steps and their outputs explicit.
Install the components and skills
Use a supported Linux/macOS host and native AMD64/ARM64 Docker runtime. Select compatible packages and images from the release catalog and follow the documentation for each installed component. Complete:
- Attack Harness image setup.
- Operator package and model setup.
- Interceptor package and provider setup.
- Skill installation, including results reporting.
Choose a tool-capable chat model for the target and one for Attack Harness. This
example uses OpenAI-compatible chat and text-embedding-3-large for target retrieval.
Configure host credentials through supported environment or secret-store settings.
The examples name TARGET_MODEL_KEY, TARGET_EMBEDDING_KEY, and CAMPAIGN_MODEL_KEY;
populate their values securely, not in these files or command arguments. Model and
embedding calls can incur provider charges, including during preparation.
On Linux, execute Operator commands as its configured account and make its inputs privately accessible to that account. Use literal absolute paths in host configuration.
Build the customer-support environment
Follow the complete environment recipe. It creates
the support-review workspace, pins the target source, builds the application image,
loads the application's migrations and grants, generates a real document embedding,
and supplies a blueprint and invocation adapter.
The database is Interceptor-managed; no separate PostgreSQL/pgvector service is installed. The recipe uses persistent native logs and disables snapshots because its application conversation lives in memory.
Return to support-review for the remaining commands. Keep a preparation record
with the source revisions, image IDs, blueprint lock, provider/model choices, and
the actual review metadata.
Start and verify the target
Choose a private Interceptor data directory and use it in every Interceptor terminal:
umask 077
mkdir -p interceptor-data
export INTERCEPTOR_DATA_DIR="$PWD/interceptor-data"
With the blueprint's model name matching YOUR_TARGET_MODEL, start the host:
interceptor run --environment ./interceptor-environment \
--model-provider openai-chat --provider-base-url https://api.openai.com/v1 \
--provider-auth api-key-env --model-api-key-env TARGET_MODEL_KEY \
--allow-model YOUR_TARGET_MODEL \
--embeddings-provider openai-compatible --embeddings-base-url https://api.openai.com/v1 \
--embeddings-auth api-key-env --embeddings-api-key-env TARGET_EMBEDDING_KEY \
--allow-embedding-model text-embedding-3-large
Keep this terminal open. Record the session ID and printed application URL. From another terminal, send a normal request to that URL, not the separate Operator API at port 8080:
curl --fail-with-body -H 'Content-Type: application/json' \
--data '{"message":"What is your return policy, and what is the status of order-456?"}' \
http://PRINTED_APPLICATION_HOST_AND_PORT/operations/invoke
Check that the response is meaningful, the customer/order relationship is correct, and the policy comes from populated document retrieval. Inspect retained evidence for database and embedding activity and collect the native log using the session guide. An HTTP 200 or empty answer alone does not pass this check.
Stop this verification session, end its owning host with Ctrl+C, and rerun the same start command to obtain a fresh session from sealed seeds. Record its new session ID. Reserve that host instance for the real campaign. Interceptor binds to the first attached campaign for the host process lifetime; avoid a throwaway attachment.
Configure Operator for this target
Save the following as operator-environment/target-profile.json. This example
permits message submission and black-box output, with no injection routes or
snapshot requirement:
{
"api_version": "operator.dev/target-profile/v1alpha1",
"id": "support-local",
"target_id": "support-review",
"adapter": "interceptor/v1",
"allow_target_stop": false,
"scopes": {
"operation_ids": ["invoke"],
"caller_principal_ids": [],
"routes": [],
"allow_retained_injections": false
},
"feedback": {
"ceiling": "black-box",
"allowed_kinds": ["target_output"],
"max_attempt_bytes": 1048576
},
"operation_timeout_ms": 180000,
"interceptor": {"protocol": "http", "address": "127.0.0.1", "port": 8080}
}
If your objectives require injections, configure only appropriate routes from the actual exported capabilities and the release's target-profile contract. Do not add technique-specific permissions by copying an unrelated example. Empty routes are an intentional starting permission boundary, not a claim that injections are unavailable.
In Operator's existing private configuration, set the model.profile_file prepared
during setup and merge these timeout values into the existing sections:
target:
max_operation_timeout_ms: 180000
operation_timeouts_ms:
engine.attempt_execute: 180000
Keep the existing image, contract, state, Docker, and model settings. This fragment is not a replacement configuration. The complete-operation allowance includes more than the target's turn timeout; remaining campaign deadlines can still shorten it.
Set OPERATOR_CONFIG in the shell to the absolute path to that config. Save this
complete example model profile at the configured model.profile_file path,
replacing the model identifier with one available to your account:
{
"api_version": "operator.dev/model-provider-profile/v1alpha1",
"id": "campaign-model",
"provider": "openai-chat",
"codec_id": "openai-chat-text-tools-v1",
"model": "YOUR_CAMPAIGN_MODEL",
"endpoint": "https://api.openai.com/v1/chat/completions",
"authentication": "api-key-env",
"api_key_env": "CAMPAIGN_MODEL_KEY",
"maximum_prompt_tokens": 32768,
"maximum_completion_tokens": 4096,
"maximum_response_bytes": 1048576,
"codec_options": {"instruction_role": "developer"}
}
Keep the profile private. Confirm the selected model supports these allowances; adjust them deliberately rather than raising budgets automatically.
chmod 600 operator-environment/target-profile.json
operatorctl config check --config "$OPERATOR_CONFIG"
operatorctl doctor --config "$OPERATOR_CONFIG"
interceptor capabilities --environment ./interceptor-environment > ./native-capabilities.json
operatorctl capabilities export --config "$OPERATOR_CONFIG" \
--target-profile "$PWD/operator-environment/target-profile.json" \
--native ./native-capabilities.json --output ./operator-environment/capabilities.json
Use new export paths. Keep capabilities.json and its generated hash-named companion
together. Exporting is offline and does not attach a campaign or grant permissions.
Create the bundle from your objectives
Use the objectives.md created in the Quick start,
or write your own objectives in that file. Include the scope, evidence you want to
inspect, and limitations you accept. The example below places the file's text
in one required objective; use the bundle guide
or scenario-generation skill if you want separately identified objectives/scenarios.
For an attack simulation, state the security question and the agent-specific exposure you want to investigate. For an operational simulation, state the intended task, the cases to exercise, and what evidence would establish correct completion. For a combined campaign, keep the security and functional criteria distinct.
If you choose the quick start's order-status objective,
copy that objective into objectives.md. The environment recipe seeds
order-456 for customer-123 with status delivered; this is the expected fact
against which to review answers. Ask for the three cases explicitly, and account
for any that the campaign does not complete. These are requested observations,
not a built-in benchmark or a guarantee that the harness will finish every case.
This example permits normal message submission without injection routes. Security objectives that require indirect injection need the appropriate target permissions described above. The support adapter retains conversation state and has snapshots disabled, so repeated requests in one session are not independent fresh trials. Record that context when assessing consistency; use separately prepared fresh sessions if your chosen comparison requires independent starting conditions.
Save preparation/make-bundle.py:
import json
from pathlib import Path
cap = json.loads(Path("operator-environment/capabilities.json").read_text())
goal = Path("objectives.md").read_text().strip()
if not goal:
raise SystemExit("Supply your investigation objective in objectives.md")
bundle = {
"schema_version": "operator.dev/scenario-bundle/v1alpha1",
"bundle_id": "support-review-001",
"producer": {"kind": "user"},
"target_requirements": {
"target_id": cap["capabilities"]["target_id"],
"capability_source_digest": cap["source"]["capability_source_digest"],
},
"objectives": [{"objective_id": "user-objective", "description": goal, "required": True}],
"scenarios": [],
"feedback": {"requested_profile": "black-box"},
"evidence": {"requested_classes": [], "limitations": ["Black-box target feedback; snapshots are disabled."]},
"artifacts": [],
"coverage": {
"objective_refs": ["user-objective"],
"completion_guidance": "Account for the objective using recorded evidence and state any unresolved work.",
},
"requested_limits": {"attempt_admissions": 10, "active_seconds": 300, "model_tokens": 200000},
}
Path("scenario-bundle.json").write_text(json.dumps(bundle, indent=2) + "\n")
The requested limits are explicit example ceilings, not a recommended campaign budget. Change them to your chosen limits within the installed bounds before submitting. They do not cap the target provider's charges in currency.
python3 preparation/make-bundle.py
operatorctl submit --config "$OPERATOR_CONFIG" \
--bundle ./scenario-bundle.json --environment ./operator-environment \
--output ./runs/support-review
operatorctl validate --config "$OPERATOR_CONFIG" --run ./runs/support-review
Expect a validated-offline receipt. Resolve required-capability or policy failures
without silently dropping requirements. No campaign has run at this point. Preserve
the submission and create a new one if you change its source inputs or profile.
Run and observe
With the fresh Interceptor host still running:
operatorctl campaign start --config "$OPERATOR_CONFIG" --run ./runs/support-review
Operator performs live admission, launches its network-isolated Attack Harness, and mediates its model and target calls. Keep the foreground terminal open through finalization. From another terminal:
operatorctl inspect --config "$OPERATOR_CONFIG" --run ./runs/support-review
If you need to end the campaign, use the actual campaign ID from inspection:
operatorctl campaign terminate --config "$OPERATOR_CONFIG" --campaign CAMPAIGN_ID
A limit, failure, or uncertain operation can end work before the objective is resolved. Do not automatically replay the campaign. Inspect the retained state and account for incomplete work in the report.
Inspect, export, and generate a report
After execution closes:
operatorctl report --config "$OPERATOR_CONFIG" --run ./runs/support-review
operatorctl export --config "$OPERATOR_CONFIG" \
--run ./runs/support-review --output ./exports/support-review
interceptor logs verify SESSION_ID
interceptor logs export SESSION_ID --output ./exports/interceptor-evidence.tar
Use the actual campaign target session ID and new destinations. Collect native application logs separately while the session container is available. Verify that the export contains the accepted bundle, configuration context, activity, and available evidence. Do not interpret a model assessment as an independently proven fact.
Ask your skills-enabled assistant:
Use attack-simulation-results-reporting to generate a portable HTML review from
./exports/support-review, with ./exports/interceptor-evidence.tar and collected
native logs as additional evidence where relevant. Write to ./reports/support-review.
Explain what was tested, cite evidence, distinguish observations from conclusions,
and account for incomplete or untested work. Do not run additional experiments.
For supplied security objectives, explain the observed risks and limitations.
For supplied functional objectives, compare actual task outcomes with the stated
criteria and report correct, incorrect, incomplete, failed, and untested cases separately.
Explain which modeled conditions inform expectations about production behavior
and which differences or missing evidence limit those conclusions.
Open the returned HTML entry point and check evidence links and coverage. Keep its companion files when sharing it. The authored review complements Operator's native report; it does not replace or modify retained artifacts.
For the optional order-status objective, check that the report accounts for all
three requested cases, compares answers with the seeded delivered status, and
identifies shared conversation state. A completed campaign or successful HTTP
response alone does not establish task correctness or reliability in production.
Finish and retain the record
Confirm Operator has closed execution and cleanup, collect required files, then
stop the target with interceptor stop SESSION_ID. End its owning terminal with
Ctrl+C when finished with the API. Keep the source pins, blueprint, accepted bundle,
receipts, exports, and report. Purge is a separate deliberate deletion step.
If a step fails, use troubleshooting and retain diagnostics. An honest partial report is more useful than claiming readiness, compatibility, or a security result that the available evidence does not establish.