CI Integration

Whether you use the GitHub Action or run the CLI directly, agentiqa run exposes a governed, versioned contract for CI: a fixed exit-code model and a machine-readable JSON envelope.

To choose which plans a pipeline runs and how they execute — a labeled subset, sequential or parallel — see Labels and Parallel runs. This page covers how to gate on and parse whatever you run.

Exit codes

The CLI exits with a status that gates your job automatically. It's a fixed, governed contract, so you can rely on it:

The 1 vs 3 split is deliberate and stable: 1 is a genuine test failure you should investigate; 3 is a transient/infra problem that is safe to retry. A gate can retry on 3 without masking regressions:

- name: Run test plans (retry only on infra/runtime errors)
  run: |
    for attempt in 1 2 3; do
      npx -y agentiqa@latest run --engine https://engine.agentiqa.com
      code=$?
      # 0 = pass, 1 = plan failure (do NOT retry), 2 = usage error (do NOT retry)
      [ "$code" -ne 3 ] && exit "$code"
      echo "Infra/runtime error (exit 3) on attempt $attempt — retrying…"
      sleep 15
    done
    exit 3

JSON envelope

Add --json (or set AG_OUTPUT=json) to emit exactly one JSON document on stdout. Every document carries "schemaVersion": 1 at the top level; a breaking change to the shape bumps that number.

• All logs go to stderr. stdout is only the JSON document — no banners, no progress, no ANSI color. stdout | jq is always safe.
• The exit code is independent of the envelope — always branch on the exit code for pass/fail; use the JSON for detail.

Success (ok: true):

{
  "ok": true,
  "schemaVersion": 1,
  "outcome": "passed",
  "plans": [
    {
      "title": "Checkout flow",
      "outcome": "passed",
      "durationSec": 42,
      "exitCode": 0,
      "runUrl": "https://web.agentiqa.com/projects/proj_…/test-plans-v2/tp_…/history/run_…",
      "videoUrl": "https://assets.agentiqa.com/e2e-videos/run-…/checkout-flow.mp4"
    },
    { "title": "Login", "outcome": "failed", "durationSec": 18, "exitCode": 1, "summary": "…" }
  ]
}

outcome is "passed" only when every plan passed, otherwise "failed". Each plans[] entry carries a per-plan outcome, durationSec, exitCode, and an optional summary. When artifacts are captured, an entry may also carry runUrl, videoUrl, videoPath, artifactDir, and (with --share) a public shareUrl — each present only when available (omitted, never null, so schemaVersion stays 1).

Failure (ok: false) — emitted for usage errors and thrown infra/runtime errors:

{ "ok": false, "schemaVersion": 1, "error": { "code": "run_error", "message": "…" } }

Gate on the exit code, extract detail with jq

- name: Run test plans (JSON)
  run: |
    npx -y agentiqa@latest run --engine https://engine.agentiqa.com --json > result.json
    code=$?
    jq -r '.plans[] | "\(.outcome)\t\(.title)"' result.json || cat result.json
    exit "$code"

Because logs are on stderr, > result.json captures only the envelope; the human-readable run log still streams to the console.

Action outcome buckets

The GitHub Action maps these exits to outcome buckets for gating: 0 (with a valid envelope and ≥1 plan) → passed; 0 with zero plans or no valid envelope → config-error; 1 → plan-failure; 2/unknown → config-error; 3 → infra-error. fail-on: plan-failure (default) fails on plan failures and config errors and swallows retryable infra errors; fail-on: any gates on anything nonzero.