RunTrail SDK setup guide

Start with the repository and a project key

RunTrail receives results from tests that run locally or in your existing CI. A coding assistant needs access to the consumer repository, a terminal, this guide, the RunTrail instance URL, and a valid API key with ingest permission. Grid is optional.

Create a project and its key in RunTrail's SDK Setup. Keys created there are limited to the chosen project. Give the assistant the instructions from Copy instructions for my AI, and supply the key separately as QA_API_KEY in the test process environment.

Inspect package.json, pyproject.toml, pom.xml, build.gradle, lockfiles, runner configuration, and test scripts. Choose the adapter that matches the existing framework. Merge configuration into the existing file and preserve current reporters, hooks, settings, and test behavior.

This guide and its Markdown version are public and contain no project credentials. Listing or creating organization projects and changing account settings still requires user authentication; an SDK key supplies integration context and delivery receipts only.

Understand required and optional changes

Required for reporting: install an available adapter, declare dependencies and update the existing lockfile, activate its reporter/plugin/formatter/listener, and supply QA_API_KEY, QA_PROJECT_SLUG, and the chosen QA_API_BASE_URL to the test process. Pytest and Java listeners can auto-load after installation; the other adapters need explicit registration. npm install alone cannot configure a runner that requires registration.

Required only when applicable: merge each independently selected test configuration; load environment files if that is how the project supplies credentials; reproduce source dependency paths/builds in CI; and include local package files in Docker's dependency-install layer when tests run in an image. Preserve the project's current scripts, hooks, reporters, fixture imports, and test exit status.

Optional choices: registry versus source checkout versus versioned tarballs/wheels, conditional reporting when no key is present, extra helper files to share configuration, CI rollout, Timeline, failure evidence, BDD synchronization, and quality gates. A helper, vendor directory, or Docker edit must have a concrete purpose in that repository; do not create them for every integration.

Before editing, explain which files need to change and why. Afterward, distinguish mandatory SDK registration from the selected packaging strategy and optional features. Never replace existing tests with generated smoke tests as proof of integration.

Set the instance URL and credential

QA_API_BASE_URL is the root URL of your RunTrail instance, without /api/v1, credentials, a query, or a fragment. It is not the URL of the application under test. Use the URL supplied by SDK Setup; the example hostname below is a placeholder.

Set QA_API_KEY in the same environment that starts the test runner. Keep real keys in local environment variables or CI secrets. Use .env.example for committed placeholders. SDKs do not load .env files automatically: use the project's existing environment loader, shell exports, or CI secret injection.

export QA_API_BASE_URL="https://your-runtrail-instance.example"
export QA_PROJECT_SLUG="your-project-slug"

$env:QA_API_BASE_URL = "https://your-runtrail-instance.example"
$env:QA_PROJECT_SLUG = "your-project-slug"

Discover the project using the key

GET /api/v1/sdk/setup uses X-API-Key and requires ingest permission. If exactly one project is accessible, no project slug is needed. Read environment.QA_PROJECT_SLUG from the response and set it in the test runner's environment. The SDK itself still requires this value; the assistant resolves it during setup.

If more than one project is accessible, the endpoint returns HTTP 409 with detail.code=project_selection_required. Repeat the request with ?project_slug=<chosen-slug>; never guess the destination. HTTP 404 means no matching authorized project exists. An expired, revoked, or invalid key returns HTTP 401; a key without ingest permission returns HTTP 403.

The response contains version, project (id, name, slug), permissions, environment, and relative endpoint paths. Keep the API base URL you called; relative paths also work when the instance is behind a reverse proxy. Do not send the key to another origin or follow credentialed redirects.

curl --fail-with-body --silent --show-error \
  -H "X-API-Key: $QA_API_KEY" \
  "$QA_API_BASE_URL/api/v1/sdk/setup"

$context = Invoke-RestMethod -Uri "$env:QA_API_BASE_URL/api/v1/sdk/setup" -Headers @{ "X-API-Key" = $env:QA_API_KEY }
$env:QA_PROJECT_SLUG = $context.environment.QA_PROJECT_SLUG

Choose an available package source

SDK source repository: https://github.com/legnadev/qa-orchestrator.git. Each framework below has a source installation and a registry installation reference. Registry commands require that the named package/version is actually published and accessible; this guide does not assert publication. If neither the package registry nor the repository is accessible, report the prerequisite instead of inventing a package or URL.

Reuse an existing SDK checkout rather than cloning over it. Source examples assume .qa-orchestrator-sdk is inside the directory containing the consumer's test configuration. Build the shared core before a Node adapter. Ignore the SDK checkout, .qa-orchestrator offline buffers, and .qa-timeline output in the consumer's Git repository.

Use the consumer's existing package manager, lockfile, virtual environment, and dependency declarations. The examples use npm, pip, and Maven; adapt to pnpm, yarn, uv, or Gradle as appropriate. Python pip commands must run inside the consumer's virtual environment. Projects managed by uv should record dependencies with uv add --dev and use uv run for tests; temporary uv pip installs alone do not make CI reproducible.

Local file dependencies require the same SDK paths/build steps in CI. Pin an audited SDK commit and record it in integration documentation. A private SDK repository needs Git access in addition to the RunTrail ingest key; QA_API_KEY cannot authenticate GitHub or a package registry. Versioned package artifacts are an optional alternative to recreating a source checkout.

Node adapters are ESM packages. Preserve the consumer's module format and let its runner load the adapter as documented below. The SDK checkout's npm workspace build requires Node.js >= 20 even though several built packages advertise a Node.js >= 18 runtime minimum. The installed runner may require a newer Node version.

Make source installs reproducible in CI and Docker

Choose one strategy: recreate the pinned SDK checkout at the same relative path before installing consumer dependencies, or build and commit/package versioned artifacts. Registry installation removes this extra source preparation only when the exact package and its core dependencies have been published. Tarballs are not an SDK requirement.

For npm tarballs, build the core and adapter first, pack both into a consumer-owned directory, install both in the same command, and commit that directory with package.json and package-lock.json. Record the audited source revision and rebuild both packages together when upgrading. The Playwright recipe below adapts to other Node adapters by replacing the adapter name/path.

For Docker images using those tarballs, COPY the vendor directory before RUN npm ci whenever package.json uses file:vendor/... dependencies. Update only Dockerfiles that install those dependencies. Ensure .dockerignore includes the package files and the build context contains them. A directory checkout instead needs all source/package files and built dist at the declared paths before npm ci; copying the consumer afterward is too late.

For pip, record both editable paths in the project's existing requirements/dependency declaration and recreate them in CI; alternatively build both wheels and install them together. For uv, uv add --dev --editable records paths and updates uv.lock, then uv sync --locked reproduces the install. An absolute path from one developer's drive is not portable.

For Java, mvn install populates only the current runner's local Maven cache. Rebuild the pinned SDK on every clean runner before the consumer test command, or publish to an accessible artifact repository. A developer's mavenLocal() cache is not available in GitHub Actions by default.

# Reuse the accessible checkout and pin its audited commit before building.
npm ci --prefix .qa-orchestrator-sdk
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-playwright
mkdir -p vendor/runtrail
npm pack --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core --pack-destination "$PWD/vendor/runtrail"
npm pack --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-playwright --pack-destination "$PWD/vendor/runtrail"
# Filenames below correspond to SDK version 0.1.0; use the actual pack output.
npm install --save-dev file:vendor/runtrail/qantum-labs-runtrail-core-0.1.0.tgz file:vendor/runtrail/qantum-labs-runtrail-playwright-0.1.0.tgz

# Merge into an existing image; keep its current base image and test command.
WORKDIR /app
COPY package.json package-lock.json ./
COPY vendor/runtrail/ ./vendor/runtrail/
RUN npm ci
COPY . .
# Supply QA_* at test runtime through the CI environment, not image build arguments.

# Keep the existing dependencies. Paths assume the same checkout layout in CI.
-e .qa-orchestrator-sdk/sdk/python-core
-e .qa-orchestrator-sdk/sdk/pytest

Playwright

Reporter captures test results, annotations, traces, screenshots, video, and HAR.

Merge the configuration reference into playwright.config.ts. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Node.js >= 18 and @playwright/test >= 1.40; also satisfy the installed Playwright version's Node requirement. Register the reporter by its package name in a reporter tuple. Installing the package alone does not enable reporting.

Required: append to the existing reporter array. A string such as reporter: 'list' must first become [['list']]. Keep projects, testDir, use, retries, webServer, globalSetup, and globalTeardown. Inspect every config selected by --config in npm scripts; an audit or chat suite with its own config does not inherit the main reporter.

Optional: enable reporting only when QA_API_KEY exists, as in the merge example. A missing key then leaves tests usable and delivery verification pending. Existing browser installation remains the project's responsibility; the reporter does not install browsers or start the application.

Optional: uploadArtifacts, Timeline, BDD fixtures, and @qantum-labs/runtrail-playwright/test are separate features. Result reporting needs no test-import changes. If opting into failure capture, extend the project's existing fixture rather than losing its custom fixtures.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
npm ci --prefix .qa-orchestrator-sdk
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-playwright
npm install --save-dev file:.qa-orchestrator-sdk/sdk/core file:.qa-orchestrator-sdk/sdk/playwright

npm install --save-dev @qantum-labs/runtrail-playwright

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  reporter: [
    ["html"],
    ["@qantum-labs/runtrail-playwright", {
      apiKey: process.env.QA_API_KEY!,
      projectSlug: process.env.QA_PROJECT_SLUG ?? "your-project-slug",
      apiBaseUrl: process.env.QA_API_BASE_URL ?? "https://your-runtrail-instance.example",
      uploadArtifacts: false,
    }],
  ],
});

import { defineConfig, type ReporterDescription } from "@playwright/test";

// These represent the project's existing settings. Retain its actual values.
const existingConfig = defineConfig({
  testDir: "./tests",
  reporter: [["list"], ["html", { open: "never" }]],
  retries: 1,
});
const existingReporters: ReporterDescription[] = typeof existingConfig.reporter === "string"
  ? [[existingConfig.reporter]]
  : existingConfig.reporter ?? [["list"]];
const runtrailReporters: ReporterDescription[] = process.env.QA_API_KEY ? [[
  "@qantum-labs/runtrail-playwright",
  {
    apiKey: process.env.QA_API_KEY,
    projectSlug: process.env.QA_PROJECT_SLUG!,
    apiBaseUrl: process.env.QA_API_BASE_URL,
    uploadArtifacts: false,
  },
]] : [];

export default defineConfig({
  ...existingConfig,
  reporter: [...existingReporters, ...runtrailReporters],
});

npx playwright test

Vitest

Reporter captures Vitest task results and buffers offline failures.

Merge the configuration reference into vitest.config.ts. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Node.js >= 18, Vitest >= 2 and < 6, and the Node version required by your installed Vitest. The SDK supports legacy Vitest 2 callbacks and Vitest 3–5 hooks. Register a new QAOrchestratorReporter instance under test.reporters.

Required for the current ESM SDK: the Vitest config must load as ESM. In a CommonJS project, use vitest.config.mts (or .mjs) instead of importing this package from a bundled CommonJS .ts config. In an existing type: module project, keep its .ts config. Update --config references only when the filename changes; do not change the entire application's module type just for the reporter.

Required: preserve Vite plugins, aliases, coverage, setupFiles, environment, workspaces/projects, and existing reporters. When reporters is absent, append to ['default'] so normal console output survives. Normalize a single reporter string before spreading it. Register once for each actual runner execution; do not add the same reporter to both a root config and inherited projects.

Required for a bounded verification: run the existing command in run mode (for example npm test -- --run) if its usual script starts watch mode. Watch reruns report separate executions; select the receipt for the execution being verified.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
npm ci --prefix .qa-orchestrator-sdk
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-vitest
npm install --save-dev file:.qa-orchestrator-sdk/sdk/core file:.qa-orchestrator-sdk/sdk/vitest

npm install --save-dev @qantum-labs/runtrail-vitest

// vitest.config.mts (or vitest.config.ts in an existing ESM project)
import { defineConfig } from "vitest/config";
import { QAOrchestratorReporter } from "@qantum-labs/runtrail-vitest";

export default defineConfig({
  test: {
    reporters: [
      "default",
      new QAOrchestratorReporter({
        apiKey: process.env.QA_API_KEY!,
        projectSlug: process.env.QA_PROJECT_SLUG ?? "your-project-slug",
        apiBaseUrl: process.env.QA_API_BASE_URL ?? "https://your-runtrail-instance.example",
      }),
    ],
  },
});

// vitest.config.mts (or vitest.config.ts in an existing ESM project)
import { defineConfig } from "vitest/config";
import { QAOrchestratorReporter } from "@qantum-labs/runtrail-vitest";

const existingConfig = defineConfig({
  test: { environment: "node", reporters: ["default"] },
});
const configuredReporters = existingConfig.test?.reporters ?? ["default"];
const existingReporters = typeof configuredReporters === "string"
  ? [configuredReporters] : configuredReporters;

export default defineConfig({
  ...existingConfig,
  test: {
    ...existingConfig.test,
    reporters: [
      ...existingReporters,
      ...(process.env.QA_API_KEY ? [new QAOrchestratorReporter({
        apiKey: process.env.QA_API_KEY,
        projectSlug: process.env.QA_PROJECT_SLUG!,
        apiBaseUrl: process.env.QA_API_BASE_URL,
      })] : []),
    ],
  },
});

npm test

Jest

Reporter uses Jest reporter options and extracts requirement IDs from titles.

Merge the configuration reference into jest.config.ts. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Node.js >= 18 and Jest >= 29, subject to your Jest version's Node requirement. Add the reporter tuple to reporters and keep 'default' when no reporters are configured. Keep transform, testEnvironment, setupFiles, setupFilesAfterEnv, projects, coverage, and custom reporters.

Required for the current import-only ESM package: create runtrail-jest-reporter.mjs with the re-export below, then reference its file path in reporters. Jest 29's reporter resolver does not resolve the package's import-only export by name. The local .mjs entry lets Jest load it with dynamic import or Node's ESM interop. This extra file is needed by this package build, not by every Jest reporter.

Required: retain the current config module format. The example keeps jest.config.cjs and never require()s the ESM SDK directly. A TypeScript config additionally needs the loader already used by that repository; switching to .ts is optional.

Required: check command-line --reporters and per-project configs, which can override configuration. Use a finite test run, and avoid --forceExit while reporting is still in flight.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
npm ci --prefix .qa-orchestrator-sdk
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-jest
npm install --save-dev file:.qa-orchestrator-sdk/sdk/core file:.qa-orchestrator-sdk/sdk/jest

npm install --save-dev @qantum-labs/runtrail-jest

// jest.config.ts
// First create runtrail-jest-reporter.mjs: export { default } from "@qantum-labs/runtrail-jest";
import type { Config } from "jest";

export const config: Config = {
  reporters: [
    "default",
    [
      "<rootDir>/runtrail-jest-reporter.mjs",
      {
        apiKey: process.env.QA_API_KEY!,
        projectSlug: process.env.QA_PROJECT_SLUG ?? "your-project-slug",
        apiBaseUrl: process.env.QA_API_BASE_URL ?? "https://your-runtrail-instance.example",
      },
    ],
  ],
};

export default config;

export { default } from "@qantum-labs/runtrail-jest";

// jest.config.cjs
const existingConfig = {
  testEnvironment: "node",
  reporters: ["default"],
};
module.exports = {
  ...existingConfig,
  reporters: [
    ...(existingConfig.reporters ?? ["default"]),
    ...(process.env.QA_API_KEY ? [["<rootDir>/runtrail-jest-reporter.mjs", {
      apiKey: process.env.QA_API_KEY,
      projectSlug: process.env.QA_PROJECT_SLUG,
      apiBaseUrl: process.env.QA_API_BASE_URL,
    }]] : []),
  ],
};

npx jest

Cypress

Plugin registers Cypress node events and captures screenshots and video refs.

Merge the configuration reference into cypress.config.ts. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Node.js >= 18 and Cypress >= 12, subject to Cypress's own runtime requirements. Register in the Node-side setupNodeEvents for the testing type you run (e2e or component). Installation alone is insufficient. Keep the existing reporter and browser-side support file.

Required: await the existing setupNodeEvents implementation and return its updated config. The SDK registers before:run, after:spec, and after:run. Compose handlers for those events when another plugin uses them, returning/awaiting all promises so upload work completes. Pass the composed on function to both existing plugins and RunTrail; registering a second handler directly can replace a previous handler.

Required: read QA_API_KEY from process.env only in the Node config. Do not place it in config.env, Cypress.env, cypress.env.json, or browser-side support code. Use cypress run for initial delivery verification; interactive open mode has different run-event behavior.

The merge example uses an ESM cypress.config.mjs. For an existing CommonJS config, import the SDK with await import() inside async setupNodeEvents; retain that project's module format. The lifecycle composition helper needs no additional package.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
npm ci --prefix .qa-orchestrator-sdk
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-cypress
npm install --save-dev file:.qa-orchestrator-sdk/sdk/core file:.qa-orchestrator-sdk/sdk/cypress

npm install --save-dev @qantum-labs/runtrail-cypress

// cypress.config.ts
import { defineConfig } from "cypress";
import { registerQAOrchestrator } from "@qantum-labs/runtrail-cypress";

export default defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      registerQAOrchestrator(on, config, {
        apiKey: process.env.QA_API_KEY!,
        projectSlug: process.env.QA_PROJECT_SLUG ?? "your-project-slug",
        apiBaseUrl: process.env.QA_API_BASE_URL ?? "https://your-runtrail-instance.example",
      });
      return config;
    },
  },
});

// cypress.config.mjs
import { defineConfig } from "cypress";
import { registerQAOrchestrator } from "@qantum-labs/runtrail-cypress";

// Replace this body with the existing setupNodeEvents body or import it.
async function existingSetupNodeEvents(on, config) {
  on("task", { ping: () => "pong" });
  return config;
}

function composeLifecycle(on) {
  const shared = new Set(["before:run", "after:spec", "after:run"]);
  const handlers = new Map();
  return (event, handler) => {
    if (!shared.has(event)) return on(event, handler);
    if (!handlers.has(event)) {
      handlers.set(event, []);
      on(event, async (...args) => {
        for (const callback of handlers.get(event)) await callback(...args);
      });
    }
    handlers.get(event).push(handler);
  };
}

export default defineConfig({
  e2e: {
    async setupNodeEvents(on, config) {
      const composedOn = composeLifecycle(on);
      const resolved = await existingSetupNodeEvents(composedOn, config) ?? config;
      if (process.env.QA_API_KEY) {
        registerQAOrchestrator(composedOn, resolved, {
          apiKey: process.env.QA_API_KEY,
          projectSlug: process.env.QA_PROJECT_SLUG,
          apiBaseUrl: process.env.QA_API_BASE_URL,
        });
      }
      return resolved;
    },
  },
});

npx cypress run

Mocha

Reporter uses Mocha reporterOptions and extracts requirement IDs from titles.

Merge the configuration reference into .mocharc.cjs. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Mocha >= 10. The SDK itself requires Node.js >= 18, but the CLI bridge shown here requires synchronous require(ESM), available in Node.js >= 22.13. Use a Node version supported by your installed Mocha. This recipe was checked with Node.js 24 and Mocha 12. Do not present the SDK's Node 18 minimum as sufficient for this bridge.

Required for the current ESM SDK package: use the two-file reporter bridge below instead of --reporter @qantum-labs/runtrail-mocha. Mocha's CLI loads custom reporters with require(), while the package exposes an import entry. The bridge imports the SDK and passes flat SDK options; Mocha's reporterOptions wrapper is not the SDK constructor's options object. Older Node projects need an ESM programmatic runner that imports the reporter, or a compatible package build.

Mocha selects one reporter. Preserve the current reporter in a composite reporter or its existing multi-reporter mechanism. The example keeps Spec; substitute the project's custom reporter constructor and options if present. Keep spec/globs, require hooks, timeout, retries, grep, parallel settings, and npm scripts in .mocharc.cjs.

Required for complete delivery: let asynchronous reporting finish. An existing --exit flag or process.exit() can terminate HTTP delivery; resolve that conflict explicitly and verify the receipt. Merely installing a multi-reporter package does not fix ESM loading or constructor options.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
npm ci --prefix .qa-orchestrator-sdk
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core
npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-mocha
npm install --save-dev file:.qa-orchestrator-sdk/sdk/core file:.qa-orchestrator-sdk/sdk/mocha

npm install --save-dev @qantum-labs/runtrail-mocha

// .mocharc.cjs
// First create runtrail-reporter.mjs and runtrail-reporter.cjs from the guide's Mocha examples.
// The CLI bridge requires Node.js >= 22.13; retain existing options and console reporters.
module.exports = {
  reporter: "./runtrail-reporter.cjs",
};

// runtrail-reporter.mjs — imported by the CommonJS bridge below.
import Mocha from "mocha";
import { QAOrchestratorReporter } from "@qantum-labs/runtrail-mocha";

export default class CombinedReporter {
  constructor(runner, options) {
    // Keep the existing console reporter. Substitute your existing reporter here.
    new Mocha.reporters.Spec(runner, options);
    if (process.env.QA_API_KEY) {
      // The SDK constructor accepts flat SDK options, not Mocha's options wrapper.
      new QAOrchestratorReporter(runner, {
        apiKey: process.env.QA_API_KEY,
        projectSlug: process.env.QA_PROJECT_SLUG,
        apiBaseUrl: process.env.QA_API_BASE_URL,
      });
    }
  }
}

// runtrail-reporter.cjs — requires Node.js with synchronous ESM loading.
module.exports = require("./runtrail-reporter.mjs").default;

module.exports = {
  // Retain the project's existing spec, hooks, timeout, retries, and other options.
  reporter: "./runtrail-reporter.cjs",
};

npx mocha

Pytest

Plugin reads pytest ini options, CLI flags, or QA_* environment variables.

Merge the configuration reference into pyproject.toml. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Python >= 3.10, pytest >= 8, and the adapter installed in the same virtual environment as pytest. The pytest11 entry point qa_orchestrator auto-loads the plugin; no conftest.py hook or test-import changes are required. Without a key or project slug, reporting is inactive.

Required when plugin auto-loading is disabled: explicitly load -p qa_orchestrator_pytest.plugin, retaining any other manually loaded plugins. Do not also register the same plugin in conftest.py or pytest_plugins. Inspect PYTEST_DISABLE_PLUGIN_AUTOLOAD and -p no:qa_orchestrator before diagnosing missing output.

Required: merge qa_project_slug and qa_api_base_url into the existing [tool.pytest.ini_options] table or active pytest.ini. Keep addopts, markers, testpaths, and fixtures. Do not create a second table with the same TOML name or store qa_api_key in configuration.

Required for source installs managed by uv: persist BOTH the core and adapter with uv add --dev --editable, commit pyproject.toml and uv.lock, and recreate their relative source paths before uv sync --locked in CI. The pip source reference is an environment install, not a persisted dependency declaration.

Optional features: --qa-timeline enables Timeline. The current plugin uploads captured stdout/stderr/log artifacts when present; it has no uploadArtifacts=false option. pytest-bdd tests report as ordinary pytest tests without automatic feature/step discovery.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
python -m pip install -e .qa-orchestrator-sdk/sdk/python-core -e .qa-orchestrator-sdk/sdk/pytest

python -m pip install runtrail-pytest

# pyproject.toml
[tool.pytest.ini_options]
qa_project_slug = "your-project-slug"
qa_api_base_url = "https://your-runtrail-instance.example"

# Keep qa_api_key out of this file. Use QA_API_KEY in your shell or CI secret store.

[tool.pytest.ini_options]
# Preserve the project's existing options; add these two keys to the same table.
qa_project_slug = "your-project-slug"
qa_api_base_url = "https://your-runtrail-instance.example"

python -m pytest -p qa_orchestrator_pytest.plugin tests/

uv add --dev --editable .qa-orchestrator-sdk/sdk/python-core .qa-orchestrator-sdk/sdk/pytest
uv run pytest

pytest

Behave

Formatter sends scenario, step, feature, and step definition metadata.

Merge the configuration reference into environment variables. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Python >= 3.10, behave >= 1.2.6, and both core and adapter in the active virtual environment. Select the formatter by its fully qualified path qa_orchestrator_behave.formatter:QAOrchestratorFormatter, or explicitly define the qa_orchestrator alias in [behave.formatters]. Do not assume the package's entry point registers the short alias in every Behave version.

Required: append the formatter to existing formatters and preserve feature paths, tags, userdata, output destinations, and features/environment.py hooks. No before_all or after_all rewrite is needed. Existing command-line -f options can override configured formatters; check the actual test script.

Optional: QA_BDD_SYNC_FEATURES=false disables feature-file content synchronization; it defaults to true. BDD scenario/step metadata still reports. Set the choice explicitly when feature-file content should not be sent.

Required for uv source projects: record editable core and adapter paths with uv add --dev --editable, commit the lockfile, and reproduce the paths in CI. Do not rely on a temporary uv pip install surviving uv run synchronization.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
python -m pip install -e .qa-orchestrator-sdk/sdk/python-core -e .qa-orchestrator-sdk/sdk/behave

python -m pip install runtrail-behave

# Behave uses environment variables plus the qa_orchestrator formatter.
export QA_PROJECT_SLUG="your-project-slug"
export QA_API_BASE_URL="https://your-runtrail-instance.example"
export QA_BDD_SYNC_FEATURES="false"

behave -f pretty -f qa_orchestrator_behave.formatter:QAOrchestratorFormatter

[behave]
# Keep existing formatters and settings; append qa_orchestrator once.
format = pretty
    qa_orchestrator

[behave.formatters]
qa_orchestrator = qa_orchestrator_behave.formatter:QAOrchestratorFormatter

behave -f pretty -f qa_orchestrator_behave.formatter:QAOrchestratorFormatter

behave -f pretty -f qa_orchestrator_behave.formatter:QAOrchestratorFormatter

Robot Framework

Listener uses Robot Framework v3 listener API and QA_* environment variables.

Merge the configuration reference into environment variables. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Python >= 3.10, Robot Framework >= 6, and the adapter in the same virtual environment. Activate --listener qa_orchestrator_robot.listener.QAOrchestratorListener. The bare class name QAOrchestratorListener is not an installed top-level module.

Required: append this listener once to the existing command or argument file. Preserve existing listeners, --variable/--variablefile, --outputdir, include/exclude tags, and suites. No Library import or test-keyword changes are needed. The listener needs QA_API_KEY and QA_PROJECT_SLUG when initialized and flushes on close; let the runner finish.

Required for uv: persist core and adapter paths with uv add --dev --editable and use uv run robot. For Pabot, keep the project's existing process and output-merging behavior; verify the receipts produced by the actual worker executions rather than promising one aggregate run.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
python -m pip install -e .qa-orchestrator-sdk/sdk/python-core -e .qa-orchestrator-sdk/sdk/robot

python -m pip install runtrail-robot

# Robot Framework listener configuration
export QA_PROJECT_SLUG="your-project-slug"
export QA_API_BASE_URL="https://your-runtrail-instance.example"

robot --listener qa_orchestrator_robot.listener.QAOrchestratorListener tests/

# robot.args — retain all existing arguments and listeners.
--listener
qa_orchestrator_robot.listener.QAOrchestratorListener

python -m robot --argumentfile robot.args tests/

robot --listener qa_orchestrator_robot.listener.QAOrchestratorListener tests/

JUnit 5

ServiceLoader auto-registers the JUnit Platform listener from the test classpath.

Merge the configuration reference into build file + environment variables. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Java >= 17, a JUnit Platform/Jupiter test runner, and the SDK JAR on the test runtime classpath. This SDK builds against Platform 1.11/Jupiter 5.11. Keep the project's compatible JUnit BOM and engine; the listener does not supply the consumer's test engine.

Required: preserve the JAR's META-INF/services/org.junit.platform.launcher.TestExecutionListener entry. ServiceLoader activates the listener; no test annotations are needed. If the project disables listener auto-registration or uses a custom Launcher, explicitly register the listener once and preserve existing listeners.

Required for Gradle: keep useJUnitPlatform() on the actual Test tasks and include a compatible junit-platform-launcher at test runtime. mavenLocal() is needed only for source-built mvn install artifacts; build/install the same pinned SDK revision in CI before running Gradle or Maven, or use a published repository.

Required: set QA_* variables in the forked test JVM, not just a build container that does not pass them through. SDKs do not load .env files automatically. Preserve Surefire configuration, other listeners, test selection, and failure exit status.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
mvn -f .qa-orchestrator-sdk/sdk/junit5/pom.xml install -DskipTests

<!-- pom.xml -->
<dependency>
  <groupId>com.qaorchestrator</groupId>
  <artifactId>runtrail-junit5</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

<!-- pom.xml -->
<dependency>
  <groupId>com.qaorchestrator</groupId>
  <artifactId>runtrail-junit5</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

# The JUnit5 listener is auto-registered through ServiceLoader.
export QA_PROJECT_SLUG="your-project-slug"
export QA_API_BASE_URL="https://your-runtrail-instance.example"
export QA_COMMIT_SHA="$(git rev-parse HEAD)"
export QA_BRANCH="$(git rev-parse --abbrev-ref HEAD)"

mvn test

<dependency>
  <groupId>com.qaorchestrator</groupId>
  <artifactId>runtrail-junit5</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

repositories {
    mavenLocal() // Only for source artifacts built with mvn install.
    mavenCentral()
}
dependencies {
    // Keep the project's existing JUnit BOM and Jupiter engine dependencies.
    testImplementation("com.qaorchestrator:runtrail-junit5:0.1.0")
    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}
tasks.withType<Test>().configureEach {
    useJUnitPlatform()
}

mvn test

TestNG

Reporter is discovered by TestNG ServiceLoader or can be listed in testng.xml.

Merge the configuration reference into build file + environment variables. Environment placeholders must be replaced with the resolved project and instance. Run the repository's existing tests; the command below is a starting point. Retain ESM/CommonJS conventions and existing package-manager scripts.

Required: Java >= 17, TestNG on the test runtime classpath, and a compatible TestNG runner. The SDK builds against TestNG 7.10.2 and marks it provided; adding the SDK does not install TestNG for the consumer.

Required: preserve META-INF/services/org.testng.ITestNGListener so ServiceLoader discovers QAOrchestratorListener. Prefer this live listener. Explicit testng.xml or @Listeners registration is an alternative for runners that disable discovery; select one registration strategy and keep the project's other listeners.

Required for Gradle: enable useTestNG() on the actual Test tasks. If using testng.xml, ensure the existing runner actually selects that suite. QAOrchestratorReporter is a fallback; when a live listener is active it skips final ingest to avoid duplication. Do not add the fallback as the default live listener.

Required for source builds: install the pinned SDK with Maven in each clean CI runner; use mavenLocal() for Gradle, or a published artifact repository. Pass QA_* into the test JVM and preserve existing suite selection and failure behavior.

git clone https://github.com/legnadev/qa-orchestrator.git .qa-orchestrator-sdk
mvn -f .qa-orchestrator-sdk/sdk/testng/pom.xml install -DskipTests

<!-- pom.xml -->
<dependency>
  <groupId>com.qaorchestrator</groupId>
  <artifactId>runtrail-testng</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

<!-- pom.xml -->
<dependency>
  <groupId>com.qaorchestrator</groupId>
  <artifactId>runtrail-testng</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

# TestNG discovers the live listener through ServiceLoader.
export QA_PROJECT_SLUG="your-project-slug"
export QA_API_BASE_URL="https://your-runtrail-instance.example"

# Optional explicit listener in testng.xml:
# <listener class-name="com.qaorchestrator.testng.QAOrchestratorListener" />

mvn test

repositories {
    mavenLocal() // Only for source artifacts built with mvn install.
    mavenCentral()
}
dependencies {
    // Keep the project's existing compatible TestNG dependency.
    testImplementation("com.qaorchestrator:runtrail-testng:0.1.0")
}
tasks.withType<Test>().configureEach {
    useTestNG()
}

<!-- Add only if ServiceLoader discovery is disabled; retain other listeners. -->
<listener class-name="com.qaorchestrator.testng.QAOrchestratorListener" />

mvn test

Optional Selenium, Appium, and WebDriver Timeline SDKs

These three Node.js >= 18 packages capture driver events and need a running upstream Selenium/Appium server. They do not install a server or replace the result reporter for your test framework. Installing a Timeline adapter alone cannot satisfy test-result receipt verification.

For source installs, build/install in dependency order: @qantum-labs/runtrail-core, @qantum-labs/runtrail-timeline-webdriver-proxy, then @qantum-labs/runtrail-timeline-selenium or @qantum-labs/runtrail-timeline-appium. Reproduce all three dependencies in CI or pack all three at matching versions; a published adapter normally resolves its dependencies automatically.

Selenium: startSeleniumTimelineProxy returns {proxy, recorder}; point the existing client at the proxy URL while leaving upstreamUrl aimed at the real WebDriver server. Appium: startAppiumTimelineProxy works the same way; preserve the upstream base path (for example /wd/hub when actually used), capabilities, and existing session hooks. The default ports are 4445 and 4724 respectively.

WebDriver proxy: construct WebDriverTimelineProxy with a TimelineRecorder. Supply JsonlTimelineSink for local output and QAOrchestratorTimelineSink for remote batches. Keep the proxy on loopback by default; binding a non-loopback address requires authToken/QA_WEBDRIVER_PROXY_TOKEN and authenticated client requests. This proxy token is separate from the RunTrail API key.

Required when enabled: pass credentials only to the HTTP sink, load the environment in the bootstrap, preserve test hooks, and flush the recorder and close the proxy in awaited teardown. Pass real run/test correlation IDs when available; a driver proxy cannot infer every assertion or original test name. Snapshot references need separately stored/uploaded files. Local JSONL alone proves capture, not remote receipt.

import { JsonlTimelineSink, QAOrchestratorTimelineSink, TimelineRecorder } from "@qantum-labs/runtrail-core";
import { WebDriverTimelineProxy } from "@qantum-labs/runtrail-timeline-webdriver-proxy";

const recorder = new TimelineRecorder({
  source: { framework: "selenium", adapter: "webdriver-proxy", mode: "proxy" },
  runId: process.env.QA_TIMELINE_RUN_ID,
  sinks: [
    new JsonlTimelineSink(".qa-timeline/webdriver.jsonl"),
    new QAOrchestratorTimelineSink({
      apiKey: process.env.QA_API_KEY!,
      projectSlug: process.env.QA_PROJECT_SLUG!,
      apiBaseUrl: process.env.QA_API_BASE_URL,
    }),
  ],
});
const proxy = new WebDriverTimelineProxy({ upstreamUrl: "http://127.0.0.1:4444", recorder });
await proxy.listen(4445, "127.0.0.1");
// Point the existing client at port 4445. After its test/session teardown:
await proxy.close();
await recorder.flush();

import { startSeleniumTimelineProxy } from "@qantum-labs/runtrail-timeline-selenium";

// Call from the runner's existing async setup hook.
const { proxy, recorder } = await startSeleniumTimelineProxy({
  upstreamUrl: "http://127.0.0.1:4444",
  port: 4445,
  jsonlPath: ".qa-timeline/selenium.jsonl",
  apiKey: process.env.QA_API_KEY,
  projectSlug: process.env.QA_PROJECT_SLUG,
  apiBaseUrl: process.env.QA_API_BASE_URL,
  runId: process.env.QA_TIMELINE_RUN_ID,
});
// Configure the existing WebDriver client to use http://127.0.0.1:4445.
// In its existing async teardown, after sessions have finished:
await proxy.close();
await recorder.flush();

import { startAppiumTimelineProxy } from "@qantum-labs/runtrail-timeline-appium";

const { proxy, recorder } = await startAppiumTimelineProxy({
  upstreamUrl: "http://127.0.0.1:4723",
  port: 4724,
  jsonlPath: ".qa-timeline/appium.jsonl",
  apiKey: process.env.QA_API_KEY,
  projectSlug: process.env.QA_PROJECT_SLUG,
  apiBaseUrl: process.env.QA_API_BASE_URL,
});
// Use http://127.0.0.1:4724 in the existing Appium client.
// After the existing test/session teardown:
await proxy.close();
await recorder.flush();

Shared Node and Python core SDKs

@qantum-labs/runtrail-core (Node.js >= 18) and runtrail-core (Python >= 3.10) provide clients, result contracts, retrying, buffers, and Timeline primitives for custom adapters. Installing a core alone does not discover or report tests. Prefer an existing framework adapter unless implementing a new one.

Custom adapters must collect actual runner outcomes, build valid BufferedResult objects, supply key/project/base URL to the client, and await delivery/flush before the process exits. Capture the returned ingest ID and verify it after the actual test execution. Environment loading and hook composition belong to the consumer. An uploader or TimelineRecorder by itself is not a result reporter.

import { AdapterRuntime, type BufferedResult } from "@qantum-labs/runtrail-core";

export async function reportCompletedTests(results: BufferedResult[]) {
  const runtime = new AdapterRuntime<BufferedResult>({
    apiKey: process.env.QA_API_KEY!,
    projectSlug: process.env.QA_PROJECT_SLUG!,
    apiBaseUrl: process.env.QA_API_BASE_URL,
  }, { bufferFile: "custom-results.jsonl", sdkVersion: "0.1.0" });
  // Caller supplies actual runner results. Buffered outcomes are pending delivery.
  const receipt = await runtime.flush(results);
  return receipt;
}

import os
from qa_orchestrator_core.client import IngestClient
from qa_orchestrator_core.contracts import BufferedResult


def report_completed_tests(results: list[BufferedResult]) -> str:
    client = IngestClient(
        api_key=os.environ["QA_API_KEY"],
        api_base_url=os.environ["QA_API_BASE_URL"],
    )
    # Caller supplies real test results; never use fabricated results for verification.
    return client.ingest(os.environ["QA_PROJECT_SLUG"], results)

Validate the connection before running tests

POST /api/v1/ingest/dry-run validates the API key, project slug, payload, and ingest limits. It does not enqueue a job or create a test run. A successful dry-run proves configuration only; it does not prove real results were received.

Set QA_TEST_FRAMEWORK to the detected adapter ID (playwright, vitest, jest, cypress, mocha, pytest, behave, robot, junit5, or testng). Save the following as runtrail-check.py and run it with Python 3 in the configured environment. It uses the standard library and never prints the credential.

import json
import os
from urllib.error import HTTPError
from urllib.request import HTTPRedirectHandler, Request, build_opener


class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


payload = {
    "project_slug": os.environ["QA_PROJECT_SLUG"],
    "run": {"sdk_version": "setup-check", "environment": {"synthetic": True}},
    "results": [{
        "test_name": "RunTrail connection check",
        "framework": os.environ["QA_TEST_FRAMEWORK"],
        "status": "passed",
        "duration_ms": 1,
        "metadata": {"synthetic": True},
        "artifacts": [],
    }],
}
request = Request(
    os.environ["QA_API_BASE_URL"].rstrip("/") + "/api/v1/ingest/dry-run",
    data=json.dumps(payload).encode(),
    headers={"X-API-Key": os.environ["QA_API_KEY"], "Content-Type": "application/json"},
    method="POST",
)
try:
    with build_opener(NoRedirect).open(request, timeout=30) as response:
        print(json.dumps(json.load(response), indent=2))
except HTTPError as error:
    raise SystemExit(f"Connection check failed: HTTP {error.code}")

Verify receipt of this test execution

Run at least one real test using the SDK and capture the exact run_id from its lifecycle receipt/log, or ingest_id from its final-ingest receipt/log. GET /api/v1/sdk/verification accepts exactly one of run_id or ingest_id, authenticates with X-API-Key, and requires ingest permission. It returns project_slug, run_id, ingest_id, status, received_count, and verified. It does not expose test contents or error traces.

Poll queued, processing, or running deliveries for a bounded period, for example once every two seconds for up to sixty seconds. Only report completed integration when verified=true and received_count>0 for this execution. Verification requires a finished run with stored results and, for final ingest, a completed job. Failed tests can still have verified delivery. An empty, running, deleted, failed-processing, or unknown delivery is not proof of integration.

A dry-run, an HTTP 202 acknowledgment, local buffered JSONL, or results from an earlier run are insufficient. Do not fabricate a synthetic ingest to claim success. If the SDK provides no receipt ID, report verification as pending and use SDK Setup or the dashboard to confirm the actual execution.

curl --fail-with-body --silent --show-error \
  -H "X-API-Key: $QA_API_KEY" \
  "$QA_API_BASE_URL/api/v1/sdk/verification?run_id=<actual-run-id>"

curl --fail-with-body --silent --show-error \
  -H "X-API-Key: $QA_API_KEY" \
  "$QA_API_BASE_URL/api/v1/sdk/verification?ingest_id=<actual-ingest-id>"

Connect the existing CI pipeline

Prepare the existing CI job with the same adapter, configuration, and test command used locally. Supply QA_API_KEY through the provider's secret store and QA_PROJECT_SLUG and QA_API_BASE_URL as environment variables. Never put a real key in pipeline YAML. Configuring secrets in GitHub, GitLab, Buildkite, or CircleCI requires access to that provider in addition to the RunTrail key.

A quality gate is optional. GET /api/v1/projects/<project-slug>/quality-gate/ci?fail_on_error=true requires read permission in addition to the ingest permission used by the SDK. Preserve the runner's test exit status; configuring reporting should not turn failing tests into passing CI.

Finish with the changed files, the test command and outcome, dry-run outcome, exact run/ingest ID, verification response, and any remaining CI or publication prerequisites. Do not mark copying instructions as completed installation or verified delivery.

# Keep the job's existing checkout, setup, install, and test prerequisites.
- name: Run the existing tests
  env:
    QA_API_KEY: ${{ secrets.QA_API_KEY }}
    QA_PROJECT_SLUG: your-project-slug
    QA_API_BASE_URL: https://your-runtrail-instance.example
  run: npm run test:e2e # Replace with the repository's actual existing command.

# Use only for checkout-based file: dependencies; vendored tarballs need no SDK clone.
- uses: actions/checkout@v4
  with:
    repository: legnadev/qa-orchestrator
    ref: <audited-full-commit-sha>
    path: .qa-orchestrator-sdk # Must match the consumer dependency paths.
    token: ${{ secrets.SDK_REPO_TOKEN }} # Only needed when SDK Git access requires it.
- name: Build SDK source dependencies
  run: |
    npm ci --prefix .qa-orchestrator-sdk
    npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-core
    npm run build --prefix .qa-orchestrator-sdk --workspace @qantum-labs/runtrail-playwright
- run: npm ci # Run in the directory containing the consumer package.json.

Resolve common setup issues

401: check that QA_API_KEY is present in the actual runner process, valid, unexpired, and not revoked. 403: check ingest permission for reporting and read permission for optional quality gates. 404: check the instance URL and that the key allows the chosen project; verification also returns 404 for unknown or deleted deliveries. 409 on setup: pass the intended project_slug.

422: check the payload, detected framework, project settings, or UUID format. Verification needs exactly one run_id or ingest_id. Plan or project limits can also prevent ingestion; report the returned error instead of changing the account plan or destination project.

No reporter output: check registration in the runner's configuration and environment loading. No remote results: check instance reachability, SDK warnings, .qa-orchestrator offline buffers, and the exact delivery receipt. Queued ingestion requires the RunTrail worker to process the job; wait for verified delivery rather than treating acceptance as completion.

Package not found: confirm the registry and published version, or use an accessible source checkout. CI cannot resolve a local file dependency: provide the same checkout/build paths or switch to published/versioned packages. Never claim registry availability from this guide alone.