Skip to main content

Running the test harness

The Edge template QA harness is provided by Kaptio during onboarding. It runs locally with Node.js against a checkout of your configuration repository, reads your tenant's qa-manifest.yaml, and verifies every declared expectation. This page is a step-by-step guide a QA engineer can follow without prior context.

What the harness does​

For each run, the harness:

  1. Validates every dataset against its schema using ajv (JSON Schema validation). Bad sample data fails before anything renders.
  2. Renders each dataset — and each action's display_overrides set — through the same render engine the platform uses in production.
  3. Asserts the manifest's selectors (expect_selectors / absent_selectors) at both declared viewports: desktop 1200×900 and mobile 390×844.
  4. Scans rendered output for forbidden strings (undefined, [object Object], NaN, {{, Invalid date).
  5. Produces screenshots and PDFs for every render, so failures can be inspected visually and print output can be reviewed.

Step 1 — Prerequisites​

You need:

  • Node.js 20 or later (node --version to check).
  • The harness package, provided by Kaptio during onboarding.
  • A checkout of your configuration repository, on the branch you want to test.

Step 2 — Install​

From the harness directory, install dependencies once:

npm install

Step 3 — Run the full suite​

node run-tests.mjs --configs <your-config-repo> --tenant <your-tenant> --actions
  • --configs points at the root of your configuration repository checkout.
  • --tenant names the tenant folder to test.
  • --actions includes the action matrix — every action variant declared in the manifest, not just the default renders. Always include it before a merge request.

Step 4 — Narrow the loop while iterating​

Run a single document type while you work on it:

node run-tests.mjs --configs <your-config-repo> --tenant <your-tenant> --doc option

Render one dataset straight to an HTML file you can open in a browser:

node render.mjs --configs <your-config-repo> --tenant <your-tenant> --doc option \
  --data <your-config-repo>/tenants/<your-tenant>/sample-data/option.json \
  --out /tmp/option.html

Run the helper parity gate — required whenever a template uses a new Handlebars helper, because helpers must exist in every production render registry:

node run-tests.mjs --configs <your-config-repo> --tenant <your-tenant> \
  --parity --platform <path-to-the-platform-package-supplied-with-the-harness>

The parity gate needs the platform package (supplied alongside the harness) to compare helper registries, which is why --platform is required here and not for the render suite.

Step 5 — Read the report​

The suite prints a per-dataset, per-action result and writes screenshots and PDFs for each render. On failure, start with the artifact for the failing render — most failures are obvious the moment you see the page.

Step 6 — Meet the quality gate​

Before a change merges: all datasets and all document types green, including the action matrix. A red action entry is a real defect even when the default render passes — it means one of your configured document actions produces broken output. The full release checklist is in The quality bar.

Common failures and what they mean​

Selector missing (expect_selectors not found)​

The section did not render. There are two usual causes:

  • The backing data is empty. Every section collapses entirely when it has no data (the empty-collapse guard), regardless of display flags. Check the dataset for the fields the section reads.
  • A _display flag hid it. Check the action's display_overrides in the manifest against the blueprint's hide_sections / show_sections for that action — they must agree.

Open the screenshot for the failing render first; it usually shows immediately whether the section is absent or merely empty.

Selector present (absent_selectors found)​

A section rendered that should not have. Usual causes:

  • A missing or wrong _display guard in the template. Sections that are hidden by default must use the positive pattern {{#if _display.showX}}; content sections use the unless-false pattern. Using the wrong pattern inverts the default.
  • Data leaked into a dataset that should not carry it — for example price-line data in a dataset meant to exercise a document without a breakdown.

Forbidden string in output​

An unrendered placeholder or bad data reached the output. Each string fingerprints a different defect:

  • {{ — a template expression did not resolve: a helper missing from a render registry, or a typo in the expression.
  • undefined / NaN — missing or non-numeric data reached a template without a guard.
  • [object Object] — an object was interpolated without formatting.
  • Invalid date — a date helper received a nil or malformed value; date expressions must be guarded against missing data.

Open the rendered HTML, search for the string, and trace the field back through the dataset and schema.

Schema validation error (ajv)​

The dataset and the schema disagree — a data/contract mismatch. The ajv error names the exact JSON path. Either the data is wrong, or the schema needs adjusting — remember that optional fields must be nullable (type: ["string", "null"]) because the data layer emits nulls; only required fields stay strict. See Schemas.

Parity gate failure​

A template uses a Handlebars helper that is not present in every production render registry; it will render in one context and fail in another. Replace the helper with one that exists everywhere, or use an inline partial. and/or/not helpers do not exist — nest blocks instead.

Deeper regression checks​

Beyond the standard suite, the harness ships deeper regression checks that are run after structural changes (new sections, new document types, layout reworks):

  • Minimal-data and empty/null renders — documents render correctly when optional data is absent.
  • Action-by-dataset and flag matrices — every flag combination against every dataset.
  • Helper parity — the registry check described above.
  • Image audit — every image URL reachable and subject-appropriate.
  • DOM, contrast, responsive, and print probes — accessibility contrast, horizontal-overflow detection at mobile width, and print/PDF flattening.

Each of these protects against a specific class of failure. They are not required on every copy change, but run the relevant ones whenever you change document structure — and expect Kaptio to run them on review.