Skip to content

Quality and security gates

This page owns what CI enforces, at PR time and at publish time. Each workflow file is the source of truth for its own steps, and this page names the rule. To run the same checks locally, see Code quality.

CI quality gates

Workflow: .github/workflows/quality-gates.yml

Runs on every pull request and on pushes to main.

Required jobs:

  • bandit: writes bandit-report.json (with --exit-zero so low/medium findings do not fail the job before the gate), prints a log summary, uploads the artifact, then fails only on high-severity findings in the enforce step. Scan roots: src/, nw-cli/src/, nw-mcp-builder/src/, nw-connector-builder/src/.

Workflow: .github/workflows/codeql.yml

Runs GitHub CodeQL static analysis for Python on pull requests, pushes to main, and weekly (Mondays). No repository secrets are required.

Workflow: .github/workflows/pytest.yml

Runs the full test suite on Linux, macOS, and Windows (Python 3.13) with coverage on every pull request and push to main. Branch protection requires the Ubuntu cells only; macOS/Windows remain informational (OSS portability signal without blocking merges on runner flakiness). Playground integration tests remain manual (workflow_dispatch) on Ubuntu only.

Workflow: .github/workflows/lint.yml also runs lockfile-check (uv lock --check) to fail PRs when pyproject.toml changes without an updated uv.lock.

Workflow: .github/workflows/mcp-builder-e2e.yml

End-to-end gate for the priority surface (nw CLI → connector codegen → MCP Docker image → live tools/list). The heavy pet-store job is path-filtered (plus weekly schedule / workflow_dispatch). An always-run job MCP builder e2e gate succeeds when e2e is skipped and fails when e2e fails — require that check name in branch protection, not the path-filtered job.

Workflow: .github/workflows/secret-scan.yml

Runs Gitleaks on pull requests, pushes to main, weekly (Mondays), and on manual dispatch. The workflow checks out full git history (fetch-depth: 0) so secrets in past commits are scanned, not only the working tree.

Required checks to add in branch protection:

  • Lint and Type Check / Lockfile freshness
  • Lint and Type Check / Ruff Linters
  • Lint and Type Check / Mypy Type Check
  • Quality gates / Bandit security scan
  • CodeQL / Analyze (Python)
  • Secret scan / Gitleaks secret scan
  • CI – Pytest / Run pytest (ubuntu-latest, Python 3.13)
  • MCP builder e2e (pet-store) / MCP builder e2e gate
  • Python package security PR checks / Vulnerability scan (packages/runtime)

Configure branch protection so pull requests cannot merge unless all required checks pass.

Informational (run in CI, not required to merge): macOS/Windows pytest cells; connector-package pip-audit matrix (path-filtered on PRs, always on schedule); REUSE lint (still runs in lint.yml); DCO (skips bot PRs — do not require it or Dependabot merges will stick on a skipped check).

CVE scanning policy

  • PR and push-to-main scanning runs in .github/workflows/security-pr.yml.
  • packages/runtime is audited on every PR / push to main (required check).
  • Connector packages under packages/connectors/* are audited when those paths (or lockfile / related workflows) change, and on the daily schedule — not required in branch protection.
  • Release-time scanning remains in .github/workflows/publish.yml as defense in depth.
  • The PR/push gate (security-pr.yml) runs pip-audit with no --fail-on threshold, so it blocks on any vulnerability. The release workflow (publish.yml) uses pip-audit --fail-on HIGH as defense in depth.
  • Scheduled scans catch newly disclosed CVEs even when code does not change.

Monorepo install note: Connector packages under packages/connectors/* declare node-wire-runtime>=1.0.0 as a normal PyPI dependency name. The security workflow installs packages/runtime from the checkout together with each matrix package (pip install packages/runtime "<matrix path>") so pip can resolve node-wire-runtime without requiring a published wheel on PyPI. Locally, mirror that when auditing a single connector: pip install packages/runtime packages/connectors/<name>.

Secret scanning

Workflow: .github/workflows/secret-scan.yml (Gitleaks).

Policy:

  • Scan on every PR and push to main, plus a weekly scheduled run.
  • Full repository history is included (fetch-depth: 0).
  • Findings fail the workflow; remediate by rotating exposed credentials and removing secrets from the codebase (never commit live secrets).

Run it locally: Code quality — Security scans.

Bandit policy

Bandit is configured in pyproject.toml under [tool.bandit].

Exit codes and CI behavior

By default, Bandit exits with a non-zero status whenever it reports any finding, including low and medium severity. That affects -f json -o ... the same as text output.

CI splits responsibilities:

  1. JSON artifact + log summary — bandit ... -f json -o bandit-report.json --exit-zero so the workflow always produces the report and runs scripts/bandit_report_summary.py for readable logs. Low/medium issues are visible here without failing the job.
  2. Enforcement — bandit ... --severity-level high fails the job only on high-severity findings (matches branch-protection intent).

Locally, mirror CI with the commands in Code quality — Security scans.

Scope

Policy:

  • Scan targets: src/ (runtime, bindings, in-tree connectors), plus nw-cli/src/, nw-mcp-builder/src/, and nw-connector-builder/src/ (CLI → MCP pipeline).
  • Exclude: .venv, venv, tests, playground, dist, htmlcov.
  • CI enforcement threshold: --severity-level high.
  • Packages tree: connector distributions under packages/connectors/* are audited for CVEs in .github/workflows/security-pr.yml (pip-audit). Run Bandit against those paths separately if you need SAST on a standalone checkout.

SBOM generation

CycloneDX SBOM (sbom.json) is generated by:

  • scripts/run-compliance-checks.sh for local compliance runs.
  • .github/workflows/github-release.yml at release time (attached to the GitHub Release).

Acceptance criteria mapping

  • Security scan runs on every PR: enforced by quality-gates.yml (Bandit) and codeql.yml (CodeQL).
  • Builds fail on high-severity Bandit findings: Bandit gate in CI.
  • Static analysis visible in GitHub Security tab: CodeQL upload from CI.
  • Tests run on every PR: enforced by pytest.yml (required: Ubuntu × Python 3.13).
  • MCP pipeline integrity: mcp-builder-e2e.yml gate required; heavy pet-store job path-filtered.
  • Developers run checks locally: documented commands and pre-commit (Bandit).
  • Config version-controlled: pyproject.toml, .pre-commit-config.yaml, workflow files.