QAP Rollout Strategy

Plan for ramping the Quality Assurance Pipeline from internal test to 100% of tenants. Updated whenever the rollout state changes — keep the "Current state" section accurate.

Current state

  • Phase: Internal test only (as of 2026-05).
  • Gate: ENABLE_QA_FOR_FROM_SCRATCH, ENABLE_QA_FOR_IMPORT env flags (legacy boolean gate) — see also the GrowthBook-driven gate below.
  • Ramp owners: deployment-pipeline engineers.

Ramp stages

Stage Audience Trigger Rollback
0 Off everywhere Env flag = False n/a
1 Internal-test tenant only GrowthBook rule: tenant_id == 1 Flip rule off
2 10% of from-scratch tenants GrowthBook percentage rule Flip percentage to 0
3 50% of from-scratch Same, increase percentage Same
4 100% from-scratch Same Same
5 10% of import tenants Separate rule on site_origin=import Same
6 100% all flows Both rules at 100% Same

Bake time between stages: 48 hours with no qa_failed spike on the Grafana dashboard.

GrowthBook gate

Feature key: qap_enabled (default off).

Attributes supplied at evaluation time (see idk/sites/feature_flags.py::qap_enabled_for_session): - tenant_id: session.site.client.subscription_id (stable across runs). - site_origin: "from_scratch" or "import".

Suggested ramp rules (in evaluation order):

  1. Internal test tenant override — tenant_id IN [1, 2, ...] → on.
  2. Per-flow percentage — split on tenant_id to keep tenants on a consistent side of the line; site_origin=from_scratch ramps before import.
  3. Default — off.

Fall-through: if GrowthBook is unreachable, code falls back to the env flags. Treat that as a degraded-but-safe state, not an incident.

Pre-ramp checks before each stage

  • qap_failures_total{phase!~".*provisioning.*"} < 5% of completed sessions in the last 24h.
  • p95 of qap_phase_duration_seconds per phase within the SLOs in qap_slos.md.
  • No active P0 in the QAP Linear project.
  • The runbook (qap_failure.md) covers every reason that fired in the prior stage.

Rollback procedure

  1. Set the GrowthBook rule to 0% (or disable the rule).
  2. Existing in-flight QAP sessions continue — the gate only affects new kickoffs. To halt in-flight sessions: 
    from idk.sites.models import SiteDiscoverySession
from idk.sites.pipeline_phases import QAP_PHASES
SiteDiscoverySession.objects.filter(
    pipeline_phase__in=QAP_PHASES,
).update(pipeline_phase="completed", pipeline_error="QAP rolled back")
    (Confirm with the deployment-pipeline owner first — this is destructive.)
  3. Notify #engineering with the reason for rollback so the post-mortem blocks the next ramp.

Cross-repo dependencies

  • agno-platform must be at version ≥ X.Y (the version that ships the /workflows/triage-failures agent). Older versions return 404 and QAP fails closed.
  • The schemas repo must exist and be writable by the QAP bot.