QAP Failure Runbook

Operational guide for sessions that land on qa_failed. Every stopped_reason value the trackers persist into SiteDiscoverySession.pipeline_result["qa_failure"]["reason"] has a section here. The canonical list of reasons lives in idk/sites/tasks/pipeline/quality_flow/stopped_reasons.py; adding a new reason without an entry below should fail CI (see test_stopped_reasons.py).

Top-level triage

Find recently failed sessions:

SELECT s.uuid, ss.pipeline_phase, ss.pipeline_result->'qa_failure' AS failure,
       ss.updated_at
FROM   sites_sitediscoverysession ss
JOIN   sites_site s ON s.id = ss.site_id
WHERE  ss.pipeline_phase = 'qa_failed'
   AND ss.updated_at > NOW() - INTERVAL '24 hours'
ORDER  BY ss.updated_at DESC;

Or via the Django ORM:

from idk.sites.models import SiteDiscoverySession
SiteDiscoverySession.objects.filter(
    pipeline_phase="qa_failed",
    updated_at__gte=timezone.now() - timedelta(hours=24),
).values("site__uuid", "pipeline_result__qa_failure", "updated_at")

Dashboard panel: QAP failures (last 24h, by reason) in docs/dashboards/qap.json (qap_failures_total).

Slot-level reasons (per repo)

missing_issue_number

Symptoms. Slot stuck waiting for a Copilot PR; tracker logs missing_issue_number. Frontend shows "Quality-check issue could not be located".

Cause. The bootstrap kickoff returned without issue_number populated on qa_test_issues[role]. Either GitHub failed to create the issue, or the Copilot assignment shadowed the response.

Remediation. 1. Check the target repo's Issues tab for an orphan quality-check issue. 2. If it exists, hand-paste the issue number into pipeline_result["qa_test_issues"][role]["number"] via the Django shell and call enqueue_quality_pipeline(site_uuid=..., force=True). 3. If no issue exists, the GitHub API call failed silently — escalate to the GitHub-integration owner with the request-id from logs.

missing_pr_metadata

Symptoms. PR was opened by Copilot but tracker can't find head SHA / branch. Logs show pr_walker returning empty.

Cause. Race between issue creation and Copilot's first commit. Usually self-recovers; if it doesn't, Copilot didn't attach a PR to the issue (account paused, repo permissions changed).

Remediation. 1. Open the issue on GitHub. Check the timeline for a linked PR. 2. If the PR exists, retry: enqueue_quality_pipeline(..., force=True). 3. If no PR exists after 15 minutes, the Copilot agent is the root cause — contact the Copilot integration owner.

schemas_repo_unset

Symptoms. Enqueue rejected with this reason immediately.

Cause. SCHEMAS_REPO Django setting is empty or unset.

Remediation. Fix env var; redeploy. Sessions don't auto-retry — call enqueue_quality_pipeline(..., force=True) per affected site.

checks_failed / heal_checks_failed

Symptoms. PR has red checks; QAP gave up after exhausting heal attempts.

Cause. Triage agent could not fix the test failures within QA_MAX_HEAL_ATTEMPTS. Either the failures are non-deterministic (flake), genuinely broken business logic, or the agent is misconfigured.

Remediation. 1. Open the PR. Check the latest run logs for the pattern of failure. 2. If flake: bump QA_MAX_HEAL_ATTEMPTS for this session by hand-editing pipeline_result["qa_test_state"][role] and retry. 3. If real failure: developer judgement call. Either fix manually + close QAP, or push a fix to the PR and let GitHub re-run.

merge_failed / heal_merge_failed

Symptoms. PR checks passed but the merge call rejected.

Cause. Branch-protection rule blocking the bot, required reviewer missing, or merge conflict with master (master moved while QAP was running).

Remediation. 1. Open the PR. If conflicts: rebase + merge by hand; QAP doesn't recover conflicts. 2. If branch protection: confirm the bot account has bypass_pull_request_allowances. 3. Retry once via enqueue_quality_pipeline(..., force=True).

max_heal_attempts

Symptoms. Slot terminated; heal cycle ran QA_MAX_HEAL_ATTEMPTS times without producing a green PR.

Cause. Code is genuinely broken or test setup is malformed. Not a QAP bug — a code-quality signal.

Remediation. Treat the original PR as a normal failed PR. QAP did its job. Do not retry without first fixing the underlying code.

upstream_unreachable

Symptoms. Slot accumulated 5 consecutive errors_in_a_row against GitHub or Agno.

Cause. Sustained 5xx from one of the upstream APIs, or auth token expired.

Remediation. 1. Check Grafana agno_request_duration_seconds{status=~"5.*"} and the GitHub Status page. 2. Rotate the failing token if applicable. 3. After upstream recovery, retry: enqueue_quality_pipeline(..., force=True).

missing_heal_issue_number / missing_heal_pr_metadata / heal_issue_create_failed

Same playbook as their non-heal counterparts above, scoped to the heal-cycle iteration. Logs include attempt=N so you can correlate with QAP_HEAL_ATTEMPTS_TOTAL.

Session-level reasons

soft_limit_exceeded

Symptoms. Tracker task hit Celery's soft_time_limit repeatedly (>MAX_SOFT_LIMIT_RETRIES) and gave up.

Cause. Worker overload or a hot loop in a slot. Distinct from stalled: this means workers are running but exceeding budget, not sitting idle.

Remediation. 1. Check Celery worker CPU + worker count vs. active QAP sessions. 2. If symptomatic of fleet pressure, scale workers or lower AGNO_TICK_BUDGET_SECONDS to break ticks faster. 3. After recovery: enqueue_quality_pipeline(..., force=True).

stalled

Symptoms. No tracker activity for the reaper TTL of the phase. The reaper task is the time authority (QAP_TIME_AUTHORITY=reaper); it moved the session to qa_failed because nothing has touched updated_at recently.

Cause. Worker died mid-tick, broker outage, or Celery beat is down.

Remediation. 1. Check Celery beat scheduler status and broker connectivity. 2. Check for the _exec_lock_key for this site in Redis (leftover from a dead worker). It auto-expires within 360 s; if stuck, DEL manually. 3. After recovery: enqueue_quality_pipeline(..., force=True).

repo_plan

Symptoms. Enqueue or bootstrap chain failed before any repo work started.

Cause. build_repo_plan(site) raised — usually a misconfigured contract or a site with no target repos.

Remediation. Site-specific: check Site.contract and the ContractGithubRepository mappings. No retry will work until the contract is fixed.

missing_qa_test_state / missing_qa_master_state

Symptoms. Tracker started with no per-slot state in pipeline_result.

Cause. Bug in the bootstrap chain (state wasn't persisted) or a manual pipeline_result edit that dropped the keys.

Remediation. Always retry with force=True; the orchestrator clears QAP keys on retry, which usually resolves the missing-state edge case. If it persists, file a bug.

Bootstrap-chain reasons

These appear when the 4-task chain (qap_harmonize_prototype_task → qap_provision_dockerfile_task → qap_provision_build_workflow_task → qap_test_prototype_kickoff_task) fails before a tracker starts polling.

qa_failure.reason Failing task
prototype_harmonization qap_harmonize_prototype_task
dockerfile_provisioning_failed qap_provision_dockerfile_task
build_workflow_provisioning_failed qap_provision_build_workflow_task
test_prototype_kickoff_failed qap_test_prototype_kickoff_task

Each task's logs include the underlying exception in pipeline_result["qa_failure"]["error"]. Re-run via enqueue_quality_pipeline(..., force=True) after fixing the root cause.

Enqueue-skipped reasons (not actual failures)

qap_enqueue_failed_total counter labels:

  • missing_session — site has no SiteDiscoverySession.
  • already_running — session is already in a QAP phase.
  • terminal_no_force — caller hit qa_completed / qa_failed without force=True.
  • schemas_repo_unset — same as the failure above.
  • broker_error — apply_async raised; the orchestrator rolled back. Look for Connection refused to the broker.

These are skip events, not session failures. The session stays on its prior phase.