Back to principles
QA
Automation must be zero-failure
What this means in practice
- All automated tests run as part of the CI pipeline for every change (pull request and mainline).
- The pipeline is a quality gate: builds are not considered “green” unless test automation completes with a 0% failure rate.
- Teams do not ship with known test failures (e.g., “10% failing is fine”) and do not rely on manual triage to decide which failures matter.
- Flaky tests are treated as defects in the delivery system and are fixed/removed from the gating suite immediately.
Why this matters
- A non-zero failure rate normalises broken feedback loops and makes CI signals unreliable.
- If teams have to review failures to decide what is important, the pipeline is no longer a gate—it’s noise.
- A zero-failure standard preserves confidence, reduces rework, and prevents degraded quality from reaching customers.
Practices that meet this principle
- CI runs the full gating automated suite on every PR and blocks merges on any failure. If not on PR, then at least as part of the path to production
- The team maintains a clear, owned process to eliminate flakiness quickly (e.g., same-day fix or quarantine policy with strict time limit).
- Test results are visible and actionable (Show in Azure, link to logs, failures are treated as incidents when they block delivery).
- The gating suite stays fast enough to be used consistently; longer-running suites can exist, but they do not replace the zero-failure gate.
- Following Playwright standards/best practices to prevent flaky tests - build the quality from the start.
Validation
A project meets this principle when:
- The CI pipeline includes automated test execution as a required step for merge/release.
- The expected failure rate for the gating suite is 0% (no accepted baseline of failing tests).
- Any test failure blocks shipping until resolved.
- There is no routine manual “failure review” to decide whether a failed test can be ignored.