Engineering Reliability: A Testing & Quality Framework for Modern Web Platforms

A practical framework covering every layer — from fast unit tests to full-stack E2E — and the quality gates that make merging safe.

1 The Testing Pyramid

The pyramid is a prioritisation tool, not a dogma. Its core message: write many fast tests at the bottom, few slow tests at the top. Every layer catches a different class of bug. Missing a layer means bugs escape to the layer above — where they cost more to find.

The pyramid shape is a cost signal: lower layers run faster and cheaper. Push testing down as far as possible.

2 Seven Test Types — Deep Dive

Each type answers a different question. Using the wrong type costs time; skipping a type lets bugs through.

Unit Test < 1s

Tests a single function or class in complete isolation. No network, no database, no filesystem. External dependencies are replaced with fakes or mocks at the boundary. The fast feedback loop that makes TDD practical.

def test_hash_password_is_deterministic():
    # No DB, no IO — pure function test
    h1 = hash_password("secret")
    h2 = hash_password("secret")
    assert h1 == h2
    assert h1 != "secret"

Integration Test < 30s

Exercises real infrastructure — a real database, a real message queue. Each test runs inside a transaction that is rolled back at teardown, so tests remain isolated even though they touch real storage. No mocking of the layer being tested.

async def test_bootstrap_409_if_already_running(client, db):
    r1 = await client.post("/api/v1/ansible/bootstrap",
        json={"minion_id": "mac-01", "target_ip": "203.0.113.11"})
    assert r1.status_code == 200

    r2 = await client.post("/api/v1/ansible/bootstrap",
        json={"minion_id": "mac-01", "target_ip": "203.0.113.11"})
    assert r2.status_code == 409

Contract Test < 5s

The producer (API) and consumer (frontend) independently verify they agree on data shapes. The contract is explicit — a Pydantic schema on one side, a TypeScript interface on the other. A contract test fails if either side drifts. Most teams skip this layer entirely; they discover contract breaks when users see blank screens.

# Producer side — Pydantic schema IS the contract
class BootstrapResponse(BaseModel):
    node_id: uuid.UUID
    minion_id: str
    job_id: str
    bootstrap_status: str
    message: str
    salt_key_deleted: bool = False

// Consumer side — TypeScript interface IS the contract
interface BootstrapResponse {
  node_id: string
  minion_id: string
  job_id: string
  bootstrap_status: string
  message: string
  // ⚠ salt_key_deleted missing — contract is already drifted
}

Component Test < 10s

Renders a single React component with a mocked API, verifying it handles loading, success, and error states correctly. Uses Vitest + Testing Library. Does not open a browser — runs in jsdom. Faster than E2E by an order of magnitude.

// BootstrapModal.test.tsx
it('shows error state when bootstrap API fails', async () => {
  server.use(
    rest.post('/api/v1/ansible/bootstrap', (req, res, ctx) =>
      res(ctx.status(409, 'Already bootstrapping')))
  )
  render(<BootstrapModal isOpen={true} />)
  await userEvent.click(screen.getByRole('button', { name: /start/i }))
  await screen.findByText(/already being bootstrapped/i)
})

End-to-End Test < 10 min total suite

A real browser, a real server, a real database. Playwright navigates the full user journey as a user would. Tests user behaviour, not implementation. Slow to run, high confidence, expensive to maintain. Should cover journeys, not every edge case — those belong lower in the pyramid.

test('BOOT-01 modal opens from fleet dashboard', async ({ page }) => {
  await page.click('button:has-text("+ Bootstrap Node")')
  await expect(page.locator('h2:has-text("Bootstrap Mac Mini")'))
    .toBeVisible()
})

Mutation Test minutes

A mutation testing tool (mutmut, Stryker) systematically breaks production code — flips a == to !=, removes a return statement, deletes a branch — then checks if your tests catch each change. The mutation score is the percentage of mutations your tests detect. A score below 70% means your tests are passing for the wrong reasons.

# Run mutmut on a specific module
mutmut run --paths-to-mutate fleet_platform/core/auth.py
mutmut results   # shows surviving mutants = test gaps

Property-Based Test < 30s

Hypothesis (Python) or fast-check (TypeScript) generates hundreds of random inputs and verifies that an invariant always holds. Finds edge cases that humans forget to write — empty strings, negative numbers, Unicode, max-length values. Write one property test to replace dozens of example-based tests.

from hypothesis import given, strategies as st

@given(st.text(min_size=1, max_size=128))
def test_validate_minion_id_never_crashes(s):
    # Invariant: validation never raises an exception — only True/False
    result = is_valid_minion_id(s)
    assert isinstance(result, bool)

3 The TDD Cycle

Test-Driven Development is a discipline, not a suggestion. The cycle is short — three phases, typically two to ten minutes per iteration. The discipline: never write production code without a failing test first.

The constraint that makes TDD valuable: if you cannot write a test first, it means the requirement is not yet clear enough to implement. Clarify the requirement before writing code.

4 Development Workflow

Every feature follows the same sequence. The critical property: tests are written before implementation, and CI runs them automatically. Nothing can merge without passing through every gate.

5 Contract Testing — The Critical Gap

Most teams test the API in isolation and test the frontend in isolation. Neither test verifies that the two sides agree on the shape of data flowing between them. Contract tests close this gap explicitly.

The risk: without contract tests, a backend developer can rename a field, the unit tests pass, the E2E tests pass (if they test for something else), and the frontend breaks silently in production. The user sees a blank screen or NaN.

Three contract failure modes

01

Field removed

API removes message field → frontend renders undefined in the status banner

02

Type changed

Field changes from string to number → frontend renders NaN where a human-readable label should appear

03

Structure changed

Nested object refactored → TypeError: cannot read property of undefined — crashes the component silently

How to implement contract tests

Producer side (Python/FastAPI): Pydantic schemas ARE the contract. The API endpoint is declared with response_model=BootstrapResponse. FastAPI validates every response against the schema at runtime. Add a unit test asserting the schema’s JSON representation matches the expected shape.

Consumer side (TypeScript): TypeScript interfaces are the contract. Run tsc --noEmit in CI — any code that accesses a field that no longer exists in the interface will fail the build.

Bridge: extract Pydantic JSON schemas (model.model_json_schema()), export them as JSON files, and write a script that compares them to the TypeScript interfaces. Run this script in CI. Alert on any field name or type drift.

# schema_drift_check.py — run in CI after tsc
import json
from fleet_platform.schemas.ansible import BootstrapResponse

schema = BootstrapResponse.model_json_schema()
with open("frontend/src/schemas/BootstrapResponse.json", "w") as f:
    json.dump(schema, f, indent=2)

# CI then runs: npx ts-json-schema-generator --validate
# Any field mismatch exits non-zero → merge blocked

6 Quality Gates — What Blocks Merge

CI is the last line of defence before code hits main. Every gate on this list must be green before a PR can merge. There are no exceptions — removing a gate “just this once” is how technical debt accumulates.

• TypeScript builds with zero errors (tsc --noEmit)
• Unit tests pass — 100% of them, zero allowed failures
• Integration tests pass
• E2E test count must not decrease from previous run
• Coverage threshold maintained (target: ≥ 60% lines)
• Contract schema drift check passes (zero drift)
• No secrets detected in diff (pre-push hook via gitleaks or trufflehog)
• PR description links to an issue and includes a test plan

The pre-push hook is not optional. Secret leaks are permanent. Once a key is committed to a public repo, it must be rotated even if you force-push — git history is replicated immediately. Catch secrets before they leave the machine.

7 Agile Quality

Quality is not a phase after development — it is woven into every sprint event.

Acceptance criteria first

Write acceptance criteria in the issue before assigning it. AC defines done. If you cannot write AC, the work is not ready for development.

Definition of Done

Code written + tests written + no new regressions + PR reviewed and approved. All four. A feature without tests is not done — it is a liability.

Test review is code review

Reviewers check test completeness, not just implementation. Ask: does this test actually catch the bug it claims to catch? Does it test behaviour or implementation? Would this test survive a refactor?

Sprint retrospective metrics

Review flakiness rate, coverage trend, and mutation score every sprint. Trends matter more than snapshots. A declining mutation score signals test quality degradation before coverage numbers show it.

8 Metrics to Track

Metric	Target	How to measure
Unit test coverage	≥ 60%	pytest-cov, —cov-fail-under=60
Mutation score	≥ 70%	mutmut — track surviving mutants per sprint
E2E flakiness rate	< 2%	Track retry count in CI logs over 4-week window
Contract drift	0 fields	Schema comparison script in CI — exit non-zero on drift
MTTR (mean time to recovery)	< 1 sprint	Track time from incident detection to closed issue
Test suite duration	< 10 min	CI timing — fast suites get run; slow suites get skipped

---