Skip to main content

ICW-E6S07T111-Test-Design: FR-078 Comprehensive Install Event-Contract Logging Quality

Task: E6:S07:T111
Related FR: FR-078
Host Task: T111-comprehensive-install-event-contract-logging-and-feedback-quality.md
Created: 2026-04-09

Test Strategy​

Validate contract completeness, event-graph determinism, coverage across CLI/framework installers, normalized outcomes, and redaction safety. Ensure every required step can be reconstructed as intent -> action -> result.

Coverage Matrix​

  1. Contract field completeness (AC1, AC3)

    • Input: install run with representative framework set.
    • Expectation: every required event includes mandatory intent, action, and result fields.

  2. Correlation and joinability (AC2)

    • Input: mixed CLI + framework installer run with nested sub-steps.
    • Expectation: event timeline reconstruction succeeds via install_run_id, step_id, parent/child links.

  3. Decision-branch transparency (AC1, AC4)

    • Input: runs exercising normal path, fallback path, retry path, and skip path.
    • Expectation: branch rationale appears in intent/action with normalized outcome in result.

  4. Outcome normalization (AC4)

    • Input: success, warning, recoverable error, terminal failure runs.
    • Expectation: result consistently includes status, duration, exit code, validation output, error class.

  5. Failure-boundary logging (AC1, AC5)

    • Input: induced mid-run failure in framework installer.
    • Expectation: complete event history retained up to failure boundary + terminal boundary event emitted.

  6. Redaction safety (AC6)

    • Input: controlled secret-like values in env/stderr fixtures.
    • Expectation: persisted events redact sensitive values consistently.

AC-to-Test Mapping​

  • AC1 -> Contract completeness, decision-branch transparency, failure-boundary scenarios
  • AC2 -> Correlation and joinability scenarios
  • AC3 -> Schema omission negative scenarios
  • AC4 -> Outcome normalization scenarios
  • AC5 -> Failure-boundary retention + multi-framework reconstruction scenarios
  • AC6 -> Redaction regression scenarios

Scenario Set (Required)​

  1. Positive path - single framework success
    Expected: complete triad events for all steps with normalized success outputs.

  2. Positive path - multi-framework install
    Expected: deterministic per-framework and run-level event graph joinable into one timeline.

  3. Negative path - schema omission
    Expected: validator/test fails when required event fields are missing.

  4. Negative path - broken parent linkage
    Expected: reconstruction check fails with actionable diagnostics.

  5. Negative path - failure boundary
    Expected: terminal failure event present, prior history intact, no silent truncation.

  6. Negative path - redaction regression
    Expected: test fails if raw secret-like tokens appear in persisted payload.

Failure Modes​

  • Events emitted without one of triad sections.
  • Parent/child links inconsistent across adapters.
  • Framework installer events not correlated to run graph.
  • Result normalization diverges across components.
  • Retry/fallback branches logged without explicit rationale.
  • Redaction bypasses on stderr passthrough.

Red/Green Verification Sequence​

  1. Add failing contract-field completeness tests.
  2. Add failing joinability/reconstruction tests.
  3. Add failing normalization contract tests.
  4. Add failing failure-boundary and redaction tests.
  5. Implement emitters/adapters until green.
  6. Run full install logging regression suite.
  7. Record fixture outcomes for T111 handoff evidence.

Regression Guard Checklist​

  • Required triad fields present in all mandatory events.
  • Timeline reconstruction passes for CLI + installer mixed runs.
  • Failure-boundary events preserved deterministically.
  • Redaction checks pass for known secret patterns.
  • Logging compatibility mode remains functional.

Evidence for Handoff​

  • Test inventory mapped to AC1..AC6.
  • Fixture matrix (success/fallback/retry/failure/redaction) with expected outputs.
  • Pass/fail report for contract, correlation, and redaction gates.

Scope Boundary Note​

This test design intentionally excludes submission transport and governance outcomes, which are covered by FR-079 / E6:S07:T112.