Skip to main content

User Experience Research: Kanban Package Installation UAT

Type: User Experience Research (UXR)
Submitted: 2025-12-10
Submitted By: AI Agent (Cursor) acting as user/client for dev-toolkit
Priority: HIGH
Status: COMPLETE
GitHub Issue: #13
Repository Task: E7:S00:T04
Implementation (synthesis): E7:S05:T04 — see Story 5 T04

Summary​

Comprehensive User Acceptance Testing (UAT) of the ai-dev-kit Kanban package installation process, following package instructions and documenting bugs, usability issues, and workflow observations. Installation completed successfully in canonical_adoption mode after fixing blocking bugs. Semantic matching works but similarity scores are below threshold, preventing intelligent task mapping from executing.

Research Objective​

Primary Question: Does the Kanban package installation process work as intended for a project with pre-existing Kanban structure?

Secondary Questions:

  1. Are the installation instructions clear and accurate?
  2. Do the migration utilities detect, analyze, and migrate existing structures correctly?
  3. Are semantic matching and intelligent task mapping functional?
  4. What bugs or usability issues exist in the installation process?

Methodology​

Research Method: User Acceptance Testing (UAT) - Real-world installation scenario
Participants: AI Agent (Cursor) acting as user/client for dev-toolkit project
Duration: Single session (2025-12-10)
Tools/Platforms: Cursor IDE, Python 3, Git

Research Details:

  • Scenario: Installing Kanban package from ai-dev-kit GitHub into dev-toolkit with pre-existing Kanban structure
  • Existing Structure: 9 epics, 40 stories, 291 tasks
  • User Action: Followed package installation instructions exactly
  • Observation: Installation completed with hybrid mode, but multiple bugs encountered
  • Documentation: Comprehensive bug reports and workflow observations

Key Findings​

Finding 1: Missing Tuple Import (BUG - FIXED)​

  • Error: NameError: name 'Tuple' is not defined in analyze_structure.py
  • Impact: Analysis step fails completely
  • Status: ✅ FIXED (by UAT tester)

Finding 2: Semantic Matcher Regex Error (BUG - NOT FIXED)​

  • Error: Error loading canonical epic definitions: no such group
  • Impact: Semantic matching fails silently, returns 0 matches
  • Status: ❌ NOT FIXED (needs framework fix)

Finding 3: Mode Mismatch Between Scripts (BUG - NOT FIXED)​

  • Error: canonical_adoption mode not supported in migrate_structure.py
  • Impact: Canonical adoption mode cannot be executed
  • Status: ❌ NOT FIXED (needs framework fix)

Finding 4: Incorrect Analysis Recommendations​

  • Issue: Analysis recommends "fresh" mode despite detecting 9 epics, 40 stories, 291 tasks
  • Impact: Misleading recommendations
  • Status: ❌ NOT FIXED

Finding 5: Intelligent Task Mapping Did NOT Execute​

  • Issue: 80% threshold prevents feature from executing (real-world matches are 40-55%)
  • Finding: No actual agentic intelligence present - just deterministic word matching
  • Impact: Feature is non-functional and misnamed
  • Status: ⚠️ DESIGN ISSUE

Recommendations​

Immediate Fixes (HIGH PRIORITY)​

  1. ✅ Fix Semantic Matcher Regex: (COMPLETED by UAT tester)
  2. ✅ Add Canonical Adoption Mode: (COMPLETED by UAT tester)
  3. Fix Analysis Recommendations: Improve recommendation logic
  4. Implement Actual Agentic Intelligence (CRITICAL): Replace deterministic word matching with actual AI/LLM-based analysis
  5. Remove or Justify Arbitrary 80% Threshold (CRITICAL): Either implement agentic intelligence (no threshold) or document rationale

Acceptance Criteria​

  • AC-1: Package can be installed from GitHub ✅
  • AC-2: Installation instructions are clear ✅
  • AC-3: Detection utility works correctly ✅
  • AC-4: Analysis utility runs ✅
  • AC-5: Migration completes successfully ✅
  • AC-6: Semantic matching works correctly ✅ (finds matches, but scores low)
  • AC-7: Canonical adoption mode works ✅ (executes, but no remapping due to low scores)
  • AC-8: Analysis recommendations are accurate ❌
  • AC-9: Intelligent task mapping executes ⚠️ (executes but no remapping due to threshold)

Dependencies​

Blocks:

  • Reliable semantic matching for epic/task migration
  • Canonical adoption workflow
  • Accurate analysis recommendations

Blocked By:

  • Bug fixes in semantic_matcher.py
  • Mode support in migrate_structure.py
  • Analysis logic improvements

Related Work:

  • BR-007: Multiple Bugs in Kanban Package Installation Process (Issue #9)
  • BR-008: Arbitrary 80% Threshold / No Agentic Intelligence (Issue #10)
  • FR-010: Implement Actual Agentic Intelligence for Task Mapping (Issue #11)

Intake Decision​

Intake Status: PENDING
Intake Date: 2025-12-10
Intake By: AI Agent (ai-dev-kit)

Decision Flow Results:

  • Story Match Found: [TBD]

Assigned To:

  • Epic: [TBD]
  • Story: [TBD]
  • Task: [TBD]
  • Version: [TBD]

Kanban Links:

  • Epic: [TBD]
  • Story: [TBD]
  • Task: [TBD]

Notes​

This UAT report documents a real-world installation attempt following the package's own instructions. The installation completed successfully using hybrid mode, but multiple bugs were identified that prevent optimal usage (semantic matching, canonical adoption mode).

Key Takeaway:
The package is functional for basic installation and migration, but advanced features (semantic matching, canonical adoption) need bug fixes before they can be used effectively.

References​

  • Package README: packages/frameworks/kanban/README.md
  • Installation Script: packages/frameworks/kanban/scripts/install_kanban_framework.py
  • BR-007: Multiple Bugs in Kanban Package Installation Process
  • BR-008: Arbitrary 80% Threshold Contradicts Agentic Intelligence Claims
  • FR-010: Implement Actual Agentic Intelligence for Task Mapping

Template Usage:

  • This UXR follows the Kanban Framework UXR template
  • Comprehensive UAT findings documented
  • Clear bug reports and recommendations provided
  • Supporting evidence included

This UXR report is part of the Kanban Framework. See packages/frameworks/kanban/ for complete framework documentation.