P Planekeeper Internal Docs
Internal documentation — not for external distribution.

Planekeeper E2E Test Plan

Manual end-to-end test scenarios for validating the Planekeeper application. This is a living document updated with each release.

Last Updated: 2026-02-06 Current Branch: feature/improvedhealthcheck

Table of Contents

  1. Test Environment Setup
  2. Authentication & Authorization
  3. Onboarding & Organization Management
  4. Dashboard
  5. Gather Jobs
  6. Scrape Jobs
  7. Helm Sync Jobs
  8. Releases
  9. Monitoring Rules
  10. Alert Configs
  11. Alerts
  12. Notification Channels
  13. Notification Rules
  14. Notification Deliveries
  15. Notification Settings & Templates
  16. API Keys
  17. Agents
  18. Settings
  19. Health Check
  20. Cross-Feature Workflows
  21. Security
  22. Release Checklist

How to Use This Document

Each test case follows this format:

FieldDescription
IDUnique identifier (AREA-NNN)
PriorityP0 (blocker), P1 (critical), P2 (important), P3 (nice-to-have)
PreconditionsRequired state before running the test
StepsNumbered actions to perform
Expected ResultWhat should happen
StatusPass / Fail / Blocked / Not Tested
NotesObservations, bugs found, release-specific notes

Status should be updated each release cycle. Use Not Tested for new tests that haven’t been executed yet.

1. Test Environment Setup

Before running tests, ensure the following environment is available:

Prerequisites

  • Docker Compose stack running (docker compose -f go/planekeeper/docker/go/planekeeper/docker/docker-compose.yml up -d)
  • PostgreSQL accessible and migrations applied (auto on server start)
  • At least one organization created (seed data provides “Global” org)
  • At least one API key available for Internal UI login
  • Supabase project configured (for Supabase auth tests) OR legacy API key mode
  • Agent binary running and connected to server
  • A GitHub repository with releases available for gather job testing
  • A Git repository with a version file (e.g., Chart.yaml) for scrape job testing
  • A webhook receiver (e.g., https://webhook.site) for notification testing

Test Data

ItemValue
Internal UI URLhttps://localhost:8443
Client UI URLhttp://localhost:80
API Base URLhttp://localhost:3000
Health endpointhttp://localhost:3000/health
Test GitHub repo(configure per environment)
Test webhook URL(configure per environment)

2. Authentication & Authorization

Supabase Auth Flow

AUTH-001: Email/Password Login

FieldValue
PriorityP0
PreconditionsSupabase configured, approved user account exists
Steps1. Navigate to /login 2. Enter valid email and password 3. Click “Sign In”
Expected ResultRedirected to /dashboard. Session cookie planekeeper_session set. Org cookie planekeeper_org set.
StatusNot Tested

AUTH-002: Email/Password Login - Invalid Credentials

FieldValue
PriorityP0
PreconditionsSupabase configured
Steps1. Navigate to /login 2. Enter invalid email/password 3. Click “Sign In”
Expected ResultError message displayed on login page. No session cookie set. User remains on /login.
StatusNot Tested

AUTH-003: New User Signup (Email/Password)

FieldValue
PriorityP1
PreconditionsSupabase configured
Steps1. Navigate to /auth/signup 2. Fill in email and password 3. Submit signup form
Expected ResultAccount created with is_approved = FALSE. Success message shown: “Please check your email to confirm your account.”
StatusNot Tested

AUTH-004: Pending Approval Flow

FieldValue
PriorityP1
PreconditionsUnapproved user account exists
Steps1. Login as unapproved user 2. Verify /pending-approval page shown 3. Attempt to navigate to /dashboard directly
Expected ResultStep 2: Page shows “pending approval” message with user’s email. Step 3: Redirected back to /pending-approval.
StatusNot Tested

AUTH-005: OAuth Login (Auto-Detected Providers)

FieldValue
PriorityP1
PreconditionsSupabase configured with at least one OAuth provider enabled
Steps1. Navigate to /login 2. Verify OAuth buttons appear only for enabled providers 3. Click an OAuth provider button 4. Complete OAuth flow at provider 5. Verify callback at /auth/callback
Expected ResultOnly enabled providers shown (auto-detected via GET /auth/v1/settings). Tokens extracted from URL hash, exchanged via /auth/token-exchange, redirected to appropriate page (dashboard or onboarding).
StatusNot Tested

AUTH-005a: OAuth Signup (Same Flow as Login)

FieldValue
PriorityP1
PreconditionsSupabase configured with at least one OAuth provider enabled
Steps1. Navigate to /auth/signup 2. Verify OAuth buttons appear (same providers as login page) 3. Click an OAuth provider button 4. Complete OAuth flow at provider
Expected ResultOAuth buttons shown on signup page with “or sign up with” divider. Flow creates new user if not existing. Redirected to onboarding (new user) or dashboard (existing user).
StatusNot Tested

AUTH-005b: OAuth Buttons Hidden When No Providers Enabled

FieldValue
PriorityP2
PreconditionsSupabase configured but no OAuth providers enabled
Steps1. Navigate to /login 2. Navigate to /auth/signup
Expected ResultNo OAuth buttons or divider shown on either page. Email/password form works normally.
StatusNot Tested

AUTH-006: Logout

FieldValue
PriorityP0
PreconditionsLogged-in user session
Steps1. Click “Logout” in navigation 2. Verify redirect to /login 3. Attempt to navigate to /dashboard directly
Expected ResultAll cookies cleared. Step 3: Redirected to /login.
StatusNot Tested

AUTH-007: Session Expiry

FieldValue
PriorityP1
PreconditionsLogged-in user session
Steps1. Login successfully 2. Wait for session/JWT to expire (or manually delete session cookie) 3. Attempt to access protected page
Expected ResultRedirected to /login. No stale data displayed.
StatusNot Tested

Legacy API Key Auth

AUTH-008: API Key Login (Internal UI)

FieldValue
PriorityP0
PreconditionsActive API key exists
Steps1. Navigate to Internal UI /login 2. Enter valid API key 3. Submit
Expected Resultplanekeeper_api_key cookie set. Redirected to dashboard.
StatusNot Tested

AUTH-009: API Key Login - Invalid Key

FieldValue
PriorityP0
PreconditionsNone
Steps1. Navigate to Internal UI /login 2. Enter invalid API key 3. Submit
Expected ResultError message displayed. No cookie set.
StatusNot Tested

AUTH-010: API Key Login - Expired Key

FieldValue
PriorityP1
PreconditionsAPI key exists with past expiration date
Steps1. Attempt login with expired key
Expected ResultAuthentication rejected. Error message displayed.
StatusNot Tested

AUTH-011: API Key Login - Inactive Key

FieldValue
PriorityP1
PreconditionsAPI key exists but is inactive
Steps1. Attempt login with inactive key
Expected ResultAuthentication rejected. Error message displayed.
StatusNot Tested

3. Onboarding & Organization Management

ONB-001: New User Onboarding - Create Organization

FieldValue
PriorityP1
PreconditionsApproved user with no org membership
Steps1. Login as approved user without org 2. Verify redirect to /onboarding 3. Fill in organization name and description 4. Submit “Create Organization”
Expected ResultOrganization created. User added as owner. Org cookie set. Redirected to /dashboard.
StatusNot Tested

ONB-002: New User Onboarding - Accept Invite

FieldValue
PriorityP1
PreconditionsApproved user with pending org invite
Steps1. Login as invited user 2. Verify /onboarding page shows pending invites 3. Click “Accept” on invite
Expected ResultMembership created with correct role. Org cookie set. Redirected to /dashboard.
StatusNot Tested

ONB-003: Organization Switching

FieldValue
PriorityP1
PreconditionsUser is member of multiple organizations
Steps1. Login and verify current org in nav 2. Use org switcher to select different org 3. Verify dashboard data changes
Expected ResultOrg cookie updated. Dashboard shows data for new org. All subsequent API calls use new org context.
StatusNot Tested

4. Dashboard

DASH-001: Dashboard Loads with Metrics

FieldValue
PriorityP0
PreconditionsAuthenticated user, some data exists
Steps1. Navigate to /dashboard
Expected ResultPage loads with metric cards: Alert counts (Critical, High, Moderate, Unacknowledged), Job stats (Total, Pending, Completed, Failed), Release stats (Total, Unique Artifacts, Stable), System Health. Recent jobs table displayed.
StatusNot Tested

DASH-002: Dashboard Scope Filter - Organization

FieldValue
PriorityP1
PreconditionsBoth org-specific and global data exist
Steps1. Navigate to /dashboard (defaults to org scope) 2. Verify only org-scoped data shown
Expected ResultMetrics and recent jobs reflect only the current organization’s data.
StatusNot Tested

DASH-003: Dashboard Scope Filter - Global

FieldValue
PriorityP1
PreconditionsGlobal data exists
Steps1. On dashboard, change scope filter to “Global”
Expected ResultMetrics and recent jobs reflect only global (system) data. URL updates to ?scope=global.
StatusNot Tested

DASH-004: Dashboard Scope Filter - All

FieldValue
PriorityP1
PreconditionsBoth org and global data exist
Steps1. On dashboard, change scope filter to “All”
Expected ResultMetrics and recent jobs show combined org + global data. URL updates to ?scope=all.
StatusNot Tested

DASH-005: Dashboard with No Data

FieldValue
PriorityP2
PreconditionsFresh organization with no jobs, rules, or alerts
Steps1. Navigate to /dashboard
Expected ResultAll metric cards show zero values. Recent jobs table shows empty state. No errors.
StatusNot Tested

5. Gather Jobs

GJ-001: List Gather Jobs

FieldValue
PriorityP0
PreconditionsAuthenticated, gather jobs exist
Steps1. Navigate to /jobs
Expected ResultTable shows gather jobs with columns: Name, Artifact, Source, Status, Schedule, Last Run, Actions. Pagination controls visible if >50 items.
StatusNot Tested

GJ-002: Create Gather Job - GitHub Releases

FieldValue
PriorityP0
PreconditionsAuthenticated
Steps1. Navigate to /jobs 2. Click “Create” 3. Fill: Name, Artifact (e.g., kubernetes/kubernetes), Source Type = github_releases, optional cron schedule 4. Submit
Expected ResultJob created with pending status. Redirected to jobs list. New job appears in table.
StatusNot Tested

GJ-003: Create Gather Job - Validation Error

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Click “Create” 2. Submit with empty required fields
Expected ResultError message displayed. Form not submitted. User stays on form.
StatusNot Tested

GJ-004: Edit Gather Job

FieldValue
PriorityP1
PreconditionsGather job exists
Steps1. Click “Edit” on a gather job 2. Modify name or artifact 3. Submit
Expected ResultJob updated. Redirected to jobs list. Changes reflected in table.
StatusNot Tested

GJ-005: Trigger Gather Job Manually

FieldValue
PriorityP0
PreconditionsGather job exists, agent is running
Steps1. Click “Run” on a gather job 2. Wait for agent to pick up and complete the job
Expected ResultJob status transitions: pending → in_progress → completed. Releases populated after completion.
StatusNot Tested

GJ-006: Delete Gather Job

FieldValue
PriorityP1
PreconditionsGather job exists
Steps1. Click “Delete” on a gather job 2. Confirm deletion in browser dialog
Expected ResultJob removed from list. Confirmation dialog appears before deletion.
StatusNot Tested

GJ-007: Delete Gather Job - Cancel

FieldValue
PriorityP2
PreconditionsGather job exists
Steps1. Click “Delete” on a gather job 2. Cancel the confirmation dialog
Expected ResultJob remains in list. No changes made.
StatusNot Tested

GJ-008: Clear Releases for Gather Job

FieldValue
PriorityP2
PreconditionsGather job exists with releases
Steps1. Click “Clear Releases” on a gather job 2. Confirm action
Expected ResultAll cached releases for this job are deleted. Release count resets to 0.
StatusNot Tested

GJ-009: Gather Jobs - Search Filter

FieldValue
PriorityP2
PreconditionsMultiple gather jobs exist
Steps1. On /jobs, enter a search term 2. Submit search
Expected ResultTable filters to show only jobs matching the search term (by name or artifact).
StatusNot Tested

GJ-010: Gather Jobs - Scope Filter

FieldValue
PriorityP1
PreconditionsOrg-scoped and global gather jobs exist
Steps1. Toggle scope filter between Org, Global, All
Expected ResultTable updates to show jobs matching selected scope.
StatusNot Tested

GJ-011: Gather Jobs - Pagination

FieldValue
PriorityP2
Preconditions>50 gather jobs exist
Steps1. Navigate to /jobs 2. Verify first page shows 50 items 3. Click “Next”
Expected ResultPage 2 loads with remaining items. Previous/Next controls work correctly.
StatusNot Tested

6. Scrape Jobs

SJ-001: List Scrape Jobs

FieldValue
PriorityP0
PreconditionsScrape jobs exist
Steps1. Navigate to /scrape-jobs
Expected ResultTable shows scrape jobs with columns: Name, Repository, Parser Type, Status, Latest Version, Actions.
StatusNot Tested

SJ-002: Create Scrape Job

FieldValue
PriorityP0
PreconditionsAuthenticated
Steps1. Click “Create” on scrape jobs page 2. Fill: Name, Repository URL, File Path, Parse Type (yq/jq/regex), Expression, optional cron 3. Submit
Expected ResultJob created with pending status. Appears in list.
StatusNot Tested

SJ-003: Create Scrape Job - Regex Validation

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Create scrape job with Parse Type = regex 2. Enter invalid regex expression 3. Use “Validate” button (HTMX)
Expected ResultInline validation shows regex is invalid. Form does not submit with invalid regex.
StatusNot Tested

SJ-004: Trigger Scrape Job

FieldValue
PriorityP0
PreconditionsScrape job exists, agent running, repository accessible
Steps1. Click “Run” on a scrape job 2. Wait for completion
Expected ResultJob completes. Version snapshot created. Latest version shown in table.
StatusNot Tested

SJ-005: View Version History

FieldValue
PriorityP1
PreconditionsScrape job with multiple completed runs
Steps1. Navigate to /scrape-jobs/:id/versions
Expected ResultShows chronological list of discovered versions with timestamps. Most recent first.
StatusNot Tested

SJ-006: Edit Scrape Job

FieldValue
PriorityP1
PreconditionsScrape job exists
Steps1. Click “Edit” 2. Modify repository URL or expression 3. Submit
Expected ResultJob updated. Changes reflected in list.
StatusNot Tested

SJ-007: Delete Scrape Job

FieldValue
PriorityP1
PreconditionsScrape job exists
Steps1. Click “Delete” 2. Confirm deletion
Expected ResultJob removed. Confirmation dialog shown before deletion.
StatusNot Tested

7. Helm Sync Jobs

HS-001: Create Helm Sync Job

FieldValue
PriorityP1
PreconditionsAuthenticated, Helm chart repository available
Steps1. Create a Helm sync job with repository URL 2. Submit
Expected ResultHelm sync job created. Appears in list.
StatusNot Tested

HS-002: Trigger Helm Sync

FieldValue
PriorityP1
PreconditionsHelm sync job exists, agent running
Steps1. Click “Run” on Helm sync job 2. Wait for completion
Expected ResultCharts discovered. Child gather jobs created for each chart.
StatusNot Tested

HS-003: Helm Sync - Orphan Cleanup

FieldValue
PriorityP2
PreconditionsHelm sync job with auto_delete_orphans=true, previously synced charts
Steps1. Remove a chart from the Helm repo 2. Re-run Helm sync
Expected ResultChild gather job for removed chart is deleted. Remaining jobs unaffected.
StatusNot Tested

8. Releases

REL-001: List Releases

FieldValue
PriorityP0
PreconditionsGather jobs have completed and fetched releases
Steps1. Navigate to /releases
Expected ResultReleases listed with: Artifact, Version, Release Date, Prerelease flag. Default sort by date descending.
StatusNot Tested

REL-002: Filter by Artifact

FieldValue
PriorityP1
PreconditionsReleases from multiple artifacts exist
Steps1. Select artifact from dropdown filter
Expected ResultOnly releases for selected artifact shown.
StatusNot Tested

REL-003: Filter by Version Substring

FieldValue
PriorityP2
PreconditionsReleases exist
Steps1. Enter version substring (e.g., “1.28”) in version filter
Expected ResultOnly releases matching the version substring shown.
StatusNot Tested

REL-004: Toggle Prerelease Visibility

FieldValue
PriorityP2
PreconditionsBoth stable and prerelease versions exist
Steps1. Toggle “Show Prereleases” filter
Expected ResultWhen off: only stable releases shown. When on: all releases shown including alpha/beta/rc.
StatusNot Tested

REL-005: Sort Order Toggle

FieldValue
PriorityP2
PreconditionsReleases exist
Steps1. Change sort from “Newest” to “Oldest”
Expected ResultRelease list reverses order.
StatusNot Tested

REL-006: View Mode - List vs Grouped

FieldValue
PriorityP2
PreconditionsReleases exist
Steps1. Switch between “List” and “Grouped” view modes
Expected ResultList mode: flat table. Grouped mode: releases grouped by artifact.
StatusNot Tested

REL-007: Releases - Scope Filter

FieldValue
PriorityP1
PreconditionsOrg and global releases exist
Steps1. Toggle scope filter between Org, Global, All
Expected ResultRelease list updates to match selected scope.
StatusNot Tested

9. Monitoring Rules

RULE-001: List Rules

FieldValue
PriorityP0
PreconditionsRules exist
Steps1. Navigate to /rules
Expected ResultTable shows rules with: Name, Type, Thresholds, Active Status, Actions.
StatusNot Tested

RULE-002: Create Rule - Days Behind

FieldValue
PriorityP0
PreconditionsAuthenticated
Steps1. Click “Create” 2. Fill: Name, Type = days_behind, thresholds (moderate, high, critical), optional stable_only 3. Submit
Expected ResultRule created. Appears in list with correct type and thresholds.
StatusNot Tested

RULE-003: Create Rule - Majors Behind

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Create rule with Type = majors_behind and appropriate thresholds
Expected ResultRule created with correct type.
StatusNot Tested

RULE-004: Create Rule - Minors Behind

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Create rule with Type = minors_behind and appropriate thresholds
Expected ResultRule created with correct type.
StatusNot Tested

RULE-005: Toggle Rule Active/Inactive (HTMX)

FieldValue
PriorityP1
PreconditionsActive rule exists
Steps1. Click the active/inactive badge on a rule row
Expected ResultBadge toggles without full page reload. HTMX replaces the row fragment. State persists on page refresh.
StatusNot Tested

RULE-006: Edit Rule

FieldValue
PriorityP1
PreconditionsRule exists
Steps1. Navigate to rule detail 2. Modify thresholds 3. Submit
Expected ResultRule updated. Changes reflected in detail and list views.
StatusNot Tested

RULE-007: Delete Rule

FieldValue
PriorityP1
PreconditionsRule exists (preferably one NOT linked to alert configs)
Steps1. Click “Delete” on rule 2. Confirm
Expected ResultRule removed from list. Confirmation dialog shown.
StatusNot Tested

RULE-008: Trigger Rule Evaluation

FieldValue
PriorityP1
PreconditionsRules, alert configs, scrape job with version, gather job with releases all configured
Steps1. Click “Evaluate Rules” button
Expected ResultRules evaluated against current data. Alerts created/updated as appropriate. Success message shown.
StatusNot Tested

RULE-009: Create Rule - Threshold Validation

FieldValue
PriorityP2
PreconditionsAuthenticated
Steps1. Attempt to create rule with moderate > high or high > critical thresholds
Expected ResultValidation error. Thresholds must be moderate <= high <= critical.
StatusNot Tested

10. Alert Configs

AC-001: List Alert Configs

FieldValue
PriorityP0
PreconditionsAlert configs exist
Steps1. Navigate to /alert-configs
Expected ResultTable shows configs with: Name, Scrape Job, Gather Job, Rule, Active Status.
StatusNot Tested

AC-002: Create Alert Config

FieldValue
PriorityP0
PreconditionsAt least one scrape job, one gather job, one rule exist
Steps1. Click “Create” 2. Select Scrape Job, Gather Job (org or global), Rule from dropdowns 3. Fill name 4. Submit
Expected ResultAlert config created. Dropdown options load correctly from API. Triggers async rule evaluation.
StatusNot Tested

AC-003: Edit Alert Config

FieldValue
PriorityP1
PreconditionsAlert config exists
Steps1. Navigate to alert config detail 2. Modify linked rule or name 3. Submit
Expected ResultConfig updated. Triggers async rule re-evaluation.
StatusNot Tested

AC-004: Toggle Alert Config Active/Inactive

FieldValue
PriorityP1
PreconditionsActive alert config exists
Steps1. Click toggle badge on config row
Expected ResultBadge toggles via HTMX. Inactive configs skip rule evaluation.
StatusNot Tested

AC-005: Delete Alert Config

FieldValue
PriorityP1
PreconditionsAlert config exists
Steps1. Click “Delete” 2. Confirm
Expected ResultConfig removed. Associated alerts resolved (soft-deleted).
StatusNot Tested

11. Alerts

ALT-001: List Alerts

FieldValue
PriorityP0
PreconditionsAlerts have been generated by rule evaluation
Steps1. Navigate to /alerts
Expected ResultTable shows alerts with: Config Name, Severity badge, Discovered Version, Latest Version, Behind By, Acknowledged status, Actions. Summary cards show counts.
StatusNot Tested

ALT-002: Filter Alerts by Severity

FieldValue
PriorityP1
PreconditionsAlerts of different severities exist
Steps1. Select “Critical” from severity filter 2. Then “High” 3. Then “Moderate”
Expected ResultEach filter shows only alerts of that severity. Counts in summary cards update.
StatusNot Tested

ALT-003: Filter Unacknowledged Only

FieldValue
PriorityP1
PreconditionsBoth acknowledged and unacknowledged alerts exist
Steps1. Enable “Unacknowledged only” filter
Expected ResultOnly unacknowledged alerts shown.
StatusNot Tested

ALT-004: Acknowledge Single Alert (HTMX)

FieldValue
PriorityP0
PreconditionsUnacknowledged alert exists
Steps1. Click “Acknowledge” button on alert row
Expected ResultAlert row updates via HTMX (no full reload). Badge changes to “Acknowledged”. alert.acknowledged event dispatched.
StatusNot Tested

ALT-005: Unacknowledge Alert (HTMX)

FieldValue
PriorityP2
PreconditionsAcknowledged alert exists
Steps1. Click “Unacknowledge” on an acknowledged alert row
Expected ResultAlert reverts to unacknowledged state. Row updates via HTMX.
StatusNot Tested

ALT-006: Bulk Acknowledge Alerts

FieldValue
PriorityP1
PreconditionsMultiple unacknowledged alerts exist
Steps1. Select multiple alert checkboxes 2. Click “Acknowledge Selected”
Expected ResultAll selected alerts acknowledged. Page refreshes showing updated statuses.
StatusNot Tested

ALT-007: Alerts Pagination

FieldValue
PriorityP2
Preconditions>50 alerts exist
Steps1. Verify first page shows correct count 2. Navigate to next page
Expected ResultPagination works. Items don’t repeat across pages.
StatusNot Tested

12. Notification Channels

NC-001: List Notification Channels

FieldValue
PriorityP0
PreconditionsNotification channels exist
Steps1. Navigate to /notification-channels
Expected ResultTable shows channels with: Name, Type badge, URL (truncated), Active badge, Last Test result, Actions.
StatusNot Tested

NC-002: Create Webhook Channel

FieldValue
PriorityP0
PreconditionsAuthenticated
Steps1. Click “Create” 2. Fill: Name, URL (use webhook.site), optional headers, optional HMAC secret 3. Submit
Expected ResultChannel created and active. Appears in list.
StatusNot Tested

NC-003: Test Channel Connectivity

FieldValue
PriorityP0
PreconditionsChannel exists with valid webhook URL
Steps1. Click “Test” on channel 2. Wait for test delivery
Expected ResultSuccess message displayed with HTTP status. Webhook receiver gets test payload. Last test timestamp updated.
StatusNot Tested

NC-004: Test Channel - Failed Delivery

FieldValue
PriorityP1
PreconditionsChannel exists with invalid/unreachable URL
Steps1. Click “Test” on channel with bad URL
Expected ResultError message displayed with details (HTTP status, error message). User-friendly formatting via formatTestErrorMessage().
StatusNot Tested

NC-005: Edit Channel

FieldValue
PriorityP1
PreconditionsChannel exists
Steps1. Navigate to channel detail 2. Modify URL or headers 3. Submit
Expected ResultChannel updated. Changes reflected in detail view.
StatusNot Tested

NC-006: Toggle Channel Active/Inactive

FieldValue
PriorityP1
PreconditionsActive channel exists
Steps1. Click active/inactive badge
Expected ResultBadge toggles via HTMX. Inactive channels skip delivery.
StatusNot Tested

NC-007: Delete Channel

FieldValue
PriorityP1
PreconditionsChannel exists
Steps1. Click “Delete” 2. Confirm
Expected ResultChannel removed. Confirmation dialog shown.
StatusNot Tested

NC-008: Create Channel with Custom Headers

FieldValue
PriorityP2
PreconditionsAuthenticated
Steps1. Create channel with custom headers (e.g., Authorization: Bearer token123) 2. Test the channel
Expected ResultChannel created. Test payload includes custom headers (verify at webhook receiver).
StatusNot Tested

NC-009: Create Channel with HMAC Secret

FieldValue
PriorityP2
PreconditionsAuthenticated
Steps1. Create channel with HMAC signing secret 2. Trigger a notification delivery
Expected ResultWebhook payload includes HMAC signature header. Signature is valid against configured secret.
StatusNot Tested

NC-010: SSRF Protection - Private URL Blocked

FieldValue
PriorityP1
PreconditionsNOTIFICATION_ALLOW_PRIVATE_URLS=false (default)
Steps1. Create channel with URL pointing to localhost or RFC1918 address (e.g., http://192.168.1.1/webhook) 2. Test the channel
Expected ResultRequest blocked. Error message indicates private URLs not allowed.
StatusNot Tested

13. Notification Rules

NR-001: List Notification Rules

FieldValue
PriorityP0
PreconditionsNotification rules exist
Steps1. Navigate to /notification-rules
Expected ResultTable shows rules with: Name, Channel, Severity filter, Event filter, Active badge, Actions.
StatusNot Tested

NR-002: Create Notification Rule

FieldValue
PriorityP0
PreconditionsAt least one active notification channel exists
Steps1. Click “Create” 2. Fill: Name, select Channel, set Severity filters (critical/high/moderate), set Event filters (created/escalated/acknowledged/resolved) 3. Submit
Expected ResultRule created and active. Appears in list.
StatusNot Tested

NR-003: Edit Notification Rule

FieldValue
PriorityP1
PreconditionsRule exists
Steps1. Navigate to rule detail 2. Modify severity or event filters 3. Submit
Expected ResultRule updated. Changes reflected in detail view.
StatusNot Tested

NR-004: Toggle Notification Rule

FieldValue
PriorityP1
PreconditionsActive notification rule exists
Steps1. Click active/inactive badge
Expected ResultBadge toggles via HTMX. Inactive rules skip matching.
StatusNot Tested

NR-005: Delete Notification Rule

FieldValue
PriorityP1
PreconditionsRule exists
Steps1. Click “Delete” 2. Confirm
Expected ResultRule removed from list.
StatusNot Tested

14. Notification Deliveries

ND-001: List Deliveries

FieldValue
PriorityP1
PreconditionsNotification deliveries exist
Steps1. Navigate to /notification-deliveries
Expected ResultTable shows deliveries with: Alert, Channel, Status badge, Attempts, Created, Actions.
StatusNot Tested

ND-002: Filter Deliveries by Status

FieldValue
PriorityP2
PreconditionsDeliveries of different statuses exist
Steps1. Select “Failed” from status filter 2. Then “Succeeded” 3. Then “Dead Letter”
Expected ResultTable filters to show only deliveries matching selected status.
StatusNot Tested

ND-003: Retry Dead Letter Delivery

FieldValue
PriorityP1
PreconditionsDead letter delivery exists, channel URL now reachable
Steps1. Navigate to dead letter deliveries 2. Click “Retry” on a dead letter 3. Wait for delivery attempt
Expected ResultDelivery status changes from dead_letter to pending then processes. If webhook now reachable, succeeds.
StatusNot Tested

15. Notification Settings & Templates

NS-001: View Notification Settings

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Navigate to /notification-settings
Expected ResultPage shows global default templates for each category (new_alert, acknowledged, resolved). Shows effective (merged) settings.
StatusNot Tested

NS-002: Edit Notification Template

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Navigate to notification settings 2. Click edit on a template category (e.g., new_alert) 3. Modify template text 4. Save
Expected ResultTemplate updated. Org-level override saved. Used for subsequent deliveries.
StatusNot Tested

NS-003: Reset Template to Default

FieldValue
PriorityP2
PreconditionsOrg-level template override exists
Steps1. Navigate to template with org override 2. Click “Reset to Default”
Expected ResultOrg override removed. Template reverts to global default.
StatusNot Tested

NS-004: Template Variable Rendering

FieldValue
PriorityP2
PreconditionsTemplate with variables configured
Steps1. Configure template using variables: {{.Alert.Severity}}, {{.Alert.ConfigName}}, {{.AcknowledgeURL}} 2. Trigger an alert notification 3. Check webhook payload
Expected ResultVariables replaced with actual values. No raw {{...}} in payload.
StatusNot Tested

16. API Keys

AK-001: List API Keys

FieldValue
PriorityP0
PreconditionsAPI keys exist
Steps1. Navigate to /api-keys
Expected ResultTable shows: Name, Key ID (prefix), Last Used, Expires, Active badge, Actions. Summary cards: Total, Active, Inactive.
StatusNot Tested

AK-002: Create API Key

FieldValue
PriorityP0
PreconditionsAuthenticated
Steps1. Click “Create” 2. Fill: Name, optional Description, optional Expiration date 3. Submit
Expected ResultKey created. Plaintext key shown ONCE (format: pk_<id>_<secret>). Warning to store securely displayed. Key never retrievable again.
StatusNot Tested

AK-003: API Key - Plaintext Not Retrievable

FieldValue
PriorityP0
PreconditionsAPI key just created
Steps1. Note the plaintext key 2. Navigate away 3. Return to key detail
Expected ResultOnly key prefix shown (pk_<id>_...). Full secret NOT displayed.
StatusNot Tested

AK-004: Toggle API Key Active/Inactive

FieldValue
PriorityP1
PreconditionsActive API key exists
Steps1. Click active/inactive badge on key row
Expected ResultBadge toggles via HTMX. Inactive key immediately rejected by auth middleware on subsequent use.
StatusNot Tested

AK-005: Deactivate API Key

FieldValue
PriorityP1
PreconditionsActive API key exists
Steps1. Navigate to key detail 2. Click “Deactivate” 3. Confirm
Expected ResultKey soft-deleted. Cannot be reactivated. Audit trail preserved.
StatusNot Tested

AK-006: Edit API Key

FieldValue
PriorityP2
PreconditionsAPI key exists
Steps1. Navigate to key detail 2. Edit name or description 3. Submit
Expected ResultMetadata updated. Key itself unchanged.
StatusNot Tested

AK-007: API Key Expiration Enforcement

FieldValue
PriorityP1
PreconditionsAPI key with past expiration
Steps1. Attempt API call with expired key (via curl or agent config)
Expected ResultRequest rejected with 401. Key shows as expired in UI.
StatusNot Tested

17. Agents

AGT-001: List Agents

FieldValue
PriorityP0
PreconditionsAt least one agent has sent a heartbeat
Steps1. Navigate to /agents
Expected ResultTable shows agents with: UUID, Hostname, Health badge, Last Heartbeat, Version, Capabilities.
StatusNot Tested

AGT-002: Agent Health Status

FieldValue
PriorityP1
PreconditionsAgent is running and sending heartbeats
Steps1. View agent in list 2. Stop the agent 3. Wait >2 minutes 4. Refresh page
Expected ResultStep 2: Health badge shows “Healthy”. Step 4: Health badge changes to “Unhealthy” or “Unknown”.
StatusNot Tested

AGT-003: View Agent Submission Logs

FieldValue
PriorityP2
PreconditionsAgent has submitted task results
Steps1. Click on agent to expand submission logs (HTMX)
Expected ResultLogs show recent task submissions with success/failure status and timestamps.
StatusNot Tested

18. Settings

SET-001: View Settings

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Navigate to /settings
Expected ResultShows settings list with effective values (global defaults + org overrides). Indicates which are org-overridden.
StatusNot Tested

SET-002: Edit Organization Setting Override

FieldValue
PriorityP1
PreconditionsAuthenticated
Steps1. Click “Edit” on a setting (HTMX inline edit) 2. Change value 3. Save
Expected ResultOrg override saved. Setting shows new effective value. Edit form replaces with view mode via HTMX.
StatusNot Tested

SET-003: Clear Organization Setting Override

FieldValue
PriorityP2
PreconditionsOrg override exists on a setting
Steps1. Click “Clear Override” on a setting 2. Confirm
Expected ResultOrg override removed. Setting reverts to global default value.
StatusNot Tested

19. Health Check

HC-001: Health Endpoint Accessible

FieldValue
PriorityP0
PreconditionsServer running
Steps1. curl http://localhost:3000/health
Expected ResultReturns 200 with {"status":"healthy"} (or similar). No auth required.
StatusNot Tested

HC-002: Health Endpoint - Database Down

FieldValue
PriorityP1
PreconditionsServer running, database stopped
Steps1. Stop PostgreSQL container 2. curl http://localhost:3000/health
Expected ResultReturns unhealthy status with database connectivity error.
StatusNot Tested

20. Cross-Feature Workflows

These tests validate the end-to-end integration between multiple features.

E2E-001: Full Alert Lifecycle

FieldValue
PriorityP0
PreconditionsClean environment with no existing alerts
Steps1. Create a gather job for a GitHub repo with many releases 2. Run the gather job → wait for completion 3. Create a scrape job for a repo with an older version 4. Run the scrape job → wait for completion 5. Create a days_behind rule with low thresholds 6. Create an alert config linking the scrape job, gather job, and rule 7. Verify alert is created on /alerts page 8. Check alert severity matches expected threshold 9. Acknowledge the alert 10. Update the scraped repo to latest version 11. Re-run the scrape job 12. Verify alert is resolved
Expected ResultFull lifecycle: Creation → Severity assignment → Acknowledgment → Resolution. Each state visible in UI.
StatusNot Tested

E2E-002: Full Notification Lifecycle

FieldValue
PriorityP0
PreconditionsWorking webhook receiver (webhook.site)
Steps1. Create a notification channel (webhook to webhook.site) 2. Test the channel → verify success 3. Create a notification rule routing critical alerts to the channel 4. Trigger an alert (via E2E-001 steps or rule evaluation) 5. Check /notification-deliveries for delivery attempt 6. Verify webhook receiver got the payload 7. Check payload contains correct alert data and acknowledge URL 8. Call acknowledge URL from webhook payload 9. Verify alert acknowledged in /alerts
Expected ResultEnd-to-end: Alert → Notification Rule Match → Delivery → Webhook Received → External Acknowledgment
StatusNot Tested

E2E-003: Smart Acknowledgment Reset

FieldValue
PriorityP1
PreconditionsAlert config with active alert
Steps1. Acknowledge an existing alert 2. Re-run the scrape job (same version discovered) 3. Trigger rule evaluation 4. Verify alert stays acknowledged 5. Change the scrape target to discover a DIFFERENT version 6. Re-run the scrape job 7. Trigger rule evaluation 8. Verify alert acknowledgment is RESET
Expected ResultStep 4: is_acknowledged remains TRUE (same version). Step 8: is_acknowledged reset to FALSE (different version).
StatusNot Tested

E2E-004: Alert Escalation

FieldValue
PriorityP1
PreconditionsAlert config with active moderate-severity alert
Steps1. Start with a moderate alert (version slightly behind) 2. Wait for more releases to be gathered (or add releases to make version further behind) 3. Trigger rule evaluation 4. Verify alert severity escalates to high or critical 5. Check that alert.escalated event is dispatched
Expected ResultAlert severity increases. Escalation event triggers notification if rules match.
StatusNot Tested

E2E-005: Job Failure and Recovery

FieldValue
PriorityP1
PreconditionsAgent running
Steps1. Create a gather job with an invalid repository (e.g., nonexistent/repo) 2. Run the job 3. Verify job fails 4. Update the job with a valid repository 5. Run again 6. Verify job completes successfully
Expected ResultStep 3: Job status = failed. Step 6: Job status = completed, releases populated.
StatusNot Tested

E2E-006: Orphan Job Recovery

FieldValue
PriorityP1
PreconditionsJob in in_progress state
Steps1. Trigger a job 2. Kill the agent while job is in_progress 3. Wait >1 hour (or adjust timeout for testing) 4. Verify job status resets to pending 5. Restart agent 6. Verify job is picked up and completes
Expected ResultStale job detection resets orphaned jobs. Agent picks them up on restart.
StatusNot Tested

E2E-007: Multi-Tenant Data Isolation

FieldValue
PriorityP0
PreconditionsTwo organizations with separate data
Steps1. Login as Org A user 2. Create jobs, rules, alerts in Org A 3. Switch to Org B (or login as Org B user) 4. Verify Org A’s data is NOT visible 5. Create data in Org B 6. Switch back to Org A 7. Verify Org B’s data is NOT visible
Expected ResultComplete data isolation between organizations. No cross-tenant data leakage.
StatusNot Tested

E2E-008: Stable-Only Rule Filtering

FieldValue
PriorityP2
PreconditionsGather job with both stable and prerelease versions
Steps1. Create rule with stable_only = true 2. Create alert config 3. Trigger evaluation 4. Verify prerelease versions (alpha, beta, rc) are excluded from comparison 5. Create same rule with stable_only = false 6. Verify prereleases are included
Expected ResultWith stable_only: comparison uses only stable releases. Without: all releases considered.
StatusNot Tested

E2E-009: Notification Retry and Dead Letter

FieldValue
PriorityP1
PreconditionsChannel with unreachable URL
Steps1. Create channel with intentionally broken URL 2. Create notification rule 3. Trigger alert (creates delivery) 4. Watch delivery status over time: pending → failed → retry → dead_letter 5. Fix the channel URL 6. Retry the dead letter delivery 7. Verify delivery succeeds
Expected ResultRetry backoff observed. After max retries, delivery enters dead_letter. Manual retry succeeds after URL fix.
StatusNot Tested

E2E-010: Cron-Scheduled Job Execution

FieldValue
PriorityP1
PreconditionsAgent running
Steps1. Create gather job with cron schedule (e.g., */2 * * * * for every 2 minutes) 2. Trigger initial run 3. Wait for cron activation 4. Verify job runs again automatically
Expected ResultJob automatically re-enters pending at scheduled time. Agent picks up and executes.
StatusNot Tested

21. Security

SEC-001: Unauthenticated Access Blocked

FieldValue
PriorityP0
PreconditionsNo active session
Steps1. Clear all cookies 2. Attempt to access /dashboard, /jobs, /alerts, /api-keys directly
Expected ResultAll protected routes redirect to /login. No data leakage.
StatusNot Tested

SEC-002: API Key Not Logged in Plaintext

FieldValue
PriorityP0
PreconditionsAccess to server logs
Steps1. Create an API key 2. Use it in API calls 3. Search server logs for the plaintext secret
Expected ResultPlaintext API key secret never appears in logs. Only key ID/prefix logged.
StatusNot Tested

SEC-003: API Key Argon2id Storage

FieldValue
PriorityP0
PreconditionsDatabase access
Steps1. Create an API key 2. Query api_keys table directly 3. Check key_hash column
Expected ResultKey stored as Argon2id hash. Raw secret not in any database column.
StatusNot Tested

SEC-004: Cross-Organization API Access

FieldValue
PriorityP0
PreconditionsTwo organizations, API keys for each
Steps1. Use Org A’s API key 2. Attempt to access Org B’s resources via API (modify X-Organization-Id header)
Expected ResultRequest rejected or returns only Org A’s data. No cross-tenant access.
StatusNot Tested
FieldValue
PriorityP1
PreconditionsHTTPS enabled
Steps1. Login and inspect planekeeper_session cookie in browser dev tools
Expected ResultCookie has: HttpOnly, Secure (when AUTH_COOKIE_SECURE=true), SameSite=Lax. Cookie value is encrypted (AES-GCM), not readable base64 JWT.
StatusNot Tested

SEC-006: Rate Limiting

FieldValue
PriorityP1
PreconditionsServer running with rate limiting enabled
Steps1. Send >100 API requests within 60 seconds (or configured limits)
Expected ResultAfter limit exceeded, server returns 429 Too Many Requests. Requests resume after window resets.
StatusNot Tested

SEC-007: SSRF Protection on Webhooks

FieldValue
PriorityP1
PreconditionsNOTIFICATION_ALLOW_PRIVATE_URLS=false
Steps1. Create channel with http://127.0.0.1:8080/hook 2. Create channel with http://10.0.0.1/hook 3. Create channel with http://192.168.1.1/hook 4. Test each channel
Expected ResultAll blocked with appropriate error message. No requests sent to private addresses.
StatusNot Tested

SEC-008: Webhook HMAC Signature Validation

FieldValue
PriorityP2
PreconditionsChannel configured with HMAC secret
Steps1. Configure channel with known HMAC secret 2. Trigger delivery 3. Capture webhook request at receiver 4. Validate HMAC signature header against payload using the secret
Expected ResultHMAC signature present in headers. Signature validates correctly.
StatusNot Tested

22. Release Checklist

Use this checklist before each release. Tests marked P0 are blockers.

Pre-Release Smoke Test (All P0 Tests)

  • AUTH-001: Email/Password Login
  • AUTH-002: Invalid Credentials Rejected
  • AUTH-006: Logout
  • AUTH-008: API Key Login (Internal UI)
  • AUTH-009: Invalid API Key Rejected
  • DASH-001: Dashboard Loads
  • GJ-001: List Gather Jobs
  • GJ-002: Create Gather Job
  • GJ-005: Trigger Gather Job
  • SJ-001: List Scrape Jobs
  • SJ-002: Create Scrape Job
  • SJ-004: Trigger Scrape Job
  • REL-001: List Releases
  • RULE-001: List Rules
  • RULE-002: Create Rule
  • AC-001: List Alert Configs
  • AC-002: Create Alert Config
  • ALT-001: List Alerts
  • ALT-004: Acknowledge Alert
  • NC-001: List Notification Channels
  • NC-002: Create Webhook Channel
  • NC-003: Test Channel
  • NR-001: List Notification Rules
  • NR-002: Create Notification Rule
  • AK-001: List API Keys
  • AK-002: Create API Key
  • AK-003: Plaintext Not Retrievable
  • AGT-001: List Agents
  • HC-001: Health Endpoint
  • E2E-001: Full Alert Lifecycle
  • E2E-002: Full Notification Lifecycle
  • E2E-007: Multi-Tenant Data Isolation
  • SEC-001: Unauthenticated Access Blocked
  • SEC-002: API Key Not Logged
  • SEC-003: API Key Argon2id Storage
  • SEC-004: Cross-Org Access Blocked

Full Regression (All P0 + P1 Tests)

Run all P0 tests above plus:

  • AUTH-003: Signup
  • AUTH-004: Pending Approval
  • AUTH-005: OAuth Login
  • AUTH-007: Session Expiry
  • AUTH-010: Expired Key Rejected
  • AUTH-011: Inactive Key Rejected
  • ONB-001: Create Organization
  • ONB-002: Accept Invite
  • ONB-003: Org Switching
  • DASH-002: Scope Filter - Org
  • DASH-003: Scope Filter - Global
  • DASH-004: Scope Filter - All
  • GJ-004: Edit Gather Job
  • GJ-006: Delete Gather Job
  • GJ-010: Scope Filter
  • SJ-005: Version History
  • SJ-006: Edit Scrape Job
  • SJ-007: Delete Scrape Job
  • HS-001: Create Helm Sync
  • HS-002: Trigger Helm Sync
  • REL-002: Filter by Artifact
  • REL-007: Scope Filter
  • RULE-005: Toggle Rule
  • RULE-006: Edit Rule
  • RULE-007: Delete Rule
  • RULE-008: Trigger Evaluation
  • AC-003: Edit Alert Config
  • AC-004: Toggle Alert Config
  • AC-005: Delete Alert Config
  • ALT-002: Filter by Severity
  • ALT-003: Filter Unacknowledged
  • ALT-006: Bulk Acknowledge
  • NC-004: Test Channel - Failed
  • NC-005: Edit Channel
  • NC-006: Toggle Channel
  • NC-007: Delete Channel
  • NC-010: SSRF Protection
  • NR-003: Edit Notification Rule
  • NR-004: Toggle Notification Rule
  • NR-005: Delete Notification Rule
  • ND-001: List Deliveries
  • ND-003: Retry Dead Letter
  • NS-001: View Settings
  • NS-002: Edit Template
  • AK-004: Toggle API Key
  • AK-005: Deactivate Key
  • AK-007: Key Expiration
  • AGT-002: Agent Health Status
  • SET-001: View Settings
  • SET-002: Edit Setting Override
  • HC-002: Health - DB Down
  • E2E-003: Smart Ack Reset
  • E2E-004: Alert Escalation
  • E2E-005: Job Failure Recovery
  • E2E-006: Orphan Recovery
  • E2E-009: Notification Retry
  • E2E-010: Cron Scheduling
  • SEC-005: Cookie Security
  • SEC-006: Rate Limiting
  • SEC-007: SSRF Protection