Break technical evaluation into verifiable steps

Required outputs and decision ownership

Before evaluation starts, record who runs the assessment, reviews evidence, and makes the production decision. A technical test result is not production approval by itself.

• The evaluation owner controls scope, schedule, scenario identifiers, and the evidence repository.
• The product owner approves the expected workflow and which deviations are acceptable.
• The platform owner approves topology, capacity, observability, backup and recovery, and rollback evidence.
• The security approver reviews role boundaries, secret handling, policy decisions, audit trails, and open risks.
• The final package contains the scope record, executed scenarios, evidence index, findings, retests, and signed decision.

Entry gate

If any prerequisite below is missing, record the affected scenario as BLOCKED rather than failed; do not confuse a missing prerequisite with product behavior.

• An isolated environment without production data, immutable image and chart references, and a recorded source-evidence revision
• Working PostgreSQL, selected filesystem or S3 storage, DNS, ingress and TLS, and required outbound access
• Separate administrator, operator, and limited-user accounts, with an approved secret source for tokens and passwords
• Controlled artifact and CycloneDX test inputs, expected checksums, and a safe disposal plan
• Access to read audit, task, scan, package-usage, and outbound records, with synchronized clocks

Freeze scope before starting

Record the test environment and acceptance matrix before execution so results remain comparable. If scope changes, open a new evaluation revision instead of silently updating the existing result.

• Organization, project, repository name and type, format, real client, and client version
• Application image digest, chart and values revision, PostgreSQL version, storage profile, and network topology
• Test users, roles, access-policy scope, identity provider, and secret source
• Expected client exit code, PASS, WARN, BLOCK, or NOT_EVALUATED result, and record identifiers to collect
• When the organization has not supplied capacity and timing targets, do not invent them as product assumptions; record them as open requirements.

Evaluation order

Each phase produces the entry conditions and evidence list for the next phase. When a critical boundary fails, classify the finding before running dependent scenarios.

1. Inventory package clients, formats, and Hosted, Proxy, and Group repository modes.
2. Map the application, PostgreSQL, storage, identity, ingress and TLS, and outbound-network topology.
3. Exercise identity and authorization boundaries before repository and artifact happy paths.
4. Exercise asynchronous or security-sensitive flows such as proxying, SBOMs, policy, risk acceptance, cleanup, and storage.
5. Review dependency failure, retry, rollback or recovery, and operational visibility.
6. Retest findings and do not publish an outcome before decision owners review the evidence index.

Minimum scenario catalog

Give every scenario an immutable identifier and execute successful, denied, and recoverable-failure paths as separate records. Out-of-scope scenarios must remain visible with an N/A rationale.

• EV-001 Identity and RBAC: sign-in, valid role, insufficient role, organization, project, and repository scope separation, and sign-out
• EV-010 Hosted: publish with the real client, metadata visibility, download, checksum comparison, republish behavior, and unauthorized request
• EV-020 Proxy: cache miss, upstream fetch, cache hit, upstream outage, allowed or denied request, and record trail
• EV-030 Group: member order, first match, missing package, authorization differences, and the same coordinate in multiple members
• EV-040 SBOM and policy: known CycloneDX input, findings, PASS, WARN, BLOCK, risk acceptance, reevaluation, and CI exit code
• EV-050 Lifecycle: cleanup dry run, execution boundary, archive and restore or re-upload, and storage-transfer counters
• EV-060 Operations: health and readiness, task failure and retry trail, and reconciliation of Audit, Package Usage, Operation, and Outbound records

Format and repository scope matrix

Listing a format does not mean every repository mode and every client version is accepted. Validate every production combination as an independent row with the real client.

• Row key: format, client and version, operating system, Hosted, Proxy, or Group mode, and authentication method
• For Hosted, record publish, resolve or download, metadata, and overwrite or redeploy behavior.
• For Proxy, record the upstream URL, cache miss and hit, metadata, timeout, and upstream-failure behavior.
• For Group, record member order, first match, fallback, authorization, and conflicting-coordinate behavior.
• Separate unsupported operations explicitly as N/A or FAIL, including Raw and limited-compatibility cases; do not silently remove them from scope.

Security and governance tests

A positive access test alone is insufficient. Compare allowed and denied actors, policy outcomes, and server-side record trails for the same object.

• Exercise organization, project, and repository operations for administrator, operator, and limited users with separate accounts.
• Use an API token with the least required scope and never write its value to screenshots, commands, logs, or evidence files.
• Reconcile PASS, WARN, BLOCK, and, when context is absent, NOT_EVALUATED with client behavior, policy evaluation, and audit or operation records.
• For risk acceptance, validate scope, rationale, reference, actor, expiry, revocation or expiry, and reevaluation behavior.
• Sample logs to verify that they contain no authorization headers, cookies, passwords, tokens, registry credentials, or unnecessary personal data.

SBOM, policy, and CI gate

Retain the checksum of a controlled CycloneDX input and verify the scan, component, finding, and policy-evaluation chain produced from it under one test identity.

• Use at least three separate inputs expected to produce clean, warning, and blocked outcomes, and record expectations before execution.
• Record the CLI configuration source, CLI version, project and environment scope, and actual exit code.
• For asynchronous evaluation, inspect timeout and polling behavior and the records created by resubmitting the same input.
• After risk acceptance or a policy change, link the new evaluation without deleting the historical result.

Data lifecycle and operational resilience

Cleanup, storage transfer, backup and recovery, and rollback are different evidence levels. A completed task alone does not prove data integrity or recoverability.

• Compare cleanup dry-run candidates with scope and retention rules, then reconcile metadata, objects, and counters after execution.
• For storage transfer, compare candidate, processed, failed, and byte counters with source and destination inventory and retain visibility into failed items.
• Do not pass backup and restore unless PostgreSQL, artifact objects, and raw SBOM data are restored to the same recovery point.
• During dependency failures, record readiness, user-visible errors, task retry or checkpoint behavior, and recovery duration.
• If controlled rollback or forward recovery has not been rehearsed, retain production impact and the decision owner as an explicit evidence gap.

Evidence pack

Scenario evidence must not consist only of a screenshot. Combine repeatable input, timestamp, expected and actual results, and relevant system records under one test identity.

• Executed secret-free commands or steps, client version, UTC timestamp, and actual exit code or output
• Checksum or digest for the artifact, image, or SBOM and an access-restricted test copy of the source file
• Correlation, task, scan, policy-evaluation, audit, and, when applicable, package-usage or outbound identifiers
• Authorization denial, validation failure, dependency failure, and recovery results alongside the successful path
• Severity, production impact, owner, required evidence, target date, and retest identifier for every open item

Downloadable evaluation plan

Copy the template for each evaluation, fill in its placeholders, and never add secret values to the file. The actual and result fields must remain NOT_RUN until execution.

evaluation:
  id: "SK-EVAL-YYYY-NNN"
  name: "Smart Kubaba technical evaluation"
  owner: "<evaluation-owner>"
  approvers:
    product: "<product-owner>"
    platform: "<platform-owner>"
    security: "<security-approver>"
  window:
    startsAt: "YYYY-MM-DDTHH:mm:ssZ"
    endsAt: "YYYY-MM-DDTHH:mm:ssZ"
  evidenceRoot: "<restricted-evidence-location>"

target:
  environment: "<evaluation-environment>"
  sourceEvidence: "c1be012"
  imageDigest: "<immutable-image-digest>"
  chartVersion: "<chart-version>"
  databaseVersion: "<postgresql-version>"
  storageProfile: "<filesystem-or-s3-profile>"
  ingressAndTlsProfile: "<ingress-and-ca-profile>"

clients:
  - id: "CLIENT-01"
    format: "<maven-npm-docker-pypi-etc>"
    repositoryMode: "<hosted-proxy-group>"
    client: "<real-client-name>"
    version: "<client-version>"
    operatingSystem: "<os-and-version>"

scenarios:
  - id: "EV-001"
    title: "<scenario-title>"
    owner: "<executor>"
    prerequisites: ["<prerequisite>"]
    input: "<artifact-sbom-or-request-reference>"
    expected: "<observable-outcome>"
    actual: "NOT_RUN"
    result: "NOT_RUN" # PASS | FAIL | BLOCKED | NOT_RUN
    evidence:
      clientOutput: "<path-or-record-id>"
      checksum: "<sha256-when-applicable>"
      taskOrScanId: "<identifier-when-applicable>"
      auditOrCorrelationId: "<identifier-when-applicable>"
    finding: "<finding-id-or-null>"

decision:
  outcome: "NOT_DECIDED" # PROCEED | CONDITIONAL_PROCEED | DO_NOT_PROCEED
  conditions: []
  decidedAt: null
  approvedBy: []

Findings, retest, and closure

Do not delete a failed or blocked control. Preserve the original evidence and add remediation and retest as new linked records.

• P0 or blocker: privilege bypass, data loss or integrity risk, failure of a critical client flow, or absence of a safe recovery path
• P1 or high: a material security, operational, or observability gap in target scope that must close before production
• P2 or conditional: a gap with explicit impact, compensating control, owner, and closure date that decision owners may accept in writing
• The retest record must include the original scenario or finding ID, remediation applied, new evidence, and the person approving the result.

Decision standard

Do not accept a capability only because its happy path works. In-scope clients, authorization boundaries, failure behavior, rollback plan, and operational ownership must be approved together. Any item without evidence remains a gap rather than an acceptance.

• PROCEED: every scope row ran, no P0 or P1 remains open, recovery boundaries are known, and product, platform, and security owners approved the evidence.
• CONDITIONAL_PROCEED: only conditional items with written impact, compensating controls, owners, and closure dates remain open.
• DO_NOT_PROCEED: there is an authorization or data-integrity breach, critical-flow failure, unvalidated recovery, or insufficient evidence for a decision.
• A numeric score cannot offset a critical control; publish the decision together with scope, findings, and approval records.

Revalidation triggers

Approval applies to a specific release and topology. Reopen scenarios affected by the following changes and preserve prior evidence as Superseded.

• Application image, Helm chart or values, database migration, or PostgreSQL version
• Storage backend, bucket or path layout, ingress and TLS, DNS, proxy, or outbound-network policy
• Identity provider, role or access policy, token scope, or secret-rotation method
• Package client or version, repository mode, upstream source, policy, or intelligence data
• Production load profile, capacity target, RTO or RPO requirement, or operational owner

Live demo validation — August 1, 2026

On the demo running the audited source snapshot, Repository, Artifact, Sources, Scan History, Policy, Risk Acceptances, intelligence and cleanup-storage tasks, and Audit, Package Usage, and Outbound record screens loaded successfully in an authenticated read-only session. This confirms that the evaluation surfaces exist; because the review produced no controlled test input, authorization denial, fault injection, rollback, or owner approval, the page remains Evaluation guidance.