# Storage Attachment Workflow v1 Date: 2026-04-28 ## Purpose Define the WEKAFS-first storage attachment workflow before implementing real provider mounts. Storage attachment crosses GPUaaS IAM, project storage quota, provider namespace state, Temporal orchestration, node-agent mount execution, audit, and user-facing read models. It must not be implemented as a direct HTTP handler side effect. This document complements: - `doc/architecture/Storage_Sharing_and_IAM_Model_v1.md` - `doc/architecture/Storage_Provider_Capability_Model_v1.md` - `doc/architecture/Storage_WEKA_Capability_Assessment_v1.md` - `doc/architecture/Node_Agent_Spec.md` - `doc/architecture/Compensation_Matrix.md` ## Core Decision Storage attach and detach are Temporal workflows. The API accepts intent, persists `platform_storage_attachments`, writes audit/outbox state, and starts or signals Temporal. Temporal owns provider reconciliation, node-agent task dispatch, health verification, retries, and compensation. Latency is acceptable because attach is part of provisioning, app launch, or an explicit storage operation. It is not a low-latency read-model request. ## Resource Model Storage creation produces a project-owned storage namespace/bucket record. Storage attachment binds one storage namespace to one runtime target: ```text storage_bucket -> storage_attachment -> workload_id or allocation_id -> node_id once placement is known -> mount_path -> access_mode: read_only | read_write -> write_policy: single_writer | multi_writer -> provider_backend: weka | vast | ... -> state ``` For WEKAFS, the provider reference is a filesystem/path namespace plus a mount plan. For S3 later, the same attachment may compile into credentials or an object-client configuration, but S3 is capability-gated until protocol hosts are enabled. Provider assignment is resolved when the storage namespace is created. A global environment default may select WEKA in kind or single-provider development environments, but production placement must be per storage object and consider region, storage class, protocol, capacity, quota, fabric, and provider health. Attachments inherit the storage object's provider assignment. The first production protocol is `wekafs`. The v3 storage create contract therefore accepts an optional `access_protocol` and defaults it to `wekafs`. This value is persisted on `platform_storage_buckets` and returned in the storage read model so UI, provisioning, and workflow code can distinguish WEKAFS/POSIX storage from future S3/NFS/SMB/CSI surfaces. Protocol support is checked before any create side effect; for example `access_protocol=s3` is rejected until the selected provider assignment advertises S3 capability. ## Attachment State Machine Canonical lifecycle: ```text requested -> prechecking -> grant_applying -> grant_applied -> mounting -> mounted -> detaching -> detached ``` Failure transitions: ```text requested|prechecking -> failed grant_applying -> failed grant_applied|mounting -> failed mounted|detaching -> detach_failed failed -> detaching -> detached detach_failed -> detaching -> detached ``` State meanings: | State | Meaning | |---|---| | `requested` | API accepted attach intent and persisted idempotent state. No provider or node side effect is required yet. | | `prechecking` | Workflow is validating project role, storage grant, quota posture, provider capability, target workload/allocation, node reachability, and mount-path policy. | | `grant_applying` | Workflow is reconciling provider-side grant or namespace ACL state if the backend requires it. For WEKAFS this may be a no-op initially. | | `grant_applied` | Provider/grant prerequisites are satisfied. If mount later fails, compensation must revoke or mark drift where applicable. | | `mounting` | A signed node-agent task has been queued or dispatched. | | `mounted` | Node-agent reported success and health check passed. | | `failed` | Attach failed. Persistent storage remains intact. Retry is allowed after operator/user-visible failure reason is corrected. | | `detaching` | Detach workflow is revoking runtime mount state. | | `detached` | Runtime mount is gone. Persistent storage remains unless a separate delete workflow runs. | | `detach_failed` | Release or detach could not complete after retries. Billing for the workload follows allocation policy; storage remains protected and operator retry is required. | ## Default Mount Policy Do not assume application-level multi-writer safety just because WEKA supports shared POSIX. Recommended defaults: | Purpose | Default access | Default write policy | Rationale | |---|---|---|---| | `dataset` | `read_only` | `multi_writer` not needed | Datasets should be safe to share broadly for training/inference reads. | | `artifact` | `read_only` | `multi_writer` not needed | Model weights/packages should be immutable or versioned. | | `workspace` | `read_write` | `single_writer` | Notebooks and scratch state are usually user/runtime-specific. | | `checkpoint` | `read_write` | `single_writer` | Concurrent checkpoint writers can corrupt application state unless the app is designed for it. | | `generic` | `read_only` unless explicitly requested | `single_writer` when write is requested | Safe default while intent is unclear. | `multi_writer` is a product/app-level allowance, not only a provider capability. The scheduler and app launch precheck must reject unsafe concurrent write attachments unless the storage object and workload profile explicitly allow it. ## Quota Accounting The storage namespace owns quota metadata from creation time: - `quota_bytes` is stored on `platform_storage_buckets`. - usage is provider-reconciled where possible and read-model inferred only for local/dev fallback. - attach precheck blocks new write attachments when a hard quota is already exceeded and warns when near limit. - quota attribution defaults to the owning project. Enforcement can be staged, but the contract and DB must keep quota fields from the first WEKAFS implementation so UI, billing, and scheduler behavior do not need to be redesigned later. User-home storage quota is intentionally deferred in `Storage_Sharing_and_IAM_Model_v1.md`. ## Temporal Workflow Boundaries ### Attach Workflow Workflow name: ```text storage-attachment:{attachment_id} ``` Activities: 1. `LoadAttachmentIntent` 2. `ValidateStorageAccess` 3. `ValidateTargetRuntime` 4. `ValidateProviderCapability` 5. `ValidateQuotaAndWritePolicy` 6. `ApplyProviderGrant` 7. `BuildMountPlan` 8. `DispatchNodeMountTask` 9. `VerifyMountHealth` 10. `MarkAttachmentMounted` Implementation note: node-task dispatch is gated by `STORAGE_ATTACHMENT_NODE_TASKS_ENABLED=false` by default. With the flag off, the worker keeps the local/kind no-op behavior and records `mounted`/`detached` states after compiling the sanitized mount plan. With the flag on, the worker records `mounting`, enqueues a signed `storage.mount_attach` task for the target node, waits for terminal completion through the existing node-task table, and only then records `mounted`. Failures move the attachment to `failed` and emit `storage.attachment.failed`. ### Detach Workflow Workflow name: ```text storage-detach:{attachment_id} ``` Activities: 1. `LoadAttachment` 2. `DispatchNodeUnmountTask` 3. `VerifyUnmount` 4. `RevokeProviderGrantIfEphemeral` 5. `MarkAttachmentDetached` Implementation note: detach follows the same opt-in node-task bridge. With `STORAGE_ATTACHMENT_NODE_TASKS_ENABLED=true`, the worker records `detaching`, enqueues `storage.mount_detach`, waits for terminal completion, and then records `detached`. Failures move the attachment to `detach_failed` and emit `storage.attachment.detach_failed`. ## Node-Agent Task Boundary Node-agent should receive sanitized mount intent, not provider admin credentials. Planned task types: | Task | Purpose | |---|---| | `storage.mount_attach` | Mount a provider-approved filesystem path into the allocation/workload mount path and report health details. | | `storage.mount_detach` | Unmount one attachment by ID and mount path. Never deletes persistent storage. | | `storage.mount_probe` | Read-only probe for mount presence, mode, filesystem type, and basic read/write check where allowed. | The task payload must include: - `attachment_id` - `allocation_id` or `workload_instance_id` - `username` or numeric `uid/gid` - `provider_backend` - `source_ref` as an opaque control-plane-generated mount reference - `mount_path` - `access_mode` - `write_policy` - expected filesystem type where known The task payload must not include: - provider admin credentials - raw WEKA cluster credentials - unbounded shell commands - user-supplied host paths outside approved mount roots - delete instructions for persistent storage ## Compensation Rules Attach failure: - before provider grant: mark `failed`, no provider compensation required - after provider grant before mount: revoke or mark provider grant drift - after mount task dispatched but before success: dispatch best-effort unmount, then mark `failed` - after mounted but health check fails: dispatch unmount, revoke ephemeral provider grant, mark `failed` Detach failure: - persistent storage remains intact - allocation release may continue according to allocation policy only after the mount is known detached or explicitly marked `detach_failed` - `detach_failed` requires operator retry; no direct DB fix is acceptable as the normal path All compensation side effects must be idempotent by `attachment_id`. ## Audit Events Required audit actions: - `storage.attachment.request` - `storage.attachment.precheck` - `storage.attachment.grant_apply` - `storage.attachment.mount` - `storage.attachment.fail` - `storage.attachment.detach` - `storage.attachment.detach_failed` - `storage.attachment.detach_complete` Every audit row must include project, bucket, attachment ID, target runtime, provider backend, state transition, correlation ID, and safe failure reason where present. ## Domain Events The first event slice can be coarse: - `storage.attachment.requested` - `storage.attachment.mounted` - `storage.attachment.failed` - `storage.attachment.detached` - `storage.attachment.detach_failed` Events are emitted through the outbox, not directly from HTTP handlers or Temporal activities. ## API Surface Provider-neutral v3 contracts: - `GET /api/v1/v3/storage/{bucket_id}/attachments` - `POST /api/v1/v3/storage/{bucket_id}/attachments` - `GET /api/v1/v3/storage/{bucket_id}/attachments/{attachment_id}` - `DELETE /api/v1/v3/storage/{bucket_id}/attachments/{attachment_id}` Create and delete are idempotent mutations using `X-Idempotency-Key`. Responses return the attachment read model and workflow state; they do not block until a node mount completes. ## Non-Goals For First Slice - direct S3 STS/client access - per-user persistent home directories - automatic user-home ACL reconciliation when project membership changes - provider-specific UI controls - deleting persistent storage as part of workload release - arbitrary user-supplied mount paths or host bind mounts ## First Implementation Tasks 1. Add `platform_storage_attachments` schema and indexes. 2. Add OpenAPI contracts for attachment read/create/delete. 3. Add provider capability config so WEKA advertises `posix`, `wekafs`, and `csi`, while S3 is hidden until enabled. 4. Implement local/dev no-op attach workflow that records state transitions. 5. Add node-agent task catalog entries and validators without real WEKA mount side effects. 6. Implement WEKAFS mount-plan compiler against `gpuaas-kind-fs`. 7. Add opt-in provisioning-worker node-task dispatch bridge for `storage.mount_attach` and `storage.mount_detach`. 8. Validate attach/detach in kind or a WEKA-reachable node.