# Node Agent Spec

Status:
- `MVP canonical` (security baseline)
- This specification replaces direct control-plane SSH provisioning for node lifecycle and allocation provisioning flows.

## 1. Purpose

`cmd/node-agent` is a Go binary deployed on every GPU node. It replaces direct
SSH-based provisioning from the control plane with a pull-based, mTLS-authenticated,
typed-task model.

**Security goal**: the control plane never executes arbitrary commands on nodes.
All node-side operations are performed by compiled, audited handlers in the agent binary,
dispatched only on receipt of a cryptographically signed, schema-validated task payload.
If the task type is not in the compiled catalog, the agent rejects it with no side effects.

---

## 2. Architecture

Pull model — node agent polls the control plane:

```
Control Plane (cmd/api)
  ├── node_tasks table (Postgres): queued tasks per node
  └── Internal task API (mTLS)
        ↑ poll (every 5 s)
Node Agent (cmd/node-agent)
  ├── Enrollment on first boot (one-time token via cloud-init)
  ├── Cert renewal loop (T-1h, autonomous)
  ├── Task poll → signature verify → catalog dispatch → result report
  └── Privilege model: minimal per-task sudo (no blanket root)
```

The control plane never initiates connections to nodes. Tasks are queued in Postgres;
nodes retrieve them. No SSH from control plane to node after enrollment.

Terminal access is modeled as:
- `terminal.open` (signed task) for session authorization + PTY attach intent.
- bidirectional stream frames for stdin/stdout/stderr + close.

Terminal stream frames are **not** modeled as per-keystroke tasks.
Reconnect after stream drop is a full reopen (new token + new `terminal.open` task).

This means the current node-agent binary contains two execution domains:
- lifecycle/task execution: enroll, renew, poll, verify, dispatch, result post
- terminal execution: session bootstrap via `terminal.open` plus long-lived stream relay

They intentionally share node identity/bootstrap and verifier material today, but they do not share the same transport or failure model. The architecture must preserve this seam so terminal runtime can be split later without redesigning lifecycle task execution.

---

## 3. Network Boundary

**The node agent has exactly one network destination: `api.internal`.**

The node never connects to step-ca, Postgres, Redis, Temporal, or NATS directly.
All of those are internal to the control plane. From the node's perspective they do not
exist — the API is the single interface to everything.

This applies to enrollment, cert renewal, task polling, and result reporting.

Terminal transport note:
- terminal bootstrap is still platform-controlled and authenticated by the control plane,
- but some environments may expose a distinct node-facing endpoint for long-lived stream traffic,
- this does not weaken the single-authority model; it only keeps interactive stream transport separate from lifecycle request/reply traffic.

## 4. Enrollment Flow

Full enrollment sequence: `doc/architecture/PKI_Spec.md §6`.

Summary:
1. Bootstrap bundle delivers: `node_id`, enrollment token, root CA fingerprint, API URL.
   No CA URL — the node never talks to the CA directly.
2. Agent generates Ed25519 keypair; constructs CSR.
3. Calls `POST /internal/v1/nodes/enroll` on the API with the enrollment token.
   The API proxies to step-ca internally and returns the signed cert + CA bundle.
4. Enters task polling loop. (Enrollment confirmation is embedded in the API's step 3
   response — no separate ack call needed.)

Enrollment token delivery (MVP canonical): bootstrap bundle generated by control plane
after admin node registration.
Supported delivery modes:
- `manual`: operator downloads bundle and installs it on the node.
- `maas`: bundle injected through provider automation/cloud-init.
Initial bootstrap token semantics are identical in both modes: node-bound, short-lived,
single-use. After a successful enrollment, the API promotes the same node-bound secret to
a control-plane recovery token (`node_recovery_token:*`) and consumes only the one-time
bootstrap token (`node_enroll_token:*`). This keeps first-enrollment replay bounded while
allowing an already-onboarded node with an expired local client certificate to re-enroll
without direct file surgery. `node.self_update` may rotate the recovery token by installing
a fresh `GPUAAS_ENROLLMENT_TOKEN` in the node-agent env.

---

## 5. Task Catalog (Allowlist)

The agent binary contains a compiled-in registry of allowed task types (`catalog/catalog.go`).
Any `task_type` not in the registry is rejected before dispatch — no handler is called,
no side effects occur.

### 5.1 Node Lifecycle Tasks

| Task type | Parameters | Description |
|---|---|---|
| `node.install_base_packages` | `packages: []string` | apt-get install; each name checked against compiled-in package allowlist (§6) |
| `node.install_gpu_drivers` | `rocm_version: string` | Installs AMD ROCm stack |
| `node.install_network_drivers` | `driver_version: string` | Installs Broadcom RDMA drivers |
| `node.configure_network` | `dns_servers: []string`, `netplan_patch: string` | Applies network settings; netplan YAML validated against restricted key set |
| `node.configure_hardware` | `acs_disabled: bool`, `nic_port_mode: int` | ACS and NIC port configuration |
| `node.configure_storage` | `weka_cluster_ip: string`, `weka_version: string` | Weka client setup |
| `node.reboot` | `reason: string` | Controlled reboot with audit log entry before reboot |
| `node.drain` | `reason: string` | Marks node unavailable; blocks new task dispatch until drained |
| `node.uninstall` | `install_root: string`, `env_file_path: string`, `systemd_unit_name: string`, `cert_path: string`, `key_path: string`, `ca_bundle_path: string`, `node_cert_ca_bundle_path: string`, `reason: string (optional)` | Schedules bounded node-agent uninstall cleanup for a removed node; agent stays alive long enough to post task result before deferred self-stop and file removal |
| `node.self_update` | `package_url: string`, `package_sha256: string`, `expected_version: string`, `task_signing_pubkeys: string`, `enrollment_token: string (optional)`, `install_root: string (optional)`, `env_file_path: string (optional)`, `systemd_unit_name: string (optional)`, `ca_bundle_path: string (optional)`, `registry_env_file: string (optional)`, `install_container_runtime: bool (optional)`, `container_runtime_package: string (optional)`, `reason: string (optional)` | Schedules a digest-pinned node-agent bootstrap package update using the existing installer; `task_signing_pubkeys` replaces legacy verifier env material with the current trusted verifier set, and when supplied, `enrollment_token` replaces `GPUAAS_ENROLLMENT_TOKEN` in the installed env so future cert recovery has a fresh control-plane-issued token; the agent reports task success before the deferred finalizer restarts the systemd unit |
| `node.gpu_scrub` | `allocation_id: string (uuid, optional)`, `mode: reset \| verify_only`, `gpu_pci_addresses: []string (optional)`, `container_runtime: docker \| containerd \| none (optional)`, `reason: string` | Contract for host GPU/runtime cleanup after allocation, reimage, or soft reset. Success requires `scrubbed=true`, per-device proof, and no remaining GPU processes attributable to the allocation when `mode=reset`. Execution is intentionally separate from this contract slice. |
| `node.storage_cleanup` | `allocation_id: string (uuid, optional)`, `paths: []string`, `delete_empty_dirs: bool (optional)`, `wipe_block_devices: []string (optional)`, `reason: string` | Contract for node-local runtime residue cleanup. Paths must be under approved GPUaaS runtime roots; persistent user volumes are out of scope. Success requires `cleaned=true` and per-path proof. |
| `node.validate_clean` | `allocation_id: string (uuid, optional)`, `checks: []string`, `paths: []string (optional)`, `block_devices: []string (optional)`, `gpu_pci_addresses: []string (optional)`, `reason: string` | Read-only cleanliness verification contract. Returns `clean=true` only when all requested checks pass; otherwise returns machine-readable findings without destructive action. |

### 5.2 Allocation Tasks

Called for every user session (provision on allocation creation, revoke on release).

| Task type | Parameters | Description |
|---|---|---|
| `allocation.provision_user` | `username: string`, `ssh_public_keys: []string`, `uid: int`, `gid: int`, `supplemental_gids: []int (optional)` | Creates OS user/group with stable POSIX identity; installs one or more validated SSH public keys to `authorized_keys`; writes a dedicated passwordless sudoers entry for the runtime user so platform-approved host bootstrap can proceed without ad hoc manual elevation |
| `allocation.install_authorized_keys` | `allocation_id: string (uuid)`, `username: string`, `ssh_public_keys: []string` | Rewrites `authorized_keys` for an already-provisioned allocation user; rejects when the target user does not already exist on the node |
| `allocation.revoke_user` | `username: string` | Removes user account; deletes SSH key; kills all user sessions and processes |
| `allocation.reset_environment` | `username: string` | Cleans `/home/{user}`; kills stray processes; resets GPU compute state |

Identity source:
- `username`/`uid`/`gid` are sourced from `platform_iam_user_posix_identities` in control-plane DB.
- They are stable per user across all allocations/nodes to support multi-node cluster workflows.
- Agent must treat provided uid/gid as authoritative and reject conflicting pre-existing local identities.
- `ssh_public_keys` are resolved by control-plane precedence:
  1. allocation-level key set (`allocation_ssh_public_keys`) when present
  2. otherwise user's active default key (`user_ssh_public_keys.is_default = true`)
- `allocation.install_authorized_keys` is allocation-bound and does not create users; it only updates key material for the runtime user that should already exist when the allocation is active.

### 5.3 Operational Tasks

| Task type | Parameters | Description |
|---|---|---|
| `node.health_check` | — | Returns GPU count, driver versions, disk/memory stats, uptime |
| `node.get_metrics` | — | Returns GPU utilization, temperature, VRAM per device (via `rocm-smi`) |

### 5.3a Slice VM Tasks

Bounded host-side VM lifecycle tasks for approved `gpu_slice` allocations. These
tasks are emitted only by provisioning workers after the scheduler has reserved
`node_resource_slots` and `allocation_resource_claims`; browser/user input never
becomes a raw device path, libvirt option, or shell command.

| Task type | Parameters | Description |
|---|---|---|
| `slice.vm_provision` | `allocation_id: string (uuid)`, `vm_name: string`, `image_path: string`, `image_url: string (optional)`, `image_sha256: string (optional)`, `driver_strategy: string (cloud-init \| preinstalled \| none, optional)`, `default_username: string`, `ssh_public_keys: []string`, `slots: []object`, `ovs_bridge: string`, `cloud_init_dir: string (optional)`, `graceful_timeout_seconds: int (optional)` | Verifies the digest-pinned base image when provided, validates all destination disks and PCI devices from platform-approved slot claims, downloads a missing approved base image from `image_url` when supplied, writes per-allocation cloud-init, clones the image to raw NVMe, creates/starts a libvirt VM with VFIO GPU/fabric passthrough, OVS vNIC, and NUMA tuning, then returns VM name and access endpoint metadata. `cloud-init` installs guest NVIDIA/RDMA packages during first boot; `preinstalled` skips that heavy install and only runs readiness checks. |
| `slice.vm_release` | `allocation_id: string (uuid)`, `vm_name: string`, `slots: []object`, `cloud_init_dir: string (optional)`, `graceful_timeout_seconds: int (optional)`, `wipe: bool` | Performs graceful shutdown, hard-stop fallback, libvirt undefine, cloud-init artifact cleanup, and optional raw NVMe wipe/verification before slot reuse |
| `slice.topology_discover` | — | Read-only host discovery task that reports candidate GPU PCI devices, NVMe devices, fabric devices, per-slot fabric VF identity when sysfs exposes `physfn`, OVS bridges, KVM/IOMMU/tooling prerequisite state, and a `candidate_summary` explaining blockers such as missing per-slot fabric VFs, mounted slice storage, missing VM tooling, or absent OVS bridge. Output is advisory only and must be approved into `node_resource_slots` before scheduling. Candidates without `fabric_vf_pci_address` are not schedulable GPU VM slice capacity. |

Slice VM task constraints:
- `vm_name` must be generated by the control plane as
  `gpuaas-slice-<allocation uuid without dashes>`.
- `image_path` must be an absolute path under a node-agent approved image root
  such as `/var/lib/gpuaas/slice-images` or `/var/lib/libvirt/images`.
- `image_url`, when provided, must be HTTPS and is used only to populate a
  missing approved `image_path`; the local path remains the scheduler-selected
  image identity used for cloning and digest verification.
- `driver_strategy` controls guest bootstrap cost. `preinstalled` images must
  already contain NVIDIA/RDMA tooling and are verified with readiness commands;
  `cloud-init` images install the required guest packages before readiness.
- Before provisioning, the node-agent verifies required host commands
  (`cloud-localds`, `qemu-img`, `virt-install`, `virsh`, `ovs-vsctl`) and may
  perform an idempotent package install on apt-based Ubuntu hosts when they are
  missing.
- The node-agent must verify KVM and IOMMU group availability before cloning an
  image to raw NVMe so host passthrough prerequisites fail before destructive
  disk writes.
- Slot destination disks must be approved raw block devices (`/dev/nvme*` or
  `/dev/disk/by-id/*`), never arbitrary files.
- PCI passthrough values are converted to libvirt node-device names by the
  agent; callers cannot pass raw `virt-install` arguments.
- Raw VNC is not enabled. Console access, if added later, must go through an
  authenticated gateway with audit.
- Release keeps GPU/fabric devices bound to `vfio-pci` while the node remains in
  slice mode; reattach is reserved for node drain/baremetal transition flows.
  Slice-mode hosts must set `GPUAAS_SLICE_RUNTIME_VFIO_BIND=1` in the
  node-agent environment so provision/release tasks can repair libvirt driver
  reattach drift after validating the approved slot PCI addresses.
- Release must return explicit cleanup proof. The provisioning worker only marks
  claimed slots `available` after `slice.vm_release` returns `released=true`,
  `wiped=true` when `wipe=true`, and a `slot_count` matching the claimed slots.

### 5.4 Security Tasks

| Task type | Parameters | Description |
|---|---|---|
| `node.rotate_signing_key` | `new_public_key_pem: string`, `effective_at: string` | Rotates task-signing public key; must itself be signed with the old key |

### 5.5 Runtime Operator Tasks

Bounded host-side primitives that let the control plane materialize runtime configuration and service lifecycle without turning the agent into a generic shell.

| Task type | Parameters | Description |
|---|---|---|
| `runtime.write_env_file` | `path: string`, `entries: object`, `mode: string (optional)` | Writes a sorted env file under `/etc/gpuaas/runtime/**`; rejects paths outside the runtime env root |
| `runtime.install_service_unit` | `unit_name: string`, `content: string` | Writes `/etc/systemd/system/{unit_name}` and runs `systemctl daemon-reload` |
| `runtime.service_control` | `unit_name: string`, `action: enable_start \| restart \| stop \| disable_stop \| status` | Performs bounded systemd control for an approved service unit and returns `active_state` |

These tasks are deliberately narrow:
- no arbitrary command strings,
- no unrestricted file writes,
- no direct package-manager execution for app teams,
- no hidden transport-specific behavior.

Host reuse semantics:
- `admin node reactivate` reuses the same `node_id` and existing installed identity after a `retired` pause.
- `admin node remove` starts `node.uninstall`; the host is not reusable for a new node identity while status is `removing`.
- Re-registering the same host under a new `node_id` is valid only after uninstall cleanup completes and the prior node record is gone.

Launchable OCI workload note:
- The task catalog includes OCI workload task names and a non-mutating runtime
  status probe.
- The first node-agent adapter slice implements launch/control/remove against
  approved local runtimes discovered by that probe. The app-runtime controller
  must still gate launch until it can prove runtime availability and enqueue the
  task from a curated profile.
- Those tasks must remain profile-driven and digest-pinned. They must not accept
  arbitrary shell commands, arbitrary image names, raw registry credentials, or
  unrestricted host mounts from user input.

Planned task contract:

| Task type | Parameters | Description |
|---|---|---|
| `workload.oci_runtime_status` | — | Reports whether an approved local container runtime is installed and reachable; no mutation |
| `workload.oci_launch` | `workload_instance_id: string (uuid)`, `allocation_id: string (uuid)`, `username: string`, `image_digest_ref: string`, `container_name: string`, `env: object`, `mounts: []object`, `endpoints: []object`, `gpu_request: object`, `command_args: []string (optional)`, `pull_credential_ref: string (optional)` | Launches one profile-approved, digest-pinned OCI workload for an already-provisioned allocation user |
| `workload.oci_control` | `workload_instance_id: string (uuid)`, `action: start \| stop \| restart \| status` | Bounded lifecycle control for a previously launched workload instance |
| `workload.oci_remove` | `workload_instance_id: string (uuid)`, `remove_scratch: bool` | Stops and removes the workload-owned runtime objects; when `remove_scratch=true`, deletes only the exact UUID-scoped scratch directory under an allocation home `.gpuaas/workloads/`; persistent project storage is never deleted by this task |
| `workload.compose_launch` | `workload_instance_id: string (uuid)`, `allocation_id: string (uuid)`, `username: string`, `project_name: string`, `compose_yaml: string`, `compose_renderer: string (optional)`, `topology: object (optional)`, `mounts: []object`, `endpoints: []object` | Launches a control-plane-rendered Docker Compose bundle for a profile-approved allocation-local workload such as vLLM |
| `workload.compose_control` | `workload_instance_id: string (uuid)`, `action: start \| stop \| restart \| status` | Bounded Docker Compose lifecycle control for a previously launched composed workload |
| `workload.compose_remove` | `workload_instance_id: string (uuid)`, `remove_scratch: bool` | Runs `docker compose down --remove-orphans --volumes`; when `remove_scratch=true`, deletes only the exact UUID-scoped scratch directory under an allocation home `.gpuaas/workloads/` |

Additional constraints:
- `image_digest_ref` must be a platform-resolved digest reference, not a mutable tag.
- `container_name` must be generated by the control plane and scoped to the app instance.
- `project_name` for Compose workloads must be generated from the workload instance
  ID, and `compose_yaml` must be rendered by app-runtime from a curated profile.
- `compose_renderer` and `topology` are manifest-owned metadata. They help
  app-runtime and the UI explain the service shape, but they must not make
  user-submitted Compose YAML acceptable.
- `env`, `mounts`, `endpoints`, and `gpu_request` must be rendered from a
  curated profile manifest and validated launch values.
- `mounts` must reference only profile-declared paths such as `/workspace`; no
  arbitrary host path from the browser is allowed.
- OCI workloads run as the allocation user's numeric UID/GID. The node-agent
  projects minimal workload-scoped `/etc/passwd` and `/etc/group` files into the
  container so interactive shells and notebook terminals can resolve the runtime
  identity without requiring every image to pre-create platform UID ranges.
- `pull_credential_ref` must refer to a short-lived or wrapped credential
  resolved by the control plane; raw registry username/password values must not
  be sent in the task payload.
- The node must reject launch if no approved container runtime is available.
- Compose launch/control status output must include both raw `services_json`
  from `docker compose ps --format json` and a normalized `services` array so
  control-plane/UI consumers do not need to parse Docker-version-specific JSON
  formatting.
- The current implementation accepts `file:` based `pull_credential_ref` values
  rooted under approved node-local credential directories, uses runtime
  `login --password-stdin`, and never accepts raw registry secrets in the task
  payload. It requires the deterministic container name
  `gpuaas-oci-<workload uuid without dashes>` and only accepts bind mounts rooted under
  `/home/<username>/.gpuaas/workloads/<workload_instance_id>/` with targets under
  `/workspace`. The node resolves `username` to the local allocation user's
  numeric `uid:gid`, creates missing workload-scoped mount directories, and
  chowns them to that identity before passing `--user` to the OCI runtime. Launch
  does not depend on the image containing a matching Linux username.

### 5.6 Storage Mount Tasks

Storage attach/detach is orchestrated by Temporal as defined in
`doc/architecture/Storage_Attachment_Workflow_v1.md`. The node-agent performs
only local host mount operations for a control-plane-approved attachment. It
does not create provider namespaces, issue provider credentials, or delete
persistent storage.

Planned task contract:

| Task type | Parameters | Description |
|---|---|---|
| `storage.mount_attach` | `attachment_id: string (uuid)`, `allocation_id: string (uuid, optional)`, `workload_instance_id: string (uuid, optional)`, `username: string (optional)`, `uid: int (optional)`, `gid: int (optional)`, `provider_backend: string`, `source_ref: string`, `mount_path: string`, `access_mode: read_only \| read_write`, `write_policy: single_writer \| multi_writer`, `expected_fstype: string (optional)` | Mounts one provider-approved storage namespace/path into the allocation/workload context and reports mount health. `source_ref` is an opaque control-plane-generated mount reference, not user input. UID/GID are only supplied when the control plane can resolve a concrete runtime identity. |
| `storage.mount_detach` | `attachment_id: string (uuid)`, `allocation_id: string (uuid, optional)`, `workload_instance_id: string (uuid, optional)`, `mount_path: string`, `force: bool (optional)` | Unmounts one known attachment. Never deletes persistent storage or workload data. |
| `storage.mount_probe` | `attachment_id: string (uuid)`, `mount_path: string`, `access_mode: read_only \| read_write` | Read-only probe for mount presence, filesystem type, ownership, and optional read/write test when write mode is expected. |

Storage task constraints:

- `mount_path` must be under an agent-approved allocation/workload mount root.
- `source_ref` must be generated by the control plane/provider adapter and must
  not be a raw browser-supplied host path.
- The agent must reject provider admin credentials, shell fragments, or unknown
  mount options in task payloads.
- Read-write mounts require both provider capability and product/app
  `write_policy` approval.
- `storage.mount_detach` is idempotent: already-unmounted means success with a
  status note.
- Persistent storage deletion is never a node-agent task.

### 5.7 Terminal Tasks

One-shot task that authorizes a PTY session. I/O then flows over a separate stream channel;
these are **not** discrete tasks.

| Task type | Parameters | Description |
|---|---|---|
| `terminal.open` | `session_id: string (uuid)`, `user_id: string (uuid)`, `allocation_id: string (uuid)`, `username: string`, `session_ttl_seconds: int`, `cols: int`, `rows: int` | Authorize and open a PTY session for the named OS user. Node attaches PTY, begins streaming stdin/stdout/stderr on the established relay channel. |

Parameter constraints:

| Parameter | Constraint |
|---|---|
| `session_id` | Must be a valid UUID; enforced by seen-set against replay |
| `username` | Must match platform-assigned Linux-safe username format. Current allocator emits `^g_[a-z0-9]{6,32}$` for MVP. |
| `session_ttl_seconds` | 300 ≤ value ≤ 86400 (policy key `terminal.session_max_ttl_seconds`) |
| `cols` / `rows` | 1 ≤ value ≤ 512 |

Close semantics:
- Session close is a **stream close frame**, not a separate signed task.
- Allowed close reasons: `allocation_released`, `session_timeout`, `normal_close`, `admin_terminate`, `node_stream_dropped`.

Node-side enforcement:
- Enforces single active PTY per `(node_id, allocation_id)` at handler level.
- Rejects `terminal.open` if the OS user does not exist on the node (i.e., `allocation.provision_user` was not completed).
- Emits session-open acknowledgement on success; emits terminal control error/close frame if PTY attach fails.

### 5.7 What the Agent Explicitly Never Does

- Execute arbitrary shell strings or scripts.
- Fetch and run content from remote URLs.
- Install packages not in the compiled package allowlist.
- Accept a task type not compiled into the registry.
- Run any handler before signature verification and parameter validation pass.

---

## 6. Package Allowlist

`node.install_base_packages` validates each package name against a compiled-in set.
An unknown package name causes the entire task to be rejected; no packages are installed.

Approved packages (MVP):

```
lldpd, libssl-dev, libargtable2-0, dracut-core, ipmitool, nfs-common,
environment-modules, net-tools, libibumad-dev, rdma-core, ibverbs-utils,
infiniband-diags, libelf-dev, gcc, make, libtool, autoconf,
librdmacm-dev, rdmacm-utils, perftest, ethtool, libibverbs-dev,
python3-setuptools, python3-wheel, rocm, amdgpu-dkms
```

Updating the allowlist requires:
1. Change and review in this file.
2. New agent binary build and deployment to all nodes.
3. No runtime additions — the list is immutable at runtime.

---

## 7. Protocol

### 7.1 Internal API Endpoints (cmd/api) — Node-Facing

The node agent calls exactly five endpoints, all on `api.internal`. This is the complete
list of network calls a node agent ever makes.

| Method | Path | Auth | Description |
|---|---|---|---|
| `POST` | `/internal/v1/nodes/enroll` | Enrollment token (plain TLS) | CSR submission; API proxies to step-ca; returns cert + CA bundle |
| `POST` | `/internal/v1/nodes/{id}/cert/renew` | mTLS (`nodes`) | New CSR; API proxies to step-ca; returns new cert |
| `GET` | `/internal/v1/nodes/{id}/tasks/wait` | mTLS (`nodes`) | Long-poll: holds open up to 30 s; returns 200+task or 204 (no task) |
| `POST` | `/internal/v1/nodes/{id}/tasks/{task_id}/result` | mTLS (`nodes`) | Result delivery; API calls `temporalClient.CompleteActivity()` |
| `POST` | `/internal/v1/nodes/{id}/terminal/stream` | mTLS (`nodes`) | Bidirectional terminal relay stream for stdin/stdout/stderr, resize, and close frames bound to an approved `terminal.open` session |

No other endpoints. No direct connections to step-ca, Postgres, Redis, Temporal, or NATS.

All `/internal/v1/*` paths listen on a separate port (default: 9443) with mTLS enforced
at the listener level. They are not mounted in the public route group.

These endpoints must be added to `doc/api/openapi.draft.yaml` (internal paths section)
before implementation.

**Provisioning workers** (internal Kubernetes pods) write `node_tasks` rows directly to
Postgres and publish Redis wakeup pings directly — they do not call an API endpoint to
queue tasks. Workers have DB and Redis access as internal services.

### 7.2 Task Payload Shape

```json
{
  "task_id": "uuid",
  "task_type": "allocation.provision_user",
  "node_id": "node-abc123",
  "correlation_id": "uuid",
  "issued_at": "2026-02-25T10:00:00Z",
  "expires_at": "2026-02-25T10:05:00Z",
  "params": {
    "username": "g_devadmin",
    "ssh_public_keys": ["ssh-ed25519 AAAA...", "ssh-rsa AAAA..."],
    "uid": 10042,
    "gid": 10042
  },
  "signature": "base64url-encoded-ed25519-signature"
}
```

Signature is computed over: `sha256(canonical_json_without_signature_field)`.
Canonical JSON: fields sorted lexicographically, no whitespace, UTF-8.

### 7.3 Task Result Shape

```json
{
  "task_id": "uuid",
  "node_id": "node-abc123",
  "status": "success",
  "error": null,
  "completed_at": "2026-02-25T10:00:02Z",
  "output": {}
}
```

`status` values: `success` | `failed` | `rejected`

- `rejected`: task type not in catalog, signature invalid, or params invalid.
  No side effects occurred.
- `failed`: handler ran but the operation failed (e.g., user already exists, apt error).
- `success`: handler completed successfully.

### 7.4 Task Delivery — Long-Poll

The node agent uses long-polling, not a short-interval poll loop. Redis pub/sub is a
**fast-path optimisation only**. `node_tasks` in Postgres is the authoritative source
of truth. Correctness never depends on a Redis message arriving.

**Handler sequence (per request)**:

```
1. Receive GET /internal/v1/nodes/{id}/tasks/wait  (mTLS validated)

2. CHECK POSTGRES FIRST — always, before touching Redis:
      SELECT id, task_type, params, temporal_task_token, ...
      FROM   node_tasks
      WHERE  node_id = $1 AND status = 'queued'
      ORDER  BY created_at
      LIMIT  1
      FOR UPDATE SKIP LOCKED

   ┌─ Row found ──► UPDATE status='dispatched', dispatched_at=now()
   │                COMMIT
   │                Return 200 + task payload
   │                (skip steps 3-6)
   └─ No row ──► continue

3. Subscribe to Redis pub/sub: SUBSCRIBE node_task_notify:{node_id}

4. Wait up to 30 s for wakeup signal or timeout

5. On wakeup received:
      Run same SELECT ... FOR UPDATE SKIP LOCKED
      ┌─ Row found ──► mark dispatched, return 200 + task
      └─ No row     ──► lost race to another API pod → return 204
                        (node immediately re-opens long-poll; step 2 will catch it)

6. On 30 s timeout: return 204
   Node agent immediately re-opens long-poll → back to step 1
```

Step 2 (Postgres-first check) is the safety net for every failure scenario below.

`FOR UPDATE SKIP LOCKED` ensures only one API pod wins when multiple pods race on the
same wakeup — the others find no unlocked row and return 204.

**Temporal async completion**: the provisioning worker's Temporal activity stores its
`taskToken` in the `node_tasks` row. When the node POSTs a result, the API handler
calls `temporalClient.CompleteActivity(taskToken, result)` — directly waking the
Temporal workflow with no polling at the activity level. See §13 for the activity pattern.

### 7.5 Delivery Guarantees and Failure Recovery

#### Scenario A — Pod failure BEFORE node reads (before response sent)

```
Worker ──INSERT node_tasks──► Postgres  (status=queued)
       ──PUBLISH wakeup────► Redis

API pod 1  receives wakeup
           SELECT FOR UPDATE SKIP LOCKED → gets row
           UPDATE status='dispatched' → commits
           [POD DIES — TCP reset; node never received the response]

node_tasks row: status=dispatched  ← stuck
```

Recovery path:

1. Node detects TCP reset, immediately reconnects (to pod 2).
2. New long-poll opens. **Step 2 runs**: `SELECT ... FOR UPDATE SKIP LOCKED`.
3. Row is `dispatched` — not returned. Node waits.
4. Temporal `scheduleToCloseTimeout` fires → Temporal retries the activity.
5. New activity execution inserts a **new** `node_tasks` row (new `task_id`, new
   `temporal_task_token`) and publishes a fresh Redis wakeup.
6. Node receives the new task on the next wakeup cycle. Delivers correctly.

Old `dispatched` row: ignored once `expires_at` passes; cleaned up by background job.

**Production hardening** (post-MVP): a background job re-queues stuck `dispatched`
rows after a short threshold, capping recovery time to ~2 min instead of waiting for
the full Temporal activity timeout:

```sql
-- Runs every 30 s in provisioning worker
UPDATE node_tasks
SET    status = 'queued', dispatched_at = NULL
WHERE  status = 'dispatched'
  AND  dispatched_at < now() - interval '2 minutes'
  AND  expires_at > now()
RETURNING id, node_id;
-- For each returned row: PUBLISH Redis node_task_notify:{node_id}
```

The Temporal activity is still in its heartbeat loop with a valid token — when the
re-queued task completes, `CompleteActivity()` resolves it correctly. No duplicate
execution: if the node somehow already executed and its result POST was lost, the node
agent's in-memory `task_id` seen-set blocks re-execution of the same task ID.

Inventory lifecycle tasks (`node.drain`, `node.uninstall`) must also be recoverable by
an explicit operator/control-plane resume action. That resume action must:
- no-op if a fresh `queued` task already exists
- no-op if a live `dispatched` lease still exists
- requeue a stale `dispatched` task when the lease is older than the reclaim threshold
- recreate the lifecycle task when no live task exists because the prior one expired

This prevents coarse node states such as `draining` and `removing` from becoming
unrecoverable when an agent was absent or a task lease aged out.

#### Scenario B — Pod failure BEFORE committing `dispatched`

```
API pod 1  SELECT FOR UPDATE SKIP LOCKED → row locked (not yet dispatched)
           [POD DIES — transaction rolls back; lock released]

node_tasks row: status=queued  ← reverts automatically
```

Recovery path:

1. Node detects TCP reset, reconnects to pod 2.
2. **Step 2**: `SELECT ... FOR UPDATE SKIP LOCKED` — row is `queued` again.
3. Row returned immediately. Delivered on the reconnect, no Temporal retry needed.

This is the best case: Postgres transaction semantics give automatic recovery with
zero delay beyond the reconnect.

#### Scenario C — Redis loses the wakeup message (restart or network drop)

```
Worker ──PUBLISH node_task_notify:{node_id}──► Redis restarts / message lost
                                               No API pod receives the wakeup

node_tasks row: status=queued  ← sitting in Postgres, no one knows
```

Recovery path:

1. Node's current long-poll waits 30 s → times out → 204 returned.
2. Node immediately re-opens long-poll.
3. **Step 2** (Postgres-first check) runs on every new long-poll.
4. Row found → dispatched → returned to node.

**Maximum recovery delay: 30 s** (one long-poll timeout cycle). The task is never
lost — it is durable in Postgres regardless of Redis state.

#### Scenario D — Redis restarts while API pod is subscribed

```
API pod holds open long-poll, subscribed to node_task_notify:{node_id}
Redis restarts → subscription dropped
go-redis reconnects automatically but current subscription is gone
```

Recovery path:

- The long-poll handler detects the subscription error and returns 204 immediately
  (rather than waiting out the full 30 s timeout). Node re-polls; Postgres-first check
  picks up any queued task.
- go-redis pub/sub client re-subscribes automatically after reconnect, so subsequent
  long-polls have working subscriptions.

#### Summary

| Failure | `node_tasks` state after | Recovery mechanism | Max delay |
|---|---|---|---|
| Pod dies before TX commit | `queued` (auto-rollback) | Postgres-first check on reconnect | ~1 s (reconnect time) |
| Pod dies after `dispatched` committed, before response | `dispatched` | Temporal retry → new task row | Temporal `scheduleToCloseTimeout` (MVP); dispatched-timeout job (production: ~2 min) |
| Redis message lost | `queued` | 30 s long-poll timeout → Postgres-first check | 30 s |
| Redis restart while subscribed | `queued` | Subscription error → immediate 204 → Postgres-first check | ~1 s |
| Multiple API pods race on same wakeup | `queued` for losers | `FOR UPDATE SKIP LOCKED` — one winner, others return 204 | None (handled atomically) |

**Invariant**: a task is never permanently lost as long as Postgres is available.
Redis failure degrades to at-most 30 s latency. Pod failure degrades to Temporal retry
or (with the production hardening job) ~2 min. No Redis persistence required; no NATS
needed; no sticky sessions required.

---

## 8. Task Security Model

All three checks run in order. Any failure → `rejected` result, no handler called.

### 8.1 Signature Verification

1. Deserialize task payload JSON.
2. Extract and remove the `signature` field.
3. Compute `sha256(canonical_json)`.
4. Verify Ed25519 signature against the baked-in public key.
5. Fail: return `rejected`, log warning. No handler called, no side effects.

### 8.2 Replay Protection

Two independent defenses (both must pass):

1. **Expiry**: if `now() > expires_at`, reject. Max task lifetime set by control plane
   at 5 min. Tasks issued more than 5 min ago are always rejected.
2. **task_id seen-set**: agent maintains an in-memory LRU cache (capacity 10,000) of
   recently dispatched `task_id` values. Duplicate → rejected.

### 8.3 Parameter Validation

Every task type has a compiled-in validator in `validate/params.go`.
Validation runs after signature verification, before handler dispatch.

| Parameter | Constraint |
|---|---|
| `username` | `^[a-z_][a-z0-9_-]{0,31}$` (platform-generated, not user-supplied freeform) |
| `ssh_public_keys[]` | 1..20 entries; each key must parse as valid Ed25519 or RSA ≥ 3072-bit |
| `uid` / `gid` | Must be in reserved range `10000–199999` |
| `packages` | Each name must be in compiled-in package allowlist (§5) |
| `weka_cluster_ip` | Must be valid IPv4 in allowed CIDR range |
| `rocm_version` / `driver_version` | Must match `^[0-9]+\.[0-9]+\.[0-9]+$` |
| `netplan_patch` | Must parse as valid YAML with restricted top-level keys only |
| `effective_at` | Must be RFC3339, must be ≥ 5 min in future |
| `workload_instance_id` / `allocation_id` | Must be valid UUIDs and must match the control-plane task context |
| `image_digest_ref` | Must match an approved platform registry digest reference; mutable tags are rejected |
| `container_name` | Must match `^gpuaas-[a-z0-9][a-z0-9-]{0,62}$` and be derived from the app instance |

Any parameter failing validation → entire task rejected.

---

## 9. Privilege Model

The agent binary runs as `root` only during the initial setup step (install systemd
service, configure sudoers). The polling loop runs as the unprivileged `gpuaas-agent`
user. Each handler uses a minimal, explicitly-listed `sudo` rule.

The blanket `NOPASSWD: ALL` from the prototype shell script is not used.

`/etc/sudoers.d/99-gpuaas-agent`:

```
gpuaas-agent ALL=(root) NOPASSWD: /usr/sbin/useradd, /usr/sbin/userdel
gpuaas-agent ALL=(root) NOPASSWD: /usr/bin/tee /home/*/authorized_keys
gpuaas-agent ALL=(root) NOPASSWD: /usr/bin/pkill -u *
gpuaas-agent ALL=(root) NOPASSWD: /usr/bin/apt-get install -y *
gpuaas-agent ALL=(root) NOPASSWD: /usr/sbin/reboot
```

Per-task privilege needs:

| Task type | Privilege | Mechanism |
|---|---|---|
| `allocation.provision_user` | Create user, write `authorized_keys`, install runtime-user sudoers drop-in | sudo useradd + sudo tee + sudo install into `/etc/sudoers.d` |
| `allocation.install_authorized_keys` | Rewrite `authorized_keys` for existing user | sudo tee |
| `allocation.revoke_user` | Kill processes, delete user | sudo pkill + sudo userdel |
| `allocation.reset_environment` | Clean home dir, kill processes | sudo pkill; rm under user's home |
| `node.install_base_packages` | apt-get | sudo apt-get (allowlisted packages verified before sudo) |
| `node.reboot` | systemctl reboot | sudo reboot |
| `node.self_update` | write finalizer in `/tmp`, fetch digest-pinned package, refresh `GPUAAS_TASK_SIGNING_PUBKEYS`, optionally refresh `GPUAAS_ENROLLMENT_TOKEN`, run `install-node-agent.sh`, restart `gpuaas-node-agent.service` | sudo/root execution context; package URL must be control-plane issued and `package_sha256` must be verified before installer execution; finalizer logs to `/var/log/gpuaas-node-agent-self-update.log` when writable |
| `node.gpu_scrub` | stop allocation-owned GPU/runtime residue and reset/verify approved GPUs | sudo/root execution context; implementation must be bounded to control-plane-issued allocation IDs and approved GPU PCI addresses |
| `node.storage_cleanup` | remove node-local runtime residue under approved roots | sudo/root execution context; implementation must reject persistent-storage paths and arbitrary absolute paths |
| `node.validate_clean` | inspect processes, mounts, paths, and GPU handles | read-only by default; sudo only if a check requires privileged process or mount-table inspection |
| `storage.mount_attach` | create approved mountpoint, run provider mount helper, set ownership/mode | sudo install/mkdir + approved mount helper only; no arbitrary mount options |
| `storage.mount_detach` | unmount approved mountpoint | sudo umount only for attachment-scoped mount path |
| `storage.mount_probe` | inspect mount table and optional file probe | no sudo for read-only checks; write probe only inside approved mount path when requested |
| `node.health_check` | sysfs read, rocm-smi | None (read-only; user in `render`+`video` groups) |
| `node.get_metrics` | rocm-smi | None (user in `render`+`video` groups) |

### 9.1 Terminal-specific privilege constraints (mandatory)

Normative decision reference:
- `doc/architecture/adrs/ADR-007-terminal-access-auth-model.md`

`terminal.open` is not a lifecycle/provisioning mutation task. It is a session bootstrap intent.

Non-negotiable constraints:
- Terminal bootstrap may perform only user impersonation into the target allocation user.
- After impersonation, PTY child process must run as allocation user (`uid/gid` of target user), not as root and not as the agent service account.
- Terminal code path must not call privileged lifecycle operations (`useradd`, `userdel`, package install, network/hardware/storage config, reboot).
- Terminal stream frames (`stdin`, `resize`, `close`) are data/control only; they must not map to privileged host commands.
- If impersonation command/policy fails (`sudo` policy mismatch, missing binary, invalid user), fail closed and emit terminal close/error frame with correlation-backed API error context.

Operational guardrail:
- Sudoers for node-agent must use explicit command allowlist. `NOPASSWD: ALL` is forbidden for production.
- Preferred long-term posture is split execution domains:
  - provisioning/lifecycle executor (privileged, task-based),
  - terminal executor (interactive, least-privilege, no host mutation surface).

Split-seam invariants:
- lifecycle execution must remain operational even if terminal runtime is degraded or unavailable.
- terminal execution must not gain any privileged host-mutation surface beyond user impersonation into an already-provisioned allocation user.
- terminal-specific transport or endpoint changes must not force task poll/provisioning contract changes.

---

## 10. Repository Layout

```
cmd/node-agent/
├── main.go                     # systemd integration, enrollment check, poll loop
├── config.go                   # env-var config struct with validation
├── enroll.go                   # enrollment: keygen, step-ca call, confirmation
├── renew.go                    # cert renewal loop (T-1h check, X5C call)
├── poll.go                     # task poll, dispatch, result report
├── catalog/
│   ├── catalog.go              # task type registry (allowlist map), Dispatch()
│   ├── node_lifecycle.go       # node.* handlers
│   ├── allocation.go           # allocation.* handlers
│   └── operational.go          # node.health_check, node.get_metrics
├── validate/
│   └── params.go               # per-task-type parameter validators
├── signing/
│   └── verify.go               # Ed25519 signature verification
└── agent_test.go               # dispatch, validation, replay protection unit tests
```

`catalog/catalog.go` is the authoritative allowlist in code:

```go
var taskHandlers = map[string]TaskHandler{
    "node.install_base_packages":  handleInstallBasePackages,
    "node.install_gpu_drivers":    handleInstallGPUDrivers,
    "node.install_network_drivers": handleInstallNetworkDrivers,
    "node.configure_network":      handleConfigureNetwork,
    "node.configure_hardware":     handleConfigureHardware,
    "node.configure_storage":      handleConfigureStorage,
    "node.reboot":                 handleReboot,
    "node.drain":                  handleDrain,
    "node.health_check":           handleHealthCheck,
    "node.get_metrics":            handleGetMetrics,
    "node.rotate_signing_key":     handleRotateSigningKey,
    "allocation.provision_user":   handleProvisionUser,
    "allocation.install_authorized_keys": handleInstallAuthorizedKeys,
    "allocation.revoke_user":      handleRevokeUser,
    "allocation.reset_environment": handleResetEnvironment,
}

func Dispatch(task Task) TaskResult {
    handler, ok := taskHandlers[task.Type]
    if !ok {
        return TaskResult{Status: "rejected", Error: "unknown_task_type"}
    }
    if err := validate.Params(task.Type, task.Params); err != nil {
        return TaskResult{Status: "rejected", Error: err.Error()}
    }
    return handler(task)
}
```

---

## 11. Configuration (Environment Variables)

```
# Set by cloud-init; the complete set of bootstrap config
GPUAAS_NODE_ID               (required) — node identifier
GPUAAS_API_URL               (required) — single internal API endpoint (e.g. https://api.internal.gpuaas.io)
GPUAAS_CA_FINGERPRINT        (required) — SHA-256 fingerprint of root CA cert; used to
                                           verify api.internal TLS cert before CA bundle is stored

# Set by bootstrap/runtime config
GPUAAS_TASK_SIGNING_PUBKEYS  (required) — comma/space separated base64url Ed25519 public keys for active/staged/grace verifier rollout
                                           (canonical lifecycle is versioned in
                                           `doc/architecture/Node_Task_Signing_Lifecycle_v1.md`)

# Runtime paths (defaults are fine in most cases)
GPUAAS_CERT_PATH             (default: /etc/gpuaas/agent.crt)
GPUAAS_KEY_PATH              (default: /etc/gpuaas/agent.key)
GPUAAS_CA_BUNDLE_PATH        (default: /etc/gpuaas/ca-bundle.crt) — trust bundle for control-plane HTTPS/API
GPUAAS_NODE_CERT_CA_BUNDLE_PATH (default: /var/lib/gpuaas/node-agent/node-cert-ca-bundle.crt)
                               — CA bundle returned by node enrollment/renewal for node certificate lifecycle
GPUAAS_LONGPOLL_TIMEOUT      (default: 30s) — how long to hold open the tasks/wait request

# Hardware
GPUAAS_USE_TPM               (default: false) — use TPM 2.0 for private key storage
```

Notes:
- `GPUAAS_CA_URL` does not exist. The node never connects to step-ca/Vault PKI directly.
- `GPUAAS_API_URL` is the only control-plane URL the lifecycle/task client uses.
- Interactive terminal streaming may use a distinct direct control-plane endpoint in some
  environments; that split must not change task-poll/request-reply semantics or collapse
  the lifecycle-vs-terminal seam defined above.
- `GPUAAS_CA_BUNDLE_PATH` and `GPUAAS_NODE_CERT_CA_BUNDLE_PATH` must remain separate.
  Reusing one file for both public HTTPS trust and node-cert lifecycle trust causes
  post-enrollment trust regressions.
- Task-signing verifier rollout must not depend on binary rebuild as the normal path;
  the lifecycle contract is defined in `doc/architecture/Node_Task_Signing_Lifecycle_v1.md`.
- Control-plane HTTPS bootstrap trust delivery and update are defined in
  `doc/architecture/Node_Bootstrap_Trust_Delivery_v1.md`.

---

## 12. DB Schema Addition: `node_tasks` Table

A new table queues tasks for polling agents. Add to `doc/architecture/db_schema_v1.sql`
before implementation.

```sql
CREATE TABLE node_tasks (
    id                    UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    node_id               UUID        NOT NULL REFERENCES nodes(id),
    task_type             TEXT        NOT NULL,
    params                JSONB       NOT NULL DEFAULT '{}',
    status                TEXT        NOT NULL DEFAULT 'queued'
                                      CHECK (status IN ('queued','dispatched','succeeded','failed','rejected')),
    correlation_id        UUID        NOT NULL,
    created_at            TIMESTAMPTZ NOT NULL DEFAULT now(),
    dispatched_at         TIMESTAMPTZ,
    completed_at          TIMESTAMPTZ,
    error_message         TEXT,
    output                JSONB,
    expires_at            TIMESTAMPTZ NOT NULL,
    -- Temporal task token for external activity completion.
    -- The provisioning worker activity stores its token here.
    -- The API calls temporalClient.CompleteActivity(temporal_task_token, result)
    -- when the node POSTs a result. NULL after activity completes.
    temporal_task_token   BYTEA
);

-- Long-poll query: next queued task for a given node, FIFO
CREATE INDEX ix_node_tasks_node_queued
    ON node_tasks(node_id, created_at)
    WHERE status = 'queued';
```

Status flow: `queued → dispatched → succeeded | failed | rejected`

Tasks with `expires_at < now()` and `status = 'queued'` are cleaned up by a periodic
background query in the provisioning worker (runs every 60 s).

`temporal_task_token` lifecycle:
- Set by the provisioning worker activity on insert.
- Read by the API result handler to call `temporalClient.CompleteActivity()`.
- Nulled out after the API calls CompleteActivity (no longer needed).

---

## 13. Provisioning Worker Changes (Phase 7 Impact)

Phase 7's `SSHSetupActivity` and `SSHTeardownActivity` are replaced:

**Before (prototype/raw SSH)**:
```
SSHSetupActivity  → dial node as root → run shell script
SSHTeardownActivity → dial node as root → run shell script
```

**After (node agent + Temporal async completion)**:

```go
// ProvisionUserActivity — no polling; Temporal is completed externally
func (a *Activities) ProvisionUserActivity(ctx context.Context, req ProvisionRequest) error {
    info := activity.GetInfo(ctx)

    // Write task to Postgres with the Temporal task token
    taskID, _ := a.db.InsertNodeTask(ctx, InsertNodeTaskParams{
        NodeID:            req.NodeID,
        TaskType:          "allocation.provision_user",
        Params:            req.Params,
        TemporalTaskToken: info.TaskToken,          // stored for external CompleteActivity
        ExpiresAt:         time.Now().Add(info.ScheduleToCloseTimeout),
    })

    // Wakeup ping — carries no task data, just wakes the node's long-poll
    a.redis.Publish(ctx, "node_task_notify:"+req.NodeID.String(), taskID.String())

    // Heartbeat loop — keeps activity alive; queries nothing
    // Temporal cancels ctx when CompleteActivity() is called by the API result handler
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(10 * time.Second):
            activity.RecordHeartbeat(ctx, taskID)
        }
    }
}

// In cmd/api — node result handler:
func (h *Handler) HandleTaskResult(w http.ResponseWriter, r *http.Request) {
    result := parseResult(r)
    task   := h.db.GetNodeTask(result.TaskID)

    h.db.UpdateNodeTaskStatus(result.TaskID, result.Status, result.Error, result.Output)

    if result.Status == "success" {
        h.temporalClient.CompleteActivity(r.Context(), task.TemporalTaskToken, result.Output, nil)
    } else {
        h.temporalClient.FailActivity(r.Context(), task.TemporalTaskToken,
            temporal.NewApplicationError(result.Error, result.Status))
    }
}
```

No poll loop in the activity. The node's result POST directly wakes the Temporal workflow.

`packages/services/provisioning/worker/ssh.go` is retained for:
- Admin SSH probe (`POST /api/v1/admin/nodes/{id}/probe`) — diagnostic only.
- One-time bootstrap before agent is installed (enrollment only; not in hot path).

After enrollment, `ssh.go` is never called in provisioning activities.

---

## 14. Relationship to Other Specs

| Spec | Relationship |
|---|---|
| `doc/architecture/PKI_Spec.md` | Enrollment, cert storage, renewal, task signing keypair defined there |
| `doc/architecture/Encryption_Envelope_Spec.md` | Orthogonal; node agent does not store long-lived encrypted fields |
| `doc/api/openapi.draft.yaml` | `/internal/v1/nodes/*` endpoints must be added before implementation |
| `doc/architecture/db_schema_v1.sql` | `node_tasks` table must be added before implementation |
| `doc/Implementation_Roadmap.md` | Pre-Phase Node Agent section gates Phase 7 |