Add attribute governance design: controlled attribute lists, egress regex selection, ingest signature validation

Defines the SA-V3-Ext/Int placeholder attribute scheme (external list
owned by the S3 handover spec, internal list for SA routing + internal
S3 usage), the per-egress Regex property for packing/signing selection,
the inverse-regex delete step at true external egress, optional
ingest-time signing with mandatory validation when already signed, and
the S3 JSON-fallback size attribute for suffix-range trailer reads.

.claude/memory/MEMORY.md

@@ -3,4 +3,5 @@
 - [Project Purpose](project-purpose.md) — why this repo exists, the two core problems being solved
 - [SA/MR Topology](topology.md) — deployment topology terminology and shape
 - [Packaging & Signing Design Decisions](packaging-signing-decisions.md) — key architecture calls and why
+- [Attribute Governance](attribute-governance.md) — controlled external/internal attribute lists, import/export filtering, dynamic hash algorithm
 - [Open External Dependencies](open-external-dependencies.md) — things this design assumes but doesn't own

.claude/memory/attribute-governance.md

@@ -0,0 +1,82 @@
+# Attribute Governance (SA)
+
+Full detail lives in the "Attribute Governance" and "Egress Packing &
+Signing Selection" sections of
+`docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md`.
+
+- **Placeholder naming until enumerated:** external attributes
+  `SA-V3-Ext-01`…`SA-V3-Ext-15`, internal attributes `SA-V3-Int-01`…
+  `SA-V3-Int-NN`. Real names are an open item. Placeholder regexes:
+  external `^SA-V3-Ext-(0[1-9]|1[0-5])$`, internal `^SA-V3-Int-\d{2}$`.
+- **Content hash algorithm is configurable, not fixed.** It's a dynamic
+  property on the custom ingest processor (mirrored on egress), and the
+  chosen algorithm is itself recorded as a controlled SA attribute so it's
+  self-describing at verification time — no out-of-band assumption needed.
+  This resolves what was previously an unspecified gap: earlier drafts
+  named a signing algorithm (Ed25519) but never a content-hash algorithm.
+- **~15 controlled external attributes** (`SA-V3-Ext-##`) are **owned by
+  the external S3 handover interface spec**, not authored by this design —
+  this design just consumes them. Only a subset of the ~15 is mandatory;
+  the rest are optional (absence isn't an error). Eligible to carry
+  through from ingest to final destination; which ones actually get
+  packed/signed at a given egress point is a further, per-processor
+  selection (see egress regex below), so "the attribute list" elsewhere
+  means this controlled pool or a configured subset of it, not every
+  FlowFile attribute.
+- **The permissible read (input) and write (output) attribute sets are the
+  same list** for external S3 — one shared contract, not independently-
+  evolved read/write regexes. This resolves the write-side/read-side
+  reconciliation concern in principle; still need the actual handover spec
+  document to confirm exact key-casing and check for key collisions with
+  this design's own fallback-flag/size attributes (open item).
+- **Controlled internal attributes** (`SA-V3-Int-##`, count `NN` TBD) are
+  a separate set, used for SA's own routing/processing **and** for
+  **internal S3** usage: the reassembly/staging space
+  (`staging/<generation-id>/<chunk-index>` objects) and the replay buffer
+  (Replay DB's S3 tier for large files) are internal S3, distinct from
+  external S3 (the handover-spec-governed delivery boundary). Internal
+  attributes are never signed, never packed for external delivery — **but
+  they do cross the MR hop.** MR bridges the two NiFi chains *within* SA;
+  it is not SA's external boundary. Internal attributes are only stripped
+  at true external egress (final S3/HTTP/filesystem delivery out of SA),
+  not at MR, and not at internal S3 (which is inside the boundary).
+  How internal attributes are represented on internal S3 objects
+  (metadata/tags/JSON) is still open.
+- **Import filter:** delete-attributes-by-regex, non-matches against the
+  external allowlist regex dropped at ingest.
+- **External-egress filter:** delete-attributes-by-regex using the
+  **inverse** of the external allowlist regex — one step that strips
+  internal (`SA-V3-Int-##`) attributes and anything else uncontrolled at
+  once, without needing to separately enumerate "internal." Only
+  controlled external attributes reach S3/HTTP/filesystem destinations.
+- **Egress Regex property (write-side attribute selection):** each egress
+  processor has a configurable Regex property naming which controlled
+  external attributes get packed and signed at that point in the flow —
+  deliberately per-processor so a flow author can configure different
+  subsets at different flow positions. For S3, the same regex mechanism
+  selects which matched attributes get written as `x-amz-meta-*` user
+  metadata (when the matched set is under 2KB URL-encoded); attributes
+  that don't fit go into the JSON manifest trailer instead, per the
+  existing >2KB fallback path. This is a selection *within* the
+  permissible external attribute set (see above) — the permissible set
+  itself is shared between read and write, but which subset of it a given
+  egress point actually packs/promotes to metadata is still this design's
+  own per-processor choice.
+- **Ingest-time signing is optional**, not mandatory — a configurable
+  ingest-processor property. Egress-time signing (of the regex-selected
+  subset) is what actually produces the signed artifact; ingest signing,
+  when enabled, is just an earlier in-SA integrity checkpoint.
+- **Ingest-time validation is conditional on presence, not optional
+  itself:** unsigned data at ingest is fine (signing is optional). But if
+  data arrives at ingest *already signed* (e.g. replay, or from an earlier
+  stage), that signature must be validated on arrival — invalid signature
+  routes to failure, same as any other corruption case.
+- **S3 JSON-fallback flag always pairs with a size attribute:** whenever
+  the >2KB fallback flag is set on an S3 object, this design also writes a
+  second attribute giving the JSON trailer's byte size. Together they let
+  a reader do one suffix-range `GET` (`Range: bytes=-N`) to fetch exactly
+  the appended JSON and parse it into attributes — no footer read needed
+  on S3 (footer remains the fallback mechanism for filesystem/HTTP, which
+  lack a tag/metadata side-channel).
+
+See [[packaging-signing-decisions]] and [[project-purpose]].

.claude/memory/packaging-signing-decisions.md

@@ -37,3 +37,9 @@ reversed an earlier direction during design review:
 - **S3-vs-XFS storage backend split by file size is unresolved** — open
   question whether S3 can serve small files efficiently enough to drop
   the XFS tier entirely.
+- **Attribute list is a controlled allowlist, not "all FlowFile
+  attributes."** ~15 controlled external attributes carry through
+  ingest-to-delivery (these are what get signed/packed); a separate
+  controlled internal set is SA-only and stripped before export. Import
+  drops uncontrolled external attributes; export drops internal ones. See
+  [[attribute-governance]].

docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md

@@ -138,7 +138,9 @@ true entry into SA — consistent with the single-pass design below.
                      is just the one-chunk case.
                                          |
                                          v
-                     Ordered attribute list (incl. content hash, chunk
+                     Optionally (ingest-time signing is a configurable
+                     property, see Attribute Governance): ordered
+                     attribute list (incl. content hash, chunk
                      descriptors) is clear-signed in-process (Bouncy
                      Castle OpenPGP, key via Controller Service — see
                      Signing) and stored as a new attribute.
@@ -186,6 +188,117 @@ the main justification for a front door outside the JVM. The external
 service's role is narrowed to post-NiFi work that never needs to see raw
 content: assembly, monitoring, cleanup, and catalog indexing.
 
+## Attribute Governance
+
+**Naming convention (placeholder):** until the concrete attribute list is
+enumerated (see [Open Items](#open-items-for-implementation-planning)),
+controlled attributes are referred to by placeholder names — external
+attributes as `SA-V3-Ext-01` … `SA-V3-Ext-15`, internal attributes as
+`SA-V3-Int-01` … `SA-V3-Int-NN`. These stand in for the real names
+throughout this doc and in code until finalized.
+
+**Placeholder regexes**, used for matching against these placeholder
+names until the real attribute list replaces them:
+- External allowlist: `^SA-V3-Ext-(0[1-9]|1[0-5])$` (matches
+  `SA-V3-Ext-01` through `SA-V3-Ext-15`).
+- Internal: `^SA-V3-Int-\d{2}$` (count not yet fixed — placeholder covers
+  any two-digit index until `NN` is finalized).
+
+The content hash algorithm is not fixed by this design. It is exposed as a
+**dynamic property** on the custom ingest processor (and mirrored on the
+corresponding egress/delivery processor), so it can be changed per
+deployment without a code change. Whichever algorithm is configured is
+recorded as one of SA's controlled attributes and carried with the
+FlowFile, so the algorithm used is self-describing at verification time
+rather than assumed out-of-band.
+
+SA maintains a fixed set of roughly **15 controlled external attributes**
+(`SA-V3-Ext-##`) — attributes with an externally-defined meaning/contract,
+as opposed to SA-internal bookkeeping — that are carried through unchanged
+from ingest to final destination. This list is **owned by the existing S3
+handover interface spec** (see [Storage Backend Handover](#storage-backend-handover)),
+not authored by this design — this design consumes it. Only a **subset**
+of the ~15 are mandatory; the rest are optional, so their absence on a
+given FlowFile is not itself an error. **The permissible input/read
+attribute set is the same as the permissible write/output attribute set**
+— one list governs both directions of external S3 interaction, not two
+independently-defined ones (this resolves the write-side/read-side
+reconciliation concern raised earlier — see
+[Open Items](#open-items-for-implementation-planning) for what's still
+needed to finalize it). Which of the permissible attributes actually get
+packed and signed at a given egress point is a further, configurable
+selection — see
+[Egress Packing & Signing Selection](#egress-packing--signing-selection)
+below — so "the attribute list" referenced elsewhere in this doc means
+this controlled set (or a configured subset of it), not the full/arbitrary
+FlowFile attribute map. The concrete list is **not yet provided** — only
+the placeholder names/regex above exist today.
+
+SA also uses a separate set of **controlled internal attributes**
+(`SA-V3-Int-##`) for its own routing and processing needs, and — new —
+for **internal S3 usage**: the reassembly/staging space (per-chunk
+`staging/<generation-id>/<chunk-index>` objects) and the replay buffer
+(the Replay DB's S3 tier for large files) are **internal S3**, distinct
+from the **external S3** interaction governed by the handover spec above.
+Internal S3 objects may carry this additional permissible-internal-
+attribute set — itself also not yet provided, count TBD (`NN`). Internal
+attributes are never part of the externally-defined contract and are never
+packed or signed for delivery — but they **are** allowed to cross the MR
+hop: MR bridges the two NiFi chains *within* SA, not SA's external
+boundary, so internal attributes travel with the FlowFile through MR like
+any other attribute. They're stripped only at SA's true external egress
+(final delivery to S3/HTTP/filesystem, outside SA) — internal S3 (staging,
+replay buffer) is itself inside that boundary, not the egress point.
+
+Two boundary filters enforce this split, both implemented as a
+delete-attributes-by-regex step (a standard NiFi capability, e.g. an
+`UpdateAttribute` "Delete Attributes Expression"):
+- **On import (ingest):** attributes **not** matching the external
+  allowlist regex (`^SA-V3-Ext-(0[1-9]|1[0-5])$`) are deleted.
+  Uncontrolled/unexpected external input never enters SA's attribute
+  space.
+- **On external egress (final delivery — not the MR hop):** the
+  **inverse** of the external allowlist regex is applied to delete
+  everything that does *not* match `^SA-V3-Ext-(0[1-9]|1[0-5])$`. This is
+  a single step that removes internal (`SA-V3-Int-##`) attributes and any
+  other stray/uncontrolled attributes at once — it doesn't depend on
+  separately enumerating what "internal" means, only on what counts as a
+  valid external attribute. Only controlled external attributes reach S3/
+  HTTP/filesystem destinations.
+
+### Egress Packing & Signing Selection
+
+Each egress processor exposes a **Regex property** naming which of the
+controlled external (`SA-V3-Ext-##`) attributes get packed and signed at
+that point in the flow. This is deliberately per-processor rather than one
+fixed set for every egress point, so a flow author can configure a
+different subset at each position in the flow depending on what that
+position needs to expose.
+
+For S3 destinations specifically:
+- Attributes matched by the regex whose combined `key=value` size is
+  **under 2KB when URL-encoded** are written as S3 user metadata
+  (`x-amz-meta-*`) — also selected via the same Regex property.
+- Attributes that don't fit under that 2KB threshold are instead appended
+  in JSON form per [Packaged Artifact Format](#packaged-artifact-format),
+  matching the existing >2KB fallback path described in
+  [Storage Backend Handover](#storage-backend-handover).
+
+This regex-driven selection governs the **write** side — which attributes
+this design packs or promotes to metadata. It's a distinct concern from
+the **read**-side regex in the existing S3 handover interface spec (which
+maps `x-amz-meta-*` keys back to NiFi attributes on read) — the two need
+to be reconciled so a value written via this design's regex is correctly
+recognized on read by that existing spec. Not yet confirmed; see
+[Open Items](#open-items-for-implementation-planning).
+
+**Signing at ingest is optional**, not mandatory — a configurable property
+on the ingest processor (see also [Signing](#signing)). Egress-time
+packing/signing via the Regex property above is what actually determines
+the contents of the final signed, packed artifact; ingest-time signing,
+when enabled, is an earlier integrity checkpoint within SA, not a
+substitute for it.
+
 ## Packaged Artifact Format
 
 This is the format used for filesystem/XFS delivery always, and for S3
@@ -220,9 +333,12 @@ multipart upload) can't end up in that split state.
    |   agnostic; works identically on S3, filesystem, and HTTP  |
    +---------------------------------------------------------+
 
-   Optional S3 object tag (attr-size = trailer byte length) as an
+   S3 object tag/attribute (attr-size = trailer byte length), written
-   accelerant: lets the catalog DB or a reconciliation scan jump straight
+   whenever the JSON-fallback flag is set (see Storage Backend Handover ->
-   to the trailer's byte range without reading the footer first.
+   S3 path): lets a reader jump straight to the trailer's byte range via a
+   suffix-range GET without reading the footer first. On S3 this is the
+   primary mechanism; the footer remains the storage-agnostic fallback for
+   filesystem/HTTP, which have no equivalent tag/metadata mechanism.
 ```
 
 Manifest is written **after** content by construction — the ingest pass is
@@ -244,6 +360,17 @@ content-then-trailer-then-footer shape regardless of size.
 
 ## Signing
 
+- **Where signing is triggered:** ingest-time signing is optional
+  (configurable per ingest processor); egress-time signing, of the
+  attribute subset selected by the egress Regex property, is what actually
+  produces the signed content shipped in the final artifact — see
+  [Egress Packing & Signing Selection](#egress-packing--signing-selection).
+- **Ingest-time validation:** incoming data at ingest may or may not
+  already carry a signature (e.g. from an earlier stage, or on replay).
+  Absence is fine — ingest signing is optional, per above. But if a
+  signature **is** already present on ingest, it must be validated at that
+  point rather than passed through unchecked; see
+  [Error Handling](#error-handling) for the failure case.
 - **Format:** OpenPGP detached-style trailer via `gpg --clearsign`
   semantics — clear-sign the **canonical ordered attribute list** (sorted
   by attribute key, including the content hash and any chunk descriptors),
@@ -306,21 +433,43 @@ The JSON-trailer packaged artifact format is shared by both backends, but
 S3 additionally supports an existing metadata-based mechanism for small
 attribute sets that avoids the trailer entirely — see below.
 
+This section (S3 path) covers **external S3** — SA's true delivery
+boundary. It's distinct from **internal S3** — the reassembly/staging
+space and replay buffer used *within* SA — which is governed by its own
+additional permissible-internal-attribute set instead of the handover spec
+below; see [Attribute Governance](#attribute-governance).
+
 ### S3 path
 
-**Attribute handover on S3 follows an existing, established handover
+**Attribute handover on external S3 follows an existing, established
-interface specification already used elsewhere in this environment — this
+handover interface specification already used elsewhere in this
-design integrates with it rather than replacing it.** Under that spec:
+environment — this design integrates with it rather than replacing it.**
+That spec defines the ~15 controlled external attributes (`SA-V3-Ext-##`
+placeholder), and the **same list governs both directions** — what's
+permissible to read is permissible to write, so there's one shared
+attribute contract for external S3, not independently-evolved read and
+write lists. Under that spec:
 
 - Attributes are written as S3 user object metadata (`x-amz-meta-*`
   headers) when small enough to fit. A regular expression (defined by that
   existing spec, not this one) identifies which user metadata keys get
   turned into NiFi attributes on read — so consuming the attributes is a
   single `HeadObject` call, no content touch, not even a suffix-range read.
+  Which attributes from that permissible list actually get *written* as
+  metadata at a given egress point is a separate, this-design-owned
+  selection (a configured subset, not necessarily all permissible ones) —
+  see [Egress Packing & Signing Selection](#egress-packing--signing-selection).
 - A flag (also part of that existing spec) records whether the attribute
   set exceeded S3's ~2KB user-metadata limit. When it does, the object
   falls back to **this design's** JSON manifest trailer (Packaged Artifact
   Format) instead of metadata.
+- Whenever that fallback flag is set, **this design** also writes a second
+  attribute giving the byte size of the appended JSON manifest trailer.
+  Together, the two let a reader do a single suffix-range `GET` —
+  `Range: bytes=-N` with `N` from that size attribute — to fetch exactly
+  the appended JSON and parse it directly into attributes, with no need to
+  read the fixed footer first (see
+  [Packaged Artifact Format](#packaged-artifact-format)).
 - The exact regex pattern and flag key/tag name live in the existing
   handover interface spec document, not reproduced here — see
   [Open Items](#open-items-for-implementation-planning).
@@ -397,6 +546,8 @@ FlowFiles via the same ingest path used for first-time delivery.
 | Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. |
 | Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists |
 | Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch |
+| Data arrives at ingest with no existing signature | Not an error — ingest-time signing is optional (see Signing); proceeds normally |
+| Data arrives at ingest already signed, signature fails validation | Route to failure — an already-signed FlowFile at ingest is validated on arrival, not passed through unchecked |
 | Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers |
 
 ## Testing
@@ -449,3 +600,24 @@ environment.
   keys to NiFi attributes, and the exact flag/tag name and value convention
   used to signal the >2KB fallback to the JSON trailer. This design assumes
   that spec's behavior but doesn't reproduce its details.
+- **Enumerate the controlled attribute lists**, to be provided later per
+  [Attribute Governance](#attribute-governance):
+  - The ~15 controlled external attributes (`SA-V3-Ext-##` placeholders),
+    owned by the external S3 handover interface spec — names, types, and
+    which subset of the ~15 is mandatory vs. optional.
+  - The separate controlled internal attribute list (`SA-V3-Int-##`
+    placeholders, count `NN` not yet fixed) — used for SA routing/
+    processing generally, and specifically for internal S3 (staging/
+    replay buffer) objects.
+- **Confirm the write-side/read-side regex reconciliation is fully closed
+  in practice, not just in principle.** It's now established that the
+  permissible external attribute set is the same for reads and writes (one
+  list, not two independently-defined ones) — but this still needs the
+  actual document to confirm: exact key-casing convention (S3 lowercases
+  `x-amz-meta-*` header names on read), and that the flag/size attributes
+  this design writes for the >2KB fallback don't collide with any of the
+  ~15 external attribute keys.
+- **Define how internal attributes are represented on internal S3
+  objects** (staging, replay buffer) — S3 user metadata vs. tags vs. the
+  JSON-trailer mechanism, and whether the same 2KB threshold applies
+  there.