GrandPlan/.claude/memory/attribute-governance.md

Attribute Governance (SA)

Full detail lives in the "Attribute Governance" and "Egress Packing & Signing Selection" sections of docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md.

• Placeholder naming until enumerated: external attributes SA-V3-Ext-01…SA-V3-Ext-15, internal attributes SA-V3-Int-01… SA-V3-Int-NN. Real names are an open item. Placeholder regexes: external ^SA-V3-Ext-(0[1-9]|1[0-5])$, internal ^SA-V3-Int-\d{2}$.
• Content hash algorithm is configurable, not fixed. It's a dynamic property on the custom ingest processor (mirrored on egress), and the chosen algorithm is itself recorded as a controlled SA attribute so it's self-describing at verification time — no out-of-band assumption needed. This resolves what was previously an unspecified gap: earlier drafts named a signing algorithm (Ed25519) but never a content-hash algorithm.
• ~15 controlled external attributes (SA-V3-Ext-##) are owned by the external S3 handover interface spec, not authored by this design — this design just consumes them. Only a subset of the ~15 is mandatory; the rest are optional (absence isn't an error). Eligible to carry through from ingest to final destination; which ones actually get packed/signed at a given egress point is a further, per-processor selection (see egress regex below), so "the attribute list" elsewhere means this controlled pool or a configured subset of it, not every FlowFile attribute.
• The permissible read (input) and write (output) attribute sets are the same list for external S3 — one shared contract, not independently- evolved read/write regexes. This resolves the write-side/read-side reconciliation concern in principle; still need the actual handover spec document to confirm exact key-casing and check for key collisions with this design's own fallback-flag/size attributes (open item).
• Controlled internal attributes (SA-V3-Int-##, count NN TBD) are a separate set, used for SA's own routing/processing and for internal S3 usage: the reassembly/staging space (staging/<generation-id>/<chunk-index> objects) and the replay buffer (Replay DB's S3 tier for large files) are internal S3, distinct from external S3 (the handover-spec-governed delivery boundary). Internal attributes are never signed, never packed for external delivery — but they do cross the MR hop. MR bridges the two NiFi chains within SA; it is not SA's external boundary. Internal attributes are only stripped at true external egress (final S3/HTTP/filesystem delivery out of SA), not at MR, and not at internal S3 (which is inside the boundary). How internal attributes are represented on internal S3 objects (metadata/tags/JSON) is still open.
• Import filter: delete-attributes-by-regex, non-matches against the external allowlist regex dropped at ingest.
• External-egress filter: delete-attributes-by-regex using the inverse of the external allowlist regex — one step that strips internal (SA-V3-Int-##) attributes and anything else uncontrolled at once, without needing to separately enumerate "internal." Only controlled external attributes reach S3/HTTP/filesystem destinations.
• Egress Regex property (write-side attribute selection): each egress processor has a configurable Regex property naming which controlled external attributes get packed and signed at that point in the flow — deliberately per-processor so a flow author can configure different subsets at different flow positions. For S3, the same regex mechanism selects which matched attributes get written as x-amz-meta-* user metadata (when the matched set is under 2KB URL-encoded); attributes that don't fit go into the JSON manifest trailer instead, per the existing >2KB fallback path. This is a selection within the permissible external attribute set (see above) — the permissible set itself is shared between read and write, but which subset of it a given egress point actually packs/promotes to metadata is still this design's own per-processor choice.
• Ingest-time signing is optional, not mandatory — a configurable ingest-processor property. Egress-time signing (of the regex-selected subset) is what actually produces the signed artifact; ingest signing, when enabled, is just an earlier in-SA integrity checkpoint.
• Ingest-time validation is conditional on presence, not optional itself: unsigned data at ingest is fine (signing is optional). But if data arrives at ingest already signed (e.g. replay, or from an earlier stage), that signature must be validated on arrival — invalid signature routes to failure, same as any other corruption case.
• S3 JSON-fallback flag always pairs with a size attribute: whenever the >2KB fallback flag is set on an S3 object, this design also writes a second attribute giving the JSON trailer's byte size. Together they let a reader do one suffix-range GET (Range: bytes=-N) to fetch exactly the appended JSON and parse it into attributes — no footer read needed on S3 (footer remains the fallback mechanism for filesystem/HTTP, which lack a tag/metadata side-channel).

See packaging-signing-decisions and project-purpose.