Add attribute governance design: controlled attribute lists, egress regex selection, ingest signature validation

Defines the SA-V3-Ext/Int placeholder attribute scheme (external list
owned by the S3 handover spec, internal list for SA routing + internal
S3 usage), the per-egress Regex property for packing/signing selection,
the inverse-regex delete step at true external egress, optional
ingest-time signing with mandatory validation when already signed, and
the S3 JSON-fallback size attribute for suffix-range trailer reads.
This commit is contained in:
Ubuntu
2026-07-28 07:49:21 +00:00
parent 45f4dc733d
commit 0bae727a26
4 changed files with 268 additions and 7 deletions
+82
View File
@@ -0,0 +1,82 @@
# Attribute Governance (SA)
Full detail lives in the "Attribute Governance" and "Egress Packing &
Signing Selection" sections of
`docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md`.
- **Placeholder naming until enumerated:** external attributes
`SA-V3-Ext-01``SA-V3-Ext-15`, internal attributes `SA-V3-Int-01`
`SA-V3-Int-NN`. Real names are an open item. Placeholder regexes:
external `^SA-V3-Ext-(0[1-9]|1[0-5])$`, internal `^SA-V3-Int-\d{2}$`.
- **Content hash algorithm is configurable, not fixed.** It's a dynamic
property on the custom ingest processor (mirrored on egress), and the
chosen algorithm is itself recorded as a controlled SA attribute so it's
self-describing at verification time — no out-of-band assumption needed.
This resolves what was previously an unspecified gap: earlier drafts
named a signing algorithm (Ed25519) but never a content-hash algorithm.
- **~15 controlled external attributes** (`SA-V3-Ext-##`) are **owned by
the external S3 handover interface spec**, not authored by this design —
this design just consumes them. Only a subset of the ~15 is mandatory;
the rest are optional (absence isn't an error). Eligible to carry
through from ingest to final destination; which ones actually get
packed/signed at a given egress point is a further, per-processor
selection (see egress regex below), so "the attribute list" elsewhere
means this controlled pool or a configured subset of it, not every
FlowFile attribute.
- **The permissible read (input) and write (output) attribute sets are the
same list** for external S3 — one shared contract, not independently-
evolved read/write regexes. This resolves the write-side/read-side
reconciliation concern in principle; still need the actual handover spec
document to confirm exact key-casing and check for key collisions with
this design's own fallback-flag/size attributes (open item).
- **Controlled internal attributes** (`SA-V3-Int-##`, count `NN` TBD) are
a separate set, used for SA's own routing/processing **and** for
**internal S3** usage: the reassembly/staging space
(`staging/<generation-id>/<chunk-index>` objects) and the replay buffer
(Replay DB's S3 tier for large files) are internal S3, distinct from
external S3 (the handover-spec-governed delivery boundary). Internal
attributes are never signed, never packed for external delivery — **but
they do cross the MR hop.** MR bridges the two NiFi chains *within* SA;
it is not SA's external boundary. Internal attributes are only stripped
at true external egress (final S3/HTTP/filesystem delivery out of SA),
not at MR, and not at internal S3 (which is inside the boundary).
How internal attributes are represented on internal S3 objects
(metadata/tags/JSON) is still open.
- **Import filter:** delete-attributes-by-regex, non-matches against the
external allowlist regex dropped at ingest.
- **External-egress filter:** delete-attributes-by-regex using the
**inverse** of the external allowlist regex — one step that strips
internal (`SA-V3-Int-##`) attributes and anything else uncontrolled at
once, without needing to separately enumerate "internal." Only
controlled external attributes reach S3/HTTP/filesystem destinations.
- **Egress Regex property (write-side attribute selection):** each egress
processor has a configurable Regex property naming which controlled
external attributes get packed and signed at that point in the flow —
deliberately per-processor so a flow author can configure different
subsets at different flow positions. For S3, the same regex mechanism
selects which matched attributes get written as `x-amz-meta-*` user
metadata (when the matched set is under 2KB URL-encoded); attributes
that don't fit go into the JSON manifest trailer instead, per the
existing >2KB fallback path. This is a selection *within* the
permissible external attribute set (see above) — the permissible set
itself is shared between read and write, but which subset of it a given
egress point actually packs/promotes to metadata is still this design's
own per-processor choice.
- **Ingest-time signing is optional**, not mandatory — a configurable
ingest-processor property. Egress-time signing (of the regex-selected
subset) is what actually produces the signed artifact; ingest signing,
when enabled, is just an earlier in-SA integrity checkpoint.
- **Ingest-time validation is conditional on presence, not optional
itself:** unsigned data at ingest is fine (signing is optional). But if
data arrives at ingest *already signed* (e.g. replay, or from an earlier
stage), that signature must be validated on arrival — invalid signature
routes to failure, same as any other corruption case.
- **S3 JSON-fallback flag always pairs with a size attribute:** whenever
the >2KB fallback flag is set on an S3 object, this design also writes a
second attribute giving the JSON trailer's byte size. Together they let
a reader do one suffix-range `GET` (`Range: bytes=-N`) to fetch exactly
the appended JSON and parse it into attributes — no footer read needed
on S3 (footer remains the fallback mechanism for filesystem/HTTP, which
lack a tag/metadata side-channel).
See [[packaging-signing-decisions]] and [[project-purpose]].