Ubuntu
0bae727a26
Defines the SA-V3-Ext/Int placeholder attribute scheme (external list owned by the S3 handover spec, internal list for SA routing + internal S3 usage), the per-egress Regex property for packing/signing selection, the inverse-regex delete step at true external egress, optional ingest-time signing with mandatory validation when already signed, and the S3 JSON-fallback size attribute for suffix-range trailer reads.
83 lines
5.2 KiB
Markdown
83 lines
5.2 KiB
Markdown
# Attribute Governance (SA)
|
|
|
|
Full detail lives in the "Attribute Governance" and "Egress Packing &
|
|
Signing Selection" sections of
|
|
`docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md`.
|
|
|
|
- **Placeholder naming until enumerated:** external attributes
|
|
`SA-V3-Ext-01`…`SA-V3-Ext-15`, internal attributes `SA-V3-Int-01`…
|
|
`SA-V3-Int-NN`. Real names are an open item. Placeholder regexes:
|
|
external `^SA-V3-Ext-(0[1-9]|1[0-5])$`, internal `^SA-V3-Int-\d{2}$`.
|
|
- **Content hash algorithm is configurable, not fixed.** It's a dynamic
|
|
property on the custom ingest processor (mirrored on egress), and the
|
|
chosen algorithm is itself recorded as a controlled SA attribute so it's
|
|
self-describing at verification time — no out-of-band assumption needed.
|
|
This resolves what was previously an unspecified gap: earlier drafts
|
|
named a signing algorithm (Ed25519) but never a content-hash algorithm.
|
|
- **~15 controlled external attributes** (`SA-V3-Ext-##`) are **owned by
|
|
the external S3 handover interface spec**, not authored by this design —
|
|
this design just consumes them. Only a subset of the ~15 is mandatory;
|
|
the rest are optional (absence isn't an error). Eligible to carry
|
|
through from ingest to final destination; which ones actually get
|
|
packed/signed at a given egress point is a further, per-processor
|
|
selection (see egress regex below), so "the attribute list" elsewhere
|
|
means this controlled pool or a configured subset of it, not every
|
|
FlowFile attribute.
|
|
- **The permissible read (input) and write (output) attribute sets are the
|
|
same list** for external S3 — one shared contract, not independently-
|
|
evolved read/write regexes. This resolves the write-side/read-side
|
|
reconciliation concern in principle; still need the actual handover spec
|
|
document to confirm exact key-casing and check for key collisions with
|
|
this design's own fallback-flag/size attributes (open item).
|
|
- **Controlled internal attributes** (`SA-V3-Int-##`, count `NN` TBD) are
|
|
a separate set, used for SA's own routing/processing **and** for
|
|
**internal S3** usage: the reassembly/staging space
|
|
(`staging/<generation-id>/<chunk-index>` objects) and the replay buffer
|
|
(Replay DB's S3 tier for large files) are internal S3, distinct from
|
|
external S3 (the handover-spec-governed delivery boundary). Internal
|
|
attributes are never signed, never packed for external delivery — **but
|
|
they do cross the MR hop.** MR bridges the two NiFi chains *within* SA;
|
|
it is not SA's external boundary. Internal attributes are only stripped
|
|
at true external egress (final S3/HTTP/filesystem delivery out of SA),
|
|
not at MR, and not at internal S3 (which is inside the boundary).
|
|
How internal attributes are represented on internal S3 objects
|
|
(metadata/tags/JSON) is still open.
|
|
- **Import filter:** delete-attributes-by-regex, non-matches against the
|
|
external allowlist regex dropped at ingest.
|
|
- **External-egress filter:** delete-attributes-by-regex using the
|
|
**inverse** of the external allowlist regex — one step that strips
|
|
internal (`SA-V3-Int-##`) attributes and anything else uncontrolled at
|
|
once, without needing to separately enumerate "internal." Only
|
|
controlled external attributes reach S3/HTTP/filesystem destinations.
|
|
- **Egress Regex property (write-side attribute selection):** each egress
|
|
processor has a configurable Regex property naming which controlled
|
|
external attributes get packed and signed at that point in the flow —
|
|
deliberately per-processor so a flow author can configure different
|
|
subsets at different flow positions. For S3, the same regex mechanism
|
|
selects which matched attributes get written as `x-amz-meta-*` user
|
|
metadata (when the matched set is under 2KB URL-encoded); attributes
|
|
that don't fit go into the JSON manifest trailer instead, per the
|
|
existing >2KB fallback path. This is a selection *within* the
|
|
permissible external attribute set (see above) — the permissible set
|
|
itself is shared between read and write, but which subset of it a given
|
|
egress point actually packs/promotes to metadata is still this design's
|
|
own per-processor choice.
|
|
- **Ingest-time signing is optional**, not mandatory — a configurable
|
|
ingest-processor property. Egress-time signing (of the regex-selected
|
|
subset) is what actually produces the signed artifact; ingest signing,
|
|
when enabled, is just an earlier in-SA integrity checkpoint.
|
|
- **Ingest-time validation is conditional on presence, not optional
|
|
itself:** unsigned data at ingest is fine (signing is optional). But if
|
|
data arrives at ingest *already signed* (e.g. replay, or from an earlier
|
|
stage), that signature must be validated on arrival — invalid signature
|
|
routes to failure, same as any other corruption case.
|
|
- **S3 JSON-fallback flag always pairs with a size attribute:** whenever
|
|
the >2KB fallback flag is set on an S3 object, this design also writes a
|
|
second attribute giving the JSON trailer's byte size. Together they let
|
|
a reader do one suffix-range `GET` (`Range: bytes=-N`) to fetch exactly
|
|
the appended JSON and parse it into attributes — no footer read needed
|
|
on S3 (footer remains the fallback mechanism for filesystem/HTTP, which
|
|
lack a tag/metadata side-channel).
|
|
|
|
See [[packaging-signing-decisions]] and [[project-purpose]].
|