From 0bae727a26267a71d64e2a8b1860f378282cf757 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Tue, 28 Jul 2026 07:49:21 +0000 Subject: [PATCH] Add attribute governance design: controlled attribute lists, egress regex selection, ingest signature validation Defines the SA-V3-Ext/Int placeholder attribute scheme (external list owned by the S3 handover spec, internal list for SA routing + internal S3 usage), the per-egress Regex property for packing/signing selection, the inverse-regex delete step at true external egress, optional ingest-time signing with mandatory validation when already signed, and the S3 JSON-fallback size attribute for suffix-range trailer reads. --- .claude/memory/MEMORY.md | 1 + .claude/memory/attribute-governance.md | 82 ++++++++ .claude/memory/packaging-signing-decisions.md | 6 + ...07-28-flowfile-packaging-signing-design.md | 186 +++++++++++++++++- 4 files changed, 268 insertions(+), 7 deletions(-) create mode 100644 .claude/memory/attribute-governance.md diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index d6754fc..dc38ed2 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -3,4 +3,5 @@ - [Project Purpose](project-purpose.md) — why this repo exists, the two core problems being solved - [SA/MR Topology](topology.md) — deployment topology terminology and shape - [Packaging & Signing Design Decisions](packaging-signing-decisions.md) — key architecture calls and why +- [Attribute Governance](attribute-governance.md) — controlled external/internal attribute lists, import/export filtering, dynamic hash algorithm - [Open External Dependencies](open-external-dependencies.md) — things this design assumes but doesn't own diff --git a/.claude/memory/attribute-governance.md b/.claude/memory/attribute-governance.md new file mode 100644 index 0000000..d9dcdae --- /dev/null +++ b/.claude/memory/attribute-governance.md @@ -0,0 +1,82 @@ +# Attribute Governance (SA) + +Full detail lives in the "Attribute Governance" and "Egress Packing & +Signing Selection" sections of +`docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md`. + +- **Placeholder naming until enumerated:** external attributes + `SA-V3-Ext-01`…`SA-V3-Ext-15`, internal attributes `SA-V3-Int-01`… + `SA-V3-Int-NN`. Real names are an open item. Placeholder regexes: + external `^SA-V3-Ext-(0[1-9]|1[0-5])$`, internal `^SA-V3-Int-\d{2}$`. +- **Content hash algorithm is configurable, not fixed.** It's a dynamic + property on the custom ingest processor (mirrored on egress), and the + chosen algorithm is itself recorded as a controlled SA attribute so it's + self-describing at verification time — no out-of-band assumption needed. + This resolves what was previously an unspecified gap: earlier drafts + named a signing algorithm (Ed25519) but never a content-hash algorithm. +- **~15 controlled external attributes** (`SA-V3-Ext-##`) are **owned by + the external S3 handover interface spec**, not authored by this design — + this design just consumes them. Only a subset of the ~15 is mandatory; + the rest are optional (absence isn't an error). Eligible to carry + through from ingest to final destination; which ones actually get + packed/signed at a given egress point is a further, per-processor + selection (see egress regex below), so "the attribute list" elsewhere + means this controlled pool or a configured subset of it, not every + FlowFile attribute. +- **The permissible read (input) and write (output) attribute sets are the + same list** for external S3 — one shared contract, not independently- + evolved read/write regexes. This resolves the write-side/read-side + reconciliation concern in principle; still need the actual handover spec + document to confirm exact key-casing and check for key collisions with + this design's own fallback-flag/size attributes (open item). +- **Controlled internal attributes** (`SA-V3-Int-##`, count `NN` TBD) are + a separate set, used for SA's own routing/processing **and** for + **internal S3** usage: the reassembly/staging space + (`staging//` objects) and the replay buffer + (Replay DB's S3 tier for large files) are internal S3, distinct from + external S3 (the handover-spec-governed delivery boundary). Internal + attributes are never signed, never packed for external delivery — **but + they do cross the MR hop.** MR bridges the two NiFi chains *within* SA; + it is not SA's external boundary. Internal attributes are only stripped + at true external egress (final S3/HTTP/filesystem delivery out of SA), + not at MR, and not at internal S3 (which is inside the boundary). + How internal attributes are represented on internal S3 objects + (metadata/tags/JSON) is still open. +- **Import filter:** delete-attributes-by-regex, non-matches against the + external allowlist regex dropped at ingest. +- **External-egress filter:** delete-attributes-by-regex using the + **inverse** of the external allowlist regex — one step that strips + internal (`SA-V3-Int-##`) attributes and anything else uncontrolled at + once, without needing to separately enumerate "internal." Only + controlled external attributes reach S3/HTTP/filesystem destinations. +- **Egress Regex property (write-side attribute selection):** each egress + processor has a configurable Regex property naming which controlled + external attributes get packed and signed at that point in the flow — + deliberately per-processor so a flow author can configure different + subsets at different flow positions. For S3, the same regex mechanism + selects which matched attributes get written as `x-amz-meta-*` user + metadata (when the matched set is under 2KB URL-encoded); attributes + that don't fit go into the JSON manifest trailer instead, per the + existing >2KB fallback path. This is a selection *within* the + permissible external attribute set (see above) — the permissible set + itself is shared between read and write, but which subset of it a given + egress point actually packs/promotes to metadata is still this design's + own per-processor choice. +- **Ingest-time signing is optional**, not mandatory — a configurable + ingest-processor property. Egress-time signing (of the regex-selected + subset) is what actually produces the signed artifact; ingest signing, + when enabled, is just an earlier in-SA integrity checkpoint. +- **Ingest-time validation is conditional on presence, not optional + itself:** unsigned data at ingest is fine (signing is optional). But if + data arrives at ingest *already signed* (e.g. replay, or from an earlier + stage), that signature must be validated on arrival — invalid signature + routes to failure, same as any other corruption case. +- **S3 JSON-fallback flag always pairs with a size attribute:** whenever + the >2KB fallback flag is set on an S3 object, this design also writes a + second attribute giving the JSON trailer's byte size. Together they let + a reader do one suffix-range `GET` (`Range: bytes=-N`) to fetch exactly + the appended JSON and parse it into attributes — no footer read needed + on S3 (footer remains the fallback mechanism for filesystem/HTTP, which + lack a tag/metadata side-channel). + +See [[packaging-signing-decisions]] and [[project-purpose]]. diff --git a/.claude/memory/packaging-signing-decisions.md b/.claude/memory/packaging-signing-decisions.md index e064243..f778be0 100644 --- a/.claude/memory/packaging-signing-decisions.md +++ b/.claude/memory/packaging-signing-decisions.md @@ -37,3 +37,9 @@ reversed an earlier direction during design review: - **S3-vs-XFS storage backend split by file size is unresolved** — open question whether S3 can serve small files efficiently enough to drop the XFS tier entirely. +- **Attribute list is a controlled allowlist, not "all FlowFile + attributes."** ~15 controlled external attributes carry through + ingest-to-delivery (these are what get signed/packed); a separate + controlled internal set is SA-only and stripped before export. Import + drops uncontrolled external attributes; export drops internal ones. See + [[attribute-governance]]. diff --git a/docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md b/docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md index ed85e9f..45217e0 100644 --- a/docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md +++ b/docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md @@ -138,7 +138,9 @@ true entry into SA — consistent with the single-pass design below. is just the one-chunk case. | v - Ordered attribute list (incl. content hash, chunk + Optionally (ingest-time signing is a configurable + property, see Attribute Governance): ordered + attribute list (incl. content hash, chunk descriptors) is clear-signed in-process (Bouncy Castle OpenPGP, key via Controller Service — see Signing) and stored as a new attribute. @@ -186,6 +188,117 @@ the main justification for a front door outside the JVM. The external service's role is narrowed to post-NiFi work that never needs to see raw content: assembly, monitoring, cleanup, and catalog indexing. +## Attribute Governance + +**Naming convention (placeholder):** until the concrete attribute list is +enumerated (see [Open Items](#open-items-for-implementation-planning)), +controlled attributes are referred to by placeholder names — external +attributes as `SA-V3-Ext-01` … `SA-V3-Ext-15`, internal attributes as +`SA-V3-Int-01` … `SA-V3-Int-NN`. These stand in for the real names +throughout this doc and in code until finalized. + +**Placeholder regexes**, used for matching against these placeholder +names until the real attribute list replaces them: +- External allowlist: `^SA-V3-Ext-(0[1-9]|1[0-5])$` (matches + `SA-V3-Ext-01` through `SA-V3-Ext-15`). +- Internal: `^SA-V3-Int-\d{2}$` (count not yet fixed — placeholder covers + any two-digit index until `NN` is finalized). + +The content hash algorithm is not fixed by this design. It is exposed as a +**dynamic property** on the custom ingest processor (and mirrored on the +corresponding egress/delivery processor), so it can be changed per +deployment without a code change. Whichever algorithm is configured is +recorded as one of SA's controlled attributes and carried with the +FlowFile, so the algorithm used is self-describing at verification time +rather than assumed out-of-band. + +SA maintains a fixed set of roughly **15 controlled external attributes** +(`SA-V3-Ext-##`) — attributes with an externally-defined meaning/contract, +as opposed to SA-internal bookkeeping — that are carried through unchanged +from ingest to final destination. This list is **owned by the existing S3 +handover interface spec** (see [Storage Backend Handover](#storage-backend-handover)), +not authored by this design — this design consumes it. Only a **subset** +of the ~15 are mandatory; the rest are optional, so their absence on a +given FlowFile is not itself an error. **The permissible input/read +attribute set is the same as the permissible write/output attribute set** +— one list governs both directions of external S3 interaction, not two +independently-defined ones (this resolves the write-side/read-side +reconciliation concern raised earlier — see +[Open Items](#open-items-for-implementation-planning) for what's still +needed to finalize it). Which of the permissible attributes actually get +packed and signed at a given egress point is a further, configurable +selection — see +[Egress Packing & Signing Selection](#egress-packing--signing-selection) +below — so "the attribute list" referenced elsewhere in this doc means +this controlled set (or a configured subset of it), not the full/arbitrary +FlowFile attribute map. The concrete list is **not yet provided** — only +the placeholder names/regex above exist today. + +SA also uses a separate set of **controlled internal attributes** +(`SA-V3-Int-##`) for its own routing and processing needs, and — new — +for **internal S3 usage**: the reassembly/staging space (per-chunk +`staging//` objects) and the replay buffer +(the Replay DB's S3 tier for large files) are **internal S3**, distinct +from the **external S3** interaction governed by the handover spec above. +Internal S3 objects may carry this additional permissible-internal- +attribute set — itself also not yet provided, count TBD (`NN`). Internal +attributes are never part of the externally-defined contract and are never +packed or signed for delivery — but they **are** allowed to cross the MR +hop: MR bridges the two NiFi chains *within* SA, not SA's external +boundary, so internal attributes travel with the FlowFile through MR like +any other attribute. They're stripped only at SA's true external egress +(final delivery to S3/HTTP/filesystem, outside SA) — internal S3 (staging, +replay buffer) is itself inside that boundary, not the egress point. + +Two boundary filters enforce this split, both implemented as a +delete-attributes-by-regex step (a standard NiFi capability, e.g. an +`UpdateAttribute` "Delete Attributes Expression"): +- **On import (ingest):** attributes **not** matching the external + allowlist regex (`^SA-V3-Ext-(0[1-9]|1[0-5])$`) are deleted. + Uncontrolled/unexpected external input never enters SA's attribute + space. +- **On external egress (final delivery — not the MR hop):** the + **inverse** of the external allowlist regex is applied to delete + everything that does *not* match `^SA-V3-Ext-(0[1-9]|1[0-5])$`. This is + a single step that removes internal (`SA-V3-Int-##`) attributes and any + other stray/uncontrolled attributes at once — it doesn't depend on + separately enumerating what "internal" means, only on what counts as a + valid external attribute. Only controlled external attributes reach S3/ + HTTP/filesystem destinations. + +### Egress Packing & Signing Selection + +Each egress processor exposes a **Regex property** naming which of the +controlled external (`SA-V3-Ext-##`) attributes get packed and signed at +that point in the flow. This is deliberately per-processor rather than one +fixed set for every egress point, so a flow author can configure a +different subset at each position in the flow depending on what that +position needs to expose. + +For S3 destinations specifically: +- Attributes matched by the regex whose combined `key=value` size is + **under 2KB when URL-encoded** are written as S3 user metadata + (`x-amz-meta-*`) — also selected via the same Regex property. +- Attributes that don't fit under that 2KB threshold are instead appended + in JSON form per [Packaged Artifact Format](#packaged-artifact-format), + matching the existing >2KB fallback path described in + [Storage Backend Handover](#storage-backend-handover). + +This regex-driven selection governs the **write** side — which attributes +this design packs or promotes to metadata. It's a distinct concern from +the **read**-side regex in the existing S3 handover interface spec (which +maps `x-amz-meta-*` keys back to NiFi attributes on read) — the two need +to be reconciled so a value written via this design's regex is correctly +recognized on read by that existing spec. Not yet confirmed; see +[Open Items](#open-items-for-implementation-planning). + +**Signing at ingest is optional**, not mandatory — a configurable property +on the ingest processor (see also [Signing](#signing)). Egress-time +packing/signing via the Regex property above is what actually determines +the contents of the final signed, packed artifact; ingest-time signing, +when enabled, is an earlier integrity checkpoint within SA, not a +substitute for it. + ## Packaged Artifact Format This is the format used for filesystem/XFS delivery always, and for S3 @@ -220,9 +333,12 @@ multipart upload) can't end up in that split state. | agnostic; works identically on S3, filesystem, and HTTP | +---------------------------------------------------------+ - Optional S3 object tag (attr-size = trailer byte length) as an - accelerant: lets the catalog DB or a reconciliation scan jump straight - to the trailer's byte range without reading the footer first. + S3 object tag/attribute (attr-size = trailer byte length), written + whenever the JSON-fallback flag is set (see Storage Backend Handover -> + S3 path): lets a reader jump straight to the trailer's byte range via a + suffix-range GET without reading the footer first. On S3 this is the + primary mechanism; the footer remains the storage-agnostic fallback for + filesystem/HTTP, which have no equivalent tag/metadata mechanism. ``` Manifest is written **after** content by construction — the ingest pass is @@ -244,6 +360,17 @@ content-then-trailer-then-footer shape regardless of size. ## Signing +- **Where signing is triggered:** ingest-time signing is optional + (configurable per ingest processor); egress-time signing, of the + attribute subset selected by the egress Regex property, is what actually + produces the signed content shipped in the final artifact — see + [Egress Packing & Signing Selection](#egress-packing--signing-selection). +- **Ingest-time validation:** incoming data at ingest may or may not + already carry a signature (e.g. from an earlier stage, or on replay). + Absence is fine — ingest signing is optional, per above. But if a + signature **is** already present on ingest, it must be validated at that + point rather than passed through unchecked; see + [Error Handling](#error-handling) for the failure case. - **Format:** OpenPGP detached-style trailer via `gpg --clearsign` semantics — clear-sign the **canonical ordered attribute list** (sorted by attribute key, including the content hash and any chunk descriptors), @@ -306,21 +433,43 @@ The JSON-trailer packaged artifact format is shared by both backends, but S3 additionally supports an existing metadata-based mechanism for small attribute sets that avoids the trailer entirely — see below. +This section (S3 path) covers **external S3** — SA's true delivery +boundary. It's distinct from **internal S3** — the reassembly/staging +space and replay buffer used *within* SA — which is governed by its own +additional permissible-internal-attribute set instead of the handover spec +below; see [Attribute Governance](#attribute-governance). + ### S3 path -**Attribute handover on S3 follows an existing, established handover -interface specification already used elsewhere in this environment — this -design integrates with it rather than replacing it.** Under that spec: +**Attribute handover on external S3 follows an existing, established +handover interface specification already used elsewhere in this +environment — this design integrates with it rather than replacing it.** +That spec defines the ~15 controlled external attributes (`SA-V3-Ext-##` +placeholder), and the **same list governs both directions** — what's +permissible to read is permissible to write, so there's one shared +attribute contract for external S3, not independently-evolved read and +write lists. Under that spec: - Attributes are written as S3 user object metadata (`x-amz-meta-*` headers) when small enough to fit. A regular expression (defined by that existing spec, not this one) identifies which user metadata keys get turned into NiFi attributes on read — so consuming the attributes is a single `HeadObject` call, no content touch, not even a suffix-range read. + Which attributes from that permissible list actually get *written* as + metadata at a given egress point is a separate, this-design-owned + selection (a configured subset, not necessarily all permissible ones) — + see [Egress Packing & Signing Selection](#egress-packing--signing-selection). - A flag (also part of that existing spec) records whether the attribute set exceeded S3's ~2KB user-metadata limit. When it does, the object falls back to **this design's** JSON manifest trailer (Packaged Artifact Format) instead of metadata. +- Whenever that fallback flag is set, **this design** also writes a second + attribute giving the byte size of the appended JSON manifest trailer. + Together, the two let a reader do a single suffix-range `GET` — + `Range: bytes=-N` with `N` from that size attribute — to fetch exactly + the appended JSON and parse it directly into attributes, with no need to + read the fixed footer first (see + [Packaged Artifact Format](#packaged-artifact-format)). - The exact regex pattern and flag key/tag name live in the existing handover interface spec document, not reproduced here — see [Open Items](#open-items-for-implementation-planning). @@ -397,6 +546,8 @@ FlowFiles via the same ingest path used for first-time delivery. | Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. | | Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists | | Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch | +| Data arrives at ingest with no existing signature | Not an error — ingest-time signing is optional (see Signing); proceeds normally | +| Data arrives at ingest already signed, signature fails validation | Route to failure — an already-signed FlowFile at ingest is validated on arrival, not passed through unchecked | | Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers | ## Testing @@ -449,3 +600,24 @@ environment. keys to NiFi attributes, and the exact flag/tag name and value convention used to signal the >2KB fallback to the JSON trailer. This design assumes that spec's behavior but doesn't reproduce its details. +- **Enumerate the controlled attribute lists**, to be provided later per + [Attribute Governance](#attribute-governance): + - The ~15 controlled external attributes (`SA-V3-Ext-##` placeholders), + owned by the external S3 handover interface spec — names, types, and + which subset of the ~15 is mandatory vs. optional. + - The separate controlled internal attribute list (`SA-V3-Int-##` + placeholders, count `NN` not yet fixed) — used for SA routing/ + processing generally, and specifically for internal S3 (staging/ + replay buffer) objects. +- **Confirm the write-side/read-side regex reconciliation is fully closed + in practice, not just in principle.** It's now established that the + permissible external attribute set is the same for reads and writes (one + list, not two independently-defined ones) — but this still needs the + actual document to confirm: exact key-casing convention (S3 lowercases + `x-amz-meta-*` header names on read), and that the flag/size attributes + this design writes for the >2KB fallback don't collide with any of the + ~15 external attribute keys. +- **Define how internal attributes are represented on internal S3 + objects** (staging, replay buffer) — S3 user metadata vs. tags vs. the + JSON-trailer mechanism, and whether the same 2KB threshold applies + there.