Add attribute governance design: controlled attribute lists, egress regex selection, ingest signature validation
Defines the SA-V3-Ext/Int placeholder attribute scheme (external list owned by the S3 handover spec, internal list for SA routing + internal S3 usage), the per-egress Regex property for packing/signing selection, the inverse-regex delete step at true external egress, optional ingest-time signing with mandatory validation when already signed, and the S3 JSON-fallback size attribute for suffix-range trailer reads.
This commit is contained in:
Ubuntu
@@ -138,7 +138,9 @@ true entry into SA — consistent with the single-pass design below.
|
||||
is just the one-chunk case.
|
||||
|
|
||||
v
|
||||
Ordered attribute list (incl. content hash, chunk
|
||||
Optionally (ingest-time signing is a configurable
|
||||
property, see Attribute Governance): ordered
|
||||
attribute list (incl. content hash, chunk
|
||||
descriptors) is clear-signed in-process (Bouncy
|
||||
Castle OpenPGP, key via Controller Service — see
|
||||
Signing) and stored as a new attribute.
|
||||
@@ -186,6 +188,117 @@ the main justification for a front door outside the JVM. The external
|
||||
service's role is narrowed to post-NiFi work that never needs to see raw
|
||||
content: assembly, monitoring, cleanup, and catalog indexing.
|
||||
|
||||
## Attribute Governance
|
||||
|
||||
**Naming convention (placeholder):** until the concrete attribute list is
|
||||
enumerated (see [Open Items](#open-items-for-implementation-planning)),
|
||||
controlled attributes are referred to by placeholder names — external
|
||||
attributes as `SA-V3-Ext-01` … `SA-V3-Ext-15`, internal attributes as
|
||||
`SA-V3-Int-01` … `SA-V3-Int-NN`. These stand in for the real names
|
||||
throughout this doc and in code until finalized.
|
||||
|
||||
**Placeholder regexes**, used for matching against these placeholder
|
||||
names until the real attribute list replaces them:
|
||||
- External allowlist: `^SA-V3-Ext-(0[1-9]|1[0-5])$` (matches
|
||||
`SA-V3-Ext-01` through `SA-V3-Ext-15`).
|
||||
- Internal: `^SA-V3-Int-\d{2}$` (count not yet fixed — placeholder covers
|
||||
any two-digit index until `NN` is finalized).
|
||||
|
||||
The content hash algorithm is not fixed by this design. It is exposed as a
|
||||
**dynamic property** on the custom ingest processor (and mirrored on the
|
||||
corresponding egress/delivery processor), so it can be changed per
|
||||
deployment without a code change. Whichever algorithm is configured is
|
||||
recorded as one of SA's controlled attributes and carried with the
|
||||
FlowFile, so the algorithm used is self-describing at verification time
|
||||
rather than assumed out-of-band.
|
||||
|
||||
SA maintains a fixed set of roughly **15 controlled external attributes**
|
||||
(`SA-V3-Ext-##`) — attributes with an externally-defined meaning/contract,
|
||||
as opposed to SA-internal bookkeeping — that are carried through unchanged
|
||||
from ingest to final destination. This list is **owned by the existing S3
|
||||
handover interface spec** (see [Storage Backend Handover](#storage-backend-handover)),
|
||||
not authored by this design — this design consumes it. Only a **subset**
|
||||
of the ~15 are mandatory; the rest are optional, so their absence on a
|
||||
given FlowFile is not itself an error. **The permissible input/read
|
||||
attribute set is the same as the permissible write/output attribute set**
|
||||
— one list governs both directions of external S3 interaction, not two
|
||||
independently-defined ones (this resolves the write-side/read-side
|
||||
reconciliation concern raised earlier — see
|
||||
[Open Items](#open-items-for-implementation-planning) for what's still
|
||||
needed to finalize it). Which of the permissible attributes actually get
|
||||
packed and signed at a given egress point is a further, configurable
|
||||
selection — see
|
||||
[Egress Packing & Signing Selection](#egress-packing--signing-selection)
|
||||
below — so "the attribute list" referenced elsewhere in this doc means
|
||||
this controlled set (or a configured subset of it), not the full/arbitrary
|
||||
FlowFile attribute map. The concrete list is **not yet provided** — only
|
||||
the placeholder names/regex above exist today.
|
||||
|
||||
SA also uses a separate set of **controlled internal attributes**
|
||||
(`SA-V3-Int-##`) for its own routing and processing needs, and — new —
|
||||
for **internal S3 usage**: the reassembly/staging space (per-chunk
|
||||
`staging/<generation-id>/<chunk-index>` objects) and the replay buffer
|
||||
(the Replay DB's S3 tier for large files) are **internal S3**, distinct
|
||||
from the **external S3** interaction governed by the handover spec above.
|
||||
Internal S3 objects may carry this additional permissible-internal-
|
||||
attribute set — itself also not yet provided, count TBD (`NN`). Internal
|
||||
attributes are never part of the externally-defined contract and are never
|
||||
packed or signed for delivery — but they **are** allowed to cross the MR
|
||||
hop: MR bridges the two NiFi chains *within* SA, not SA's external
|
||||
boundary, so internal attributes travel with the FlowFile through MR like
|
||||
any other attribute. They're stripped only at SA's true external egress
|
||||
(final delivery to S3/HTTP/filesystem, outside SA) — internal S3 (staging,
|
||||
replay buffer) is itself inside that boundary, not the egress point.
|
||||
|
||||
Two boundary filters enforce this split, both implemented as a
|
||||
delete-attributes-by-regex step (a standard NiFi capability, e.g. an
|
||||
`UpdateAttribute` "Delete Attributes Expression"):
|
||||
- **On import (ingest):** attributes **not** matching the external
|
||||
allowlist regex (`^SA-V3-Ext-(0[1-9]|1[0-5])$`) are deleted.
|
||||
Uncontrolled/unexpected external input never enters SA's attribute
|
||||
space.
|
||||
- **On external egress (final delivery — not the MR hop):** the
|
||||
**inverse** of the external allowlist regex is applied to delete
|
||||
everything that does *not* match `^SA-V3-Ext-(0[1-9]|1[0-5])$`. This is
|
||||
a single step that removes internal (`SA-V3-Int-##`) attributes and any
|
||||
other stray/uncontrolled attributes at once — it doesn't depend on
|
||||
separately enumerating what "internal" means, only on what counts as a
|
||||
valid external attribute. Only controlled external attributes reach S3/
|
||||
HTTP/filesystem destinations.
|
||||
|
||||
### Egress Packing & Signing Selection
|
||||
|
||||
Each egress processor exposes a **Regex property** naming which of the
|
||||
controlled external (`SA-V3-Ext-##`) attributes get packed and signed at
|
||||
that point in the flow. This is deliberately per-processor rather than one
|
||||
fixed set for every egress point, so a flow author can configure a
|
||||
different subset at each position in the flow depending on what that
|
||||
position needs to expose.
|
||||
|
||||
For S3 destinations specifically:
|
||||
- Attributes matched by the regex whose combined `key=value` size is
|
||||
**under 2KB when URL-encoded** are written as S3 user metadata
|
||||
(`x-amz-meta-*`) — also selected via the same Regex property.
|
||||
- Attributes that don't fit under that 2KB threshold are instead appended
|
||||
in JSON form per [Packaged Artifact Format](#packaged-artifact-format),
|
||||
matching the existing >2KB fallback path described in
|
||||
[Storage Backend Handover](#storage-backend-handover).
|
||||
|
||||
This regex-driven selection governs the **write** side — which attributes
|
||||
this design packs or promotes to metadata. It's a distinct concern from
|
||||
the **read**-side regex in the existing S3 handover interface spec (which
|
||||
maps `x-amz-meta-*` keys back to NiFi attributes on read) — the two need
|
||||
to be reconciled so a value written via this design's regex is correctly
|
||||
recognized on read by that existing spec. Not yet confirmed; see
|
||||
[Open Items](#open-items-for-implementation-planning).
|
||||
|
||||
**Signing at ingest is optional**, not mandatory — a configurable property
|
||||
on the ingest processor (see also [Signing](#signing)). Egress-time
|
||||
packing/signing via the Regex property above is what actually determines
|
||||
the contents of the final signed, packed artifact; ingest-time signing,
|
||||
when enabled, is an earlier integrity checkpoint within SA, not a
|
||||
substitute for it.
|
||||
|
||||
## Packaged Artifact Format
|
||||
|
||||
This is the format used for filesystem/XFS delivery always, and for S3
|
||||
@@ -220,9 +333,12 @@ multipart upload) can't end up in that split state.
|
||||
| agnostic; works identically on S3, filesystem, and HTTP |
|
||||
+---------------------------------------------------------+
|
||||
|
||||
Optional S3 object tag (attr-size = trailer byte length) as an
|
||||
accelerant: lets the catalog DB or a reconciliation scan jump straight
|
||||
to the trailer's byte range without reading the footer first.
|
||||
S3 object tag/attribute (attr-size = trailer byte length), written
|
||||
whenever the JSON-fallback flag is set (see Storage Backend Handover ->
|
||||
S3 path): lets a reader jump straight to the trailer's byte range via a
|
||||
suffix-range GET without reading the footer first. On S3 this is the
|
||||
primary mechanism; the footer remains the storage-agnostic fallback for
|
||||
filesystem/HTTP, which have no equivalent tag/metadata mechanism.
|
||||
```
|
||||
|
||||
Manifest is written **after** content by construction — the ingest pass is
|
||||
@@ -244,6 +360,17 @@ content-then-trailer-then-footer shape regardless of size.
|
||||
|
||||
## Signing
|
||||
|
||||
- **Where signing is triggered:** ingest-time signing is optional
|
||||
(configurable per ingest processor); egress-time signing, of the
|
||||
attribute subset selected by the egress Regex property, is what actually
|
||||
produces the signed content shipped in the final artifact — see
|
||||
[Egress Packing & Signing Selection](#egress-packing--signing-selection).
|
||||
- **Ingest-time validation:** incoming data at ingest may or may not
|
||||
already carry a signature (e.g. from an earlier stage, or on replay).
|
||||
Absence is fine — ingest signing is optional, per above. But if a
|
||||
signature **is** already present on ingest, it must be validated at that
|
||||
point rather than passed through unchecked; see
|
||||
[Error Handling](#error-handling) for the failure case.
|
||||
- **Format:** OpenPGP detached-style trailer via `gpg --clearsign`
|
||||
semantics — clear-sign the **canonical ordered attribute list** (sorted
|
||||
by attribute key, including the content hash and any chunk descriptors),
|
||||
@@ -306,21 +433,43 @@ The JSON-trailer packaged artifact format is shared by both backends, but
|
||||
S3 additionally supports an existing metadata-based mechanism for small
|
||||
attribute sets that avoids the trailer entirely — see below.
|
||||
|
||||
This section (S3 path) covers **external S3** — SA's true delivery
|
||||
boundary. It's distinct from **internal S3** — the reassembly/staging
|
||||
space and replay buffer used *within* SA — which is governed by its own
|
||||
additional permissible-internal-attribute set instead of the handover spec
|
||||
below; see [Attribute Governance](#attribute-governance).
|
||||
|
||||
### S3 path
|
||||
|
||||
**Attribute handover on S3 follows an existing, established handover
|
||||
interface specification already used elsewhere in this environment — this
|
||||
design integrates with it rather than replacing it.** Under that spec:
|
||||
**Attribute handover on external S3 follows an existing, established
|
||||
handover interface specification already used elsewhere in this
|
||||
environment — this design integrates with it rather than replacing it.**
|
||||
That spec defines the ~15 controlled external attributes (`SA-V3-Ext-##`
|
||||
placeholder), and the **same list governs both directions** — what's
|
||||
permissible to read is permissible to write, so there's one shared
|
||||
attribute contract for external S3, not independently-evolved read and
|
||||
write lists. Under that spec:
|
||||
|
||||
- Attributes are written as S3 user object metadata (`x-amz-meta-*`
|
||||
headers) when small enough to fit. A regular expression (defined by that
|
||||
existing spec, not this one) identifies which user metadata keys get
|
||||
turned into NiFi attributes on read — so consuming the attributes is a
|
||||
single `HeadObject` call, no content touch, not even a suffix-range read.
|
||||
Which attributes from that permissible list actually get *written* as
|
||||
metadata at a given egress point is a separate, this-design-owned
|
||||
selection (a configured subset, not necessarily all permissible ones) —
|
||||
see [Egress Packing & Signing Selection](#egress-packing--signing-selection).
|
||||
- A flag (also part of that existing spec) records whether the attribute
|
||||
set exceeded S3's ~2KB user-metadata limit. When it does, the object
|
||||
falls back to **this design's** JSON manifest trailer (Packaged Artifact
|
||||
Format) instead of metadata.
|
||||
- Whenever that fallback flag is set, **this design** also writes a second
|
||||
attribute giving the byte size of the appended JSON manifest trailer.
|
||||
Together, the two let a reader do a single suffix-range `GET` —
|
||||
`Range: bytes=-N` with `N` from that size attribute — to fetch exactly
|
||||
the appended JSON and parse it directly into attributes, with no need to
|
||||
read the fixed footer first (see
|
||||
[Packaged Artifact Format](#packaged-artifact-format)).
|
||||
- The exact regex pattern and flag key/tag name live in the existing
|
||||
handover interface spec document, not reproduced here — see
|
||||
[Open Items](#open-items-for-implementation-planning).
|
||||
@@ -397,6 +546,8 @@ FlowFiles via the same ingest path used for first-time delivery.
|
||||
| Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. |
|
||||
| Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists |
|
||||
| Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch |
|
||||
| Data arrives at ingest with no existing signature | Not an error — ingest-time signing is optional (see Signing); proceeds normally |
|
||||
| Data arrives at ingest already signed, signature fails validation | Route to failure — an already-signed FlowFile at ingest is validated on arrival, not passed through unchecked |
|
||||
| Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers |
|
||||
|
||||
## Testing
|
||||
@@ -449,3 +600,24 @@ environment.
|
||||
keys to NiFi attributes, and the exact flag/tag name and value convention
|
||||
used to signal the >2KB fallback to the JSON trailer. This design assumes
|
||||
that spec's behavior but doesn't reproduce its details.
|
||||
- **Enumerate the controlled attribute lists**, to be provided later per
|
||||
[Attribute Governance](#attribute-governance):
|
||||
- The ~15 controlled external attributes (`SA-V3-Ext-##` placeholders),
|
||||
owned by the external S3 handover interface spec — names, types, and
|
||||
which subset of the ~15 is mandatory vs. optional.
|
||||
- The separate controlled internal attribute list (`SA-V3-Int-##`
|
||||
placeholders, count `NN` not yet fixed) — used for SA routing/
|
||||
processing generally, and specifically for internal S3 (staging/
|
||||
replay buffer) objects.
|
||||
- **Confirm the write-side/read-side regex reconciliation is fully closed
|
||||
in practice, not just in principle.** It's now established that the
|
||||
permissible external attribute set is the same for reads and writes (one
|
||||
list, not two independently-defined ones) — but this still needs the
|
||||
actual document to confirm: exact key-casing convention (S3 lowercases
|
||||
`x-amz-meta-*` header names on read), and that the flag/size attributes
|
||||
this design writes for the >2KB fallback don't collide with any of the
|
||||
~15 external attribute keys.
|
||||
- **Define how internal attributes are represented on internal S3
|
||||
objects** (staging, replay buffer) — S3 user metadata vs. tags vs. the
|
||||
JSON-trailer mechanism, and whether the same 2KB threshold applies
|
||||
there.
|
||||
|
||||
Reference in New Issue
Block a user