Ubuntu
0bae727a26
Defines the SA-V3-Ext/Int placeholder attribute scheme (external list owned by the S3 handover spec, internal list for SA routing + internal S3 usage), the per-egress Regex property for packing/signing selection, the inverse-regex delete step at true external egress, optional ingest-time signing with mandatory validation when already signed, and the S3 JSON-fallback size attribute for suffix-range trailer reads.
2.6 KiB
2.6 KiB
Packaging & Signing — Key Decisions
Full detail lives in
docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md.
These are the non-obvious calls worth remembering, several of which
reversed an earlier direction during design review:
- No sidecar fronting ingestion. A claim-check sidecar to bypass NiFi's content repo for large payloads was considered and rejected — all content must transit NiFi for routing/content-assurance controls. The external service's role is post-NiFi only: assembly, stall monitoring, cleanup, catalog DB maintenance.
- No external signing service. Vault Transit / a network-isolated signer was considered and rejected in favor of in-process signing (Bouncy Castle OpenPGP, key loaded via a Controller Service) to keep the architecture self-contained within NiFi. This is a deliberate non-repudiation trade-off, stated explicitly in the spec — key custody now depends on host-level protection (sensitive-property encryption, restricting which instances have the Controller Service) rather than network isolation.
- Clear-sign (OpenPGP), not a bespoke signature format. Chosen
specifically because an operator on the far side of MR can verify an
artifact by hand with stock
gpg --verifyand a public key — no custom SDK needed. Matters because MR already has a manual-operator recovery model for missing chunks (adopted from the Phat Files design). - HMAC was considered and rejected as the integrity mechanism — it detects tampering but can't give non-repudiation, since every verifier necessarily holds the same shared secret and could in principle have forged the tag themselves.
- Content + JSON trailer as one object, not two. Chosen for end-to-end loss protection: two separate objects (content, manifest) have independent failure/lifecycle/replication paths that can diverge. A single completed write can't end up in a split state.
- S3 attribute handover defers to an existing, established handover interface spec, not this design's own invention — see open-external-dependencies.
- S3-vs-XFS storage backend split by file size is unresolved — open question whether S3 can serve small files efficiently enough to drop the XFS tier entirely.
- Attribute list is a controlled allowlist, not "all FlowFile attributes." ~15 controlled external attributes carry through ingest-to-delivery (these are what get signed/packed); a separate controlled internal set is SA-only and stripped before export. Import drops uncontrolled external attributes; export drops internal ones. See attribute-governance.