# FlowFile Packaging & Signing — Design Spec **Date:** 2026-07-28 **Status:** Draft — pending user review ## Purpose NiFi is used to manage complex workflows with payloads ranging from tiny files to TB scale. Two related problems need solving: 1. **Performance.** The current pack/unpack path (`MergeContent` + `UnpackContent`) is disk-I/O bound under high throughput. Content lives on vSAN storage, which is shared, virtualized IOPS across the vSphere cluster — so the pain isn't any single slow disk, it's *contention* across every VM sharing that storage. The current flow makes multiple redundant passes over content on disk: an ingest write, a separate read to compute a content hash, a read+write to merge/pack, and a read+write to unpack. Each pass competes for the same shared IOPS pool. NiFi runs as independent single-node instances (no clustering), each with its own logical function, scaled horizontally by adding more instances — so there's no cluster-wide load balancing to fall back on. 2. **Integrity & non-repudiation.** When a FlowFile (content + attributes) is packed and exported outside NiFi's space, its attributes — which already include a hash of the content, so content hashing is not part of packing — need to be signed to prove they haven't been tampered with and to prove who produced them, in a way that holds up even to a party who didn't produce the artifact. Uses for the packed form: storage/ retrieval to/from S3, delivery to other applications via HTTP or filesystem, and selective (date/category) replay from a catalog database. A **future** requirement extends this to TB-scale files that must be broken into pieces, tracked, transited through NiFi (for routing and content-assurance controls — all content must pass through NiFi), and reconstructed at the destination — including across a **one-way network boundary** — referred to throughout this doc as **MR**, the named relay bridging the two NiFi host chains within the overall system referred to as **SA** (see [Deployment Topology](#deployment-topology)) — with no return channel for retransmission requests. Forward error correction for that hop is assumed to be available via an existing HTTP service, with ~1GB as the target chunk size fed to it. ## Non-goals - Redesigning NiFi's clustering model. Instances remain independent, single-node, horizontally scaled by function. - Automated recovery signaling across the true one-way diode boundary — by construction there is no return channel; recovery there is necessarily operator-mediated (see [Error Handling](#error-handling)). - Building or operating the forward-error-correction transport itself — assumed to already exist as an HTTP service outside this design's scope. - A general-purpose NiFi improvement library. This design covers exactly the packaging, chunking, signing, delivery, and replay-indexing concerns described above. ## Relationship to the "Phat Files" design A separate, already-approved spec (`2026-07-23-phat-files-design.md`) covers low-side-S3-to-high-side-S3 transfer of TB-scale files across a one-way boundary: fixed-size blocks, resend-vs-duplicate handling, staging by block index, server-side assembly via `UploadPartCopy`, stall monitoring, and cleanup. That spec is treated here as a **source of proven concepts, not a fixed dependency** — this design is not an extension of it and does not modify it. The following ideas are adopted (generalized beyond S3-to-S3): - Streaming byte-range reads; never buffer a whole file. - Chunk descriptor attributes (index, total, size, checksum). - Server-side assembly via `UploadPartCopy` from independently-staged per-chunk objects, rather than a live multipart-upload session spanning the transport. - Explicit resend-vs-duplicate distinction via a flag, not inferred from bytes. - Stall detection + missing-chunk report, for hops with no return channel. - Testing convention: pure logic extracted to statically-testable methods. What's **changed**: packaging moves from stock FlowFile Package V3 to the custom manifest-trailer format below (needed for signing and attribute-independent reads); staging/monitoring/cleanup responsibilities move to an external service rather than NiFi timer-driven processors; completion tracking is dual-written to a durable marker *and* a catalog database instead of an S3-only marker object. ## Architecture ### Deployment Topology The whole system described in this doc is referred to as **SA**: a single system containing two chains of independent, single-function NiFi hosts linked by Site-to-Site (per the existing horizontal-scaling model — no clustering, just chained instances), bridged in the middle by **MR** — the named one-way relay this design has been built around (FEC via an HTTP service, ~1GB chunks, per [Purpose](#purpose)). Content flows in one direction through SA: external S3/S2S in, through the first chain, across MR, through the second chain, external S3/S2S out. The Replay DB branches off **before** MR, from node 1 of the *first* chain — i.e. at the true ingest point into SA, not after delivery. ``` +------------------------------------ SA -------------------------------------+ | | | S3 --\ /-- S3 | | S2S --+-->[1]--S2S-->[2]--S2S-->[3]--HTTP-->[MR]--HTTP-->[1]--S2S-->[2]--S2S-->[3]--+-- S2S | | | | | "copy" | | v | | +-----------+ | | | Replay DB | | | +-----------+ | | / \ | | v v | | S3 (large files) XFS (small files) | +------------------------------------------------------------------------------------+ ``` Chunking is applied uniformly at ingest (below) specifically so the same chunked representation transits **both** the intra-chain S2S hops **and** the MR relay — there's no separate chunking scheme per transport. Hashing happens at this same ingest point — node 1 of the first chain, the true entry into SA — consistent with the single-pass design below. ### Per-FlowFile Pipeline ``` INGEST (custom NiFi processor; replaces ListenHTTP for HTTP, or first-in-flow processor right after the S2S Input Port) | v Single streaming pass: write content to NiFi's content repo (required — all content must transit NiFi for routing/content-assurance) WHILE computing the content hash and emitting a chunk boundary every ~1GB. No pre-ingest size check or path branching — chunk count falls out of content length; a small file is just the one-chunk case. | v Ordered attribute list (incl. content hash, chunk descriptors) is clear-signed in-process (Bouncy Castle OpenPGP, key via Controller Service — see Signing) and stored as a new attribute. | v FlowFile(s) proceed through normal NiFi routing / content-assurance flow logic. | v DELIVERY: per-chunk content streamed once (single content-repo read) to destination staging. | v +-----------------------------------------------+ | DESTINATIONS | | S3: stage per-chunk objects, assemble via | | UploadPartCopy (no re-transfer) | | filesystem: pwrite chunks at offset, preallocated| | HTTP: deliver assembled object to receiving app | | diode: chunks -> FEC HTTP service (~1GB each) | +-----------------------------------------------+ | | write marker upsert row (durable, (fast index, per-dest) rebuildable | from markers) v v +-----------------------------------------------+ | EXTERNAL SERVICE (post-NiFi only): | | assembly (UploadPartCopy), stall monitoring, | | cleanup, catalog DB maintenance | +-----------------------------------------------+ | v +-----------------------------------------------+ | CATALOG / REPLAY DATABASE | +-----------------------------------------------+ ``` A prior iteration of this design considered a sidecar process fronting ingestion (to bypass NiFi's content repository entirely for large payloads, via a claim-check pattern). That's rejected: **all content must go through NiFi** for routing and content-assurance controls, which removes the main justification for a front door outside the JVM. The external service's role is narrowed to post-NiFi work that never needs to see raw content: assembly, monitoring, cleanup, and catalog indexing. ## Packaged Artifact Format This is the format used for filesystem/XFS delivery always, and for S3 delivery when the attribute set is too large for S3 user object metadata (see [Storage Backend Handover](#storage-backend-handover)). Content and its manifest travel as **one object**, not two, for end-to-end loss protection: two separate objects (content + manifest) have independent failure/lifecycle/replication paths that can diverge — e.g. content lands but the manifest write fails, or a lifecycle rule reaps one and not the other. A single completed write (one `PutObject`, or one completed multipart upload) can't end up in that split state. ``` PACKAGED OBJECT (single PutObject, or single completed multipart upload) +---------------------------------------------------------+ | CONTENT (1 object, or N ~1GB parts via multipart) | +---------------------------------------------------------+ | MANIFEST TRAILER (final part -- direct UploadPart, not | | UploadPartCopy, so it can be arbitrarily small) | | = JSON document, e.g.: | | { "format_version": ..., | | "clearsign": "-----BEGIN PGP SIGNED MESSAGE----- | | ...ordered attribute list... | | -----BEGIN PGP SIGNATURE-----... | | -----END PGP SIGNATURE-----" } | | The clearsign field is self-contained and human- | | verifiable via `gpg --verify` independent of the JSON | | wrapper around it. | +---------------------------------------------------------+ | FOOTER (fixed size, e.g. last 12 bytes: magic + 8-byte | | big-endian trailer length) -- self-describing, storage- | | agnostic; works identically on S3, filesystem, and HTTP | +---------------------------------------------------------+ Optional S3 object tag (attr-size = trailer byte length) as an accelerant: lets the catalog DB or a reconciliation scan jump straight to the trailer's byte range without reading the footer first. ``` Manifest is written **after** content by construction — the ingest pass is single-streaming, so the final content hash isn't known until content is fully read, ruling out a header-first format (which would require either buffering to know its size upfront, or a second pass to fix it up). **Attribute-only reads** (replay filtering by date/category, signature verification, routing) use a suffix-range read — `Range: bytes=-N` where `N` is the trailer size (from the catalog DB, the S3 tag, or the footer) — which never touches the (possibly TB-scale) content bytes and completes in one round trip when the DB already knows the offset. For chunked files, S3 multipart uploads mix `UploadPartCopy` parts (the staged content chunks) with one ordinary `UploadPart` (the manifest trailer — no 5MB minimum applies to the last part), so ` CompleteMultipartUpload` produces one final object with the same content-then-trailer-then-footer shape regardless of size. ## Signing - **Format:** OpenPGP detached-style trailer via `gpg --clearsign` semantics — clear-sign the **canonical ordered attribute list** (sorted by attribute key, including the content hash and any chunk descriptors), producing a self-contained ASCII-armored block (plaintext + embedded signature). The result is stored as a new FlowFile attribute, then becomes the `clearsign` field of the JSON manifest trailer at pack time (see [Packaged Artifact Format](#packaged-artifact-format)) — the JSON wrapper carries packaging metadata (format version, chunk descriptors) alongside the signed block, without altering the signed bytes themselves. - **Why clear-sign / OpenPGP over a bespoke signature scheme:** an operator on the far side of the one-way diode boundary can verify an artifact by hand with stock GnuPG and a public key — no custom SDK needed. This matters because that boundary already has a manual-operator recovery model (missing-chunk reports carried by hand, per Phat Files); OpenPGP fits that operational reality directly. - **Where signing runs:** in-process, via Bouncy Castle's OpenPGP API (not shelling out to the `gpg` CLI per FlowFile — too slow at throughput). No external signing service — this was evaluated and explicitly rejected in favor of keeping the architecture self-contained within NiFi. The private key is loaded via a Controller Service (same pattern as the existing `AwsCredentialsProviderService`), plain in-JVM key material protected by NiFi's sensitive-property encryption and OS-level file permissions. - **Non-repudiation trade-off, stated explicitly:** non-repudiation depends on the private key being exclusively controlled by one identity. Moving signing in-process (vs. a network-isolated external service) means the key now lives on the same host as ingestion, so custody depends on host-level protections rather than network isolation. Mitigations: only the NiFi instances actually responsible for packing/export have the signing Controller Service configured (not every instance in the deployment); NiFi's provenance repository gives an audit trail of which flow/processor invoked signing and when, partially substituting for the audit log a dedicated signing service would give for free. This is not a one-way door — the Controller Service abstraction means a PKCS11/HSM token could be swapped in later as a stronger key-custody option without changing processor logic, if ever required. - **Algorithm:** Ed25519 preferred for speed and small signature size at throughput, unless a compliance requirement (e.g. FIPS mode) forces RSA/ECDSA instead — not yet confirmed as a constraint. - **Tamper-detection vs. non-repudiation:** a symmetric HMAC alternative was considered and rejected. HMAC would detect unauthorized modification (any party without the shared secret can't forge a valid tag) but cannot prove authorship to a third party, since every verifier necessarily holds the same secret and could in principle have produced it themselves. The stated requirement is non-repudiation, which needs an asymmetric scheme. ## Storage Backend Handover Final storage backend is chosen by size, separate from the ingest-time chunking decision (which is uniform, no threshold — see Architecture): large files land in S3, small files land on a local/shared XFS filesystem. This split exists because S3's per-request overhead and cost make it a poor fit for very high volumes of tiny objects; XFS is cheaper and faster for that case. **Open question, not yet resolved:** if S3 can be shown to serve small files efficiently enough (e.g. via batching, a different bucket/storage class, or acceptable cost at the actual small-file volume expected), the XFS tier could be dropped and S3 used uniformly — this needs validating against real small-file throughput numbers before implementation, see [Open Items](#open-items-for-implementation-planning). The JSON-trailer packaged artifact format is shared by both backends, but S3 additionally supports an existing metadata-based mechanism for small attribute sets that avoids the trailer entirely — see below. ### S3 path **Attribute handover on S3 follows an existing, established handover interface specification already used elsewhere in this environment — this design integrates with it rather than replacing it.** Under that spec: - Attributes are written as S3 user object metadata (`x-amz-meta-*` headers) when small enough to fit. A regular expression (defined by that existing spec, not this one) identifies which user metadata keys get turned into NiFi attributes on read — so consuming the attributes is a single `HeadObject` call, no content touch, not even a suffix-range read. - A flag (also part of that existing spec) records whether the attribute set exceeded S3's ~2KB user-metadata limit. When it does, the object falls back to **this design's** JSON manifest trailer (Packaged Artifact Format) instead of metadata. - The exact regex pattern and flag key/tag name live in the existing handover interface spec document, not reproduced here — see [Open Items](#open-items-for-implementation-planning). So for S3 specifically, the JSON trailer is the **fallback path** (oversized attribute sets), not the only mechanism — small-attribute objects never get a trailer at all. Filesystem/XFS delivery, which has no equivalent native metadata mechanism, always uses the JSON trailer. ``` NiFi (per chunk FlowFile, content already in repo from ingest) | | PutObject (single read of the content-repo claim -- v the only disk touch on the delivery side) S3 staging: staging// | | (all N chunks staged; external service detects v completeness via catalog DB or S3 listing) UploadPartCopy x N -- server-side, zero bytes touch NiFi or the | external service v Attributes fit in 2KB? yes -> PutObjectMetadata / CompleteMultipartUpload with attributes as x-amz-meta-* headers (no trailer) no -> + UploadPart (JSON manifest trailer), oversized flag set v Final object at | v write durable completion marker + upsert catalog DB row | v delete staging//* (cleanup) ``` ### XFS path (small files, below the size threshold) No staging/assembly needed — small files are single-chunk by construction (Architecture), so NiFi writes the packaged object (content + JSON trailer + footer) directly to its final path in one `write()` call. The external service still writes the durable completion marker (a sibling file, or an xattr, depending on implementation preference) and upserts the catalog DB row, same as the S3 path, so replay and reconciliation behave identically regardless of backend. Chunk upload failures for **plain S3/filesystem/HTTP delivery** are ordinary NiFi FlowFile retry — Phat Files' resend/stall-report machinery is reserved for the actual one-way diode hop, where there's no return channel; applying it to destinations that do have bidirectional connectivity would be unnecessary complexity. ## Catalog / Replay Database One row per packed artifact, written alongside the durable completion marker (same write-marker-then-upsert-row ordering as Section "Architecture"). Indexed for date/category replay queries: which backend (S3 or XFS), object location (S3 key or filesystem path), the clear-signed attribute list (so filtering never touches the object), completion status, and generation/chunk-set id for large files. The database is explicitly a **rebuildable index, not a source of truth** — the durable marker plus the trailer/footer on disk or S3 is authoritative, and a reconciliation job can rebuild the catalog by scanning objects and reading trailers if the database is ever lost, corrupted, or migrated. Replay is: query by date/category → get object locations → re-inject as FlowFiles via the same ingest path used for first-time delivery. ## Error Handling | Condition | Handling | |---|---| | Checksum mismatch on chunk arrival | Route to failure, never staged | | Resend vs. unrequested duplicate | Explicit flag on the FlowFile, checked against the durable completion marker (adopted from Phat Files) | | Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. | | Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists | | Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch | | Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers | ## Testing Following the convention established in Phat Files: pure logic extracted to package-private static methods, unit-testable without a live NiFi/S3 connection — - Canonical attribute-list ordering and serialization (must be deterministic for clear-sign to produce consistent output for the same attribute set). - Chunk-boundary computation from a content stream (byte offset math, ~1GB target size). - Footer encode/decode (magic + length) and suffix-range calculation. - Missing-chunk diffing (expected range vs. staged contents) — adopted from Phat Files. - Resend-vs-duplicate decision logic — adopted from Phat Files. End-to-end behavior (ingest → sign → chunk → stage → assemble → catalog row → attribute-only read via suffix range → replay) verified against the lab's MinIO instance, consistent with prior processors in this environment. ## Open Items for Implementation Planning - Exact NAR/module layout for the new ingest processor(s), Controller Service (signing key), and the external post-NiFi service. - Concrete canonical attribute-list serialization format (sorted `key: value` lines vs. canonical JSON) — functionally equivalent, pick one for implementation. - Confirm whether Ed25519 is acceptable or a compliance constraint (e.g. FIPS) forces RSA/ECDSA. - Concrete default values: chunk size (default ~1GB, confirm whether it should differ between diode-bound and non-diode-bound chunked delivery), stall timeout, cleanup retention window. - Catalog database technology selection (not yet specified). - Whether HTTP delivery to receiving applications needs the same suffix- range attribute-only read support, or always transfers the full object. - **S3-vs-XFS size threshold.** Needs validating against real small-file volume/throughput numbers — if S3 can serve small files efficiently enough at the actual expected volume, the XFS tier could be dropped entirely and S3 used as the sole backend, simplifying Storage Backend Handover to one path instead of two. - Exact JSON trailer schema beyond the `clearsign` field (format version, chunk descriptor list — whether chunk descriptors are duplicated at the JSON top level for easy programmatic access, or only present inside the signed attribute list and require parsing the clearsign block to read). - **Get the existing S3 handover interface spec document** referenced in Storage Backend Handover: the exact regex used to map `x-amz-meta-*` keys to NiFi attributes, and the exact flag/tag name and value convention used to signal the >2KB fallback to the JSON trailer. This design assumes that spec's behavior but doesn't reproduce its details.