wall
4008f61ce4
Initial commit of NiFi prototyping design docs: the existing Phat Files large-file transfer spec, the new packaging/signing/chunking design, and the whiteboard sketch (grand_plan.jpg) it was reconciled against.
452 lines
25 KiB
Markdown
452 lines
25 KiB
Markdown
# FlowFile Packaging & Signing — Design Spec
|
|
|
|
**Date:** 2026-07-28
|
|
**Status:** Draft — pending user review
|
|
|
|
## Purpose
|
|
|
|
NiFi is used to manage complex workflows with payloads ranging from tiny
|
|
files to TB scale. Two related problems need solving:
|
|
|
|
1. **Performance.** The current pack/unpack path (`MergeContent` +
|
|
`UnpackContent`) is disk-I/O bound under high throughput. Content lives on
|
|
vSAN storage, which is shared, virtualized IOPS across the vSphere
|
|
cluster — so the pain isn't any single slow disk, it's *contention*
|
|
across every VM sharing that storage. The current flow makes multiple
|
|
redundant passes over content on disk: an ingest write, a separate read
|
|
to compute a content hash, a read+write to merge/pack, and a read+write
|
|
to unpack. Each pass competes for the same shared IOPS pool. NiFi runs as
|
|
independent single-node instances (no clustering), each with its own
|
|
logical function, scaled horizontally by adding more instances — so
|
|
there's no cluster-wide load balancing to fall back on.
|
|
|
|
2. **Integrity & non-repudiation.** When a FlowFile (content + attributes)
|
|
is packed and exported outside NiFi's space, its attributes — which
|
|
already include a hash of the content, so content hashing is not part of
|
|
packing — need to be signed to prove they haven't been tampered with
|
|
and to prove who produced them, in a way that holds up even to a party
|
|
who didn't produce the artifact. Uses for the packed form: storage/
|
|
retrieval to/from S3, delivery to other applications via HTTP or
|
|
filesystem, and selective (date/category) replay from a catalog
|
|
database.
|
|
|
|
A **future** requirement extends this to TB-scale files that must be
|
|
broken into pieces, tracked, transited through NiFi (for routing and
|
|
content-assurance controls — all content must pass through NiFi), and
|
|
reconstructed at the destination — including across a **one-way network
|
|
boundary** — referred to throughout this doc as **MR**, the named relay
|
|
bridging the two NiFi host chains within the overall system referred to as
|
|
**SA** (see [Deployment Topology](#deployment-topology)) — with no return
|
|
channel for retransmission requests. Forward error correction for that hop
|
|
is assumed to be available via an existing HTTP service, with ~1GB as the
|
|
target chunk size fed to it.
|
|
|
|
## Non-goals
|
|
|
|
- Redesigning NiFi's clustering model. Instances remain independent,
|
|
single-node, horizontally scaled by function.
|
|
- Automated recovery signaling across the true one-way diode boundary — by
|
|
construction there is no return channel; recovery there is necessarily
|
|
operator-mediated (see [Error Handling](#error-handling)).
|
|
- Building or operating the forward-error-correction transport itself —
|
|
assumed to already exist as an HTTP service outside this design's scope.
|
|
- A general-purpose NiFi improvement library. This design covers exactly the
|
|
packaging, chunking, signing, delivery, and replay-indexing concerns
|
|
described above.
|
|
|
|
## Relationship to the "Phat Files" design
|
|
|
|
A separate, already-approved spec
|
|
(`2026-07-23-phat-files-design.md`) covers low-side-S3-to-high-side-S3
|
|
transfer of TB-scale files across a one-way boundary: fixed-size blocks,
|
|
resend-vs-duplicate handling, staging by block index, server-side assembly
|
|
via `UploadPartCopy`, stall monitoring, and cleanup. That spec is treated
|
|
here as a **source of proven concepts, not a fixed dependency** — this
|
|
design is not an extension of it and does not modify it. The following
|
|
ideas are adopted (generalized beyond S3-to-S3):
|
|
|
|
- Streaming byte-range reads; never buffer a whole file.
|
|
- Chunk descriptor attributes (index, total, size, checksum).
|
|
- Server-side assembly via `UploadPartCopy` from independently-staged
|
|
per-chunk objects, rather than a live multipart-upload session spanning
|
|
the transport.
|
|
- Explicit resend-vs-duplicate distinction via a flag, not inferred from
|
|
bytes.
|
|
- Stall detection + missing-chunk report, for hops with no return channel.
|
|
- Testing convention: pure logic extracted to statically-testable methods.
|
|
|
|
What's **changed**: packaging moves from stock FlowFile Package V3 to the
|
|
custom manifest-trailer format below (needed for signing and
|
|
attribute-independent reads); staging/monitoring/cleanup responsibilities
|
|
move to an external service rather than NiFi timer-driven processors;
|
|
completion tracking is dual-written to a durable marker *and* a catalog
|
|
database instead of an S3-only marker object.
|
|
|
|
## Architecture
|
|
|
|
### Deployment Topology
|
|
|
|
The whole system described in this doc is referred to as **SA**: a single
|
|
system containing two chains of independent, single-function NiFi hosts
|
|
linked by Site-to-Site (per the existing horizontal-scaling model — no
|
|
clustering, just chained instances), bridged in the middle by **MR** — the
|
|
named one-way relay this design has been built around (FEC via an HTTP
|
|
service, ~1GB chunks, per [Purpose](#purpose)). Content flows in one
|
|
direction through SA: external S3/S2S in, through the first chain, across
|
|
MR, through the second chain, external S3/S2S out. The Replay DB branches
|
|
off **before** MR, from node 1 of the *first* chain — i.e. at the true
|
|
ingest point into SA, not after delivery.
|
|
|
|
```
|
|
+------------------------------------ SA -------------------------------------+
|
|
| |
|
|
| S3 --\ /-- S3 |
|
|
| S2S --+-->[1]--S2S-->[2]--S2S-->[3]--HTTP-->[MR]--HTTP-->[1]--S2S-->[2]--S2S-->[3]--+-- S2S |
|
|
| | |
|
|
| "copy" |
|
|
| v |
|
|
| +-----------+ |
|
|
| | Replay DB | |
|
|
| +-----------+ |
|
|
| / \ |
|
|
| v v |
|
|
| S3 (large files) XFS (small files) |
|
|
+------------------------------------------------------------------------------------+
|
|
```
|
|
|
|
Chunking is applied uniformly at ingest (below) specifically so the same
|
|
chunked representation transits **both** the intra-chain S2S hops **and**
|
|
the MR relay — there's no separate chunking scheme per transport.
|
|
Hashing happens at this same ingest point — node 1 of the first chain, the
|
|
true entry into SA — consistent with the single-pass design below.
|
|
|
|
### Per-FlowFile Pipeline
|
|
|
|
```
|
|
INGEST (custom NiFi processor;
|
|
replaces ListenHTTP for HTTP,
|
|
or first-in-flow processor
|
|
right after the S2S Input Port)
|
|
|
|
|
v
|
|
Single streaming pass: write content to NiFi's
|
|
content repo (required — all content must transit
|
|
NiFi for routing/content-assurance) WHILE computing
|
|
the content hash and emitting a chunk boundary every
|
|
~1GB. No pre-ingest size check or path branching —
|
|
chunk count falls out of content length; a small file
|
|
is just the one-chunk case.
|
|
|
|
|
v
|
|
Ordered attribute list (incl. content hash, chunk
|
|
descriptors) is clear-signed in-process (Bouncy
|
|
Castle OpenPGP, key via Controller Service — see
|
|
Signing) and stored as a new attribute.
|
|
|
|
|
v
|
|
FlowFile(s) proceed through normal NiFi routing /
|
|
content-assurance flow logic.
|
|
|
|
|
v
|
|
DELIVERY: per-chunk content streamed once (single
|
|
content-repo read) to destination staging.
|
|
|
|
|
v
|
|
+-----------------------------------------------+
|
|
| DESTINATIONS |
|
|
| S3: stage per-chunk objects, assemble via |
|
|
| UploadPartCopy (no re-transfer) |
|
|
| filesystem: pwrite chunks at offset, preallocated|
|
|
| HTTP: deliver assembled object to receiving app |
|
|
| diode: chunks -> FEC HTTP service (~1GB each) |
|
|
+-----------------------------------------------+
|
|
| |
|
|
write marker upsert row
|
|
(durable, (fast index,
|
|
per-dest) rebuildable
|
|
| from markers)
|
|
v v
|
|
+-----------------------------------------------+
|
|
| EXTERNAL SERVICE (post-NiFi only): |
|
|
| assembly (UploadPartCopy), stall monitoring, |
|
|
| cleanup, catalog DB maintenance |
|
|
+-----------------------------------------------+
|
|
|
|
|
v
|
|
+-----------------------------------------------+
|
|
| CATALOG / REPLAY DATABASE |
|
|
+-----------------------------------------------+
|
|
```
|
|
|
|
A prior iteration of this design considered a sidecar process fronting
|
|
ingestion (to bypass NiFi's content repository entirely for large
|
|
payloads, via a claim-check pattern). That's rejected: **all content must
|
|
go through NiFi** for routing and content-assurance controls, which removes
|
|
the main justification for a front door outside the JVM. The external
|
|
service's role is narrowed to post-NiFi work that never needs to see raw
|
|
content: assembly, monitoring, cleanup, and catalog indexing.
|
|
|
|
## Packaged Artifact Format
|
|
|
|
This is the format used for filesystem/XFS delivery always, and for S3
|
|
delivery when the attribute set is too large for S3 user object metadata
|
|
(see [Storage Backend Handover](#storage-backend-handover)). Content and
|
|
its manifest travel as **one object**, not two, for end-to-end
|
|
loss protection: two separate objects (content + manifest) have independent
|
|
failure/lifecycle/replication paths that can diverge — e.g. content lands
|
|
but the manifest write fails, or a lifecycle rule reaps one and not the
|
|
other. A single completed write (one `PutObject`, or one completed
|
|
multipart upload) can't end up in that split state.
|
|
|
|
```
|
|
PACKAGED OBJECT (single PutObject, or single completed multipart upload)
|
|
+---------------------------------------------------------+
|
|
| CONTENT (1 object, or N ~1GB parts via multipart) |
|
|
+---------------------------------------------------------+
|
|
| MANIFEST TRAILER (final part -- direct UploadPart, not |
|
|
| UploadPartCopy, so it can be arbitrarily small) |
|
|
| = JSON document, e.g.: |
|
|
| { "format_version": ..., |
|
|
| "clearsign": "-----BEGIN PGP SIGNED MESSAGE----- |
|
|
| ...ordered attribute list... |
|
|
| -----BEGIN PGP SIGNATURE-----... |
|
|
| -----END PGP SIGNATURE-----" } |
|
|
| The clearsign field is self-contained and human- |
|
|
| verifiable via `gpg --verify` independent of the JSON |
|
|
| wrapper around it. |
|
|
+---------------------------------------------------------+
|
|
| FOOTER (fixed size, e.g. last 12 bytes: magic + 8-byte |
|
|
| big-endian trailer length) -- self-describing, storage- |
|
|
| agnostic; works identically on S3, filesystem, and HTTP |
|
|
+---------------------------------------------------------+
|
|
|
|
Optional S3 object tag (attr-size = trailer byte length) as an
|
|
accelerant: lets the catalog DB or a reconciliation scan jump straight
|
|
to the trailer's byte range without reading the footer first.
|
|
```
|
|
|
|
Manifest is written **after** content by construction — the ingest pass is
|
|
single-streaming, so the final content hash isn't known until content is
|
|
fully read, ruling out a header-first format (which would require either
|
|
buffering to know its size upfront, or a second pass to fix it up).
|
|
|
|
**Attribute-only reads** (replay filtering by date/category, signature
|
|
verification, routing) use a suffix-range read — `Range: bytes=-N` where
|
|
`N` is the trailer size (from the catalog DB, the S3 tag, or the footer) —
|
|
which never touches the (possibly TB-scale) content bytes and completes in
|
|
one round trip when the DB already knows the offset.
|
|
|
|
For chunked files, S3 multipart uploads mix `UploadPartCopy` parts (the
|
|
staged content chunks) with one ordinary `UploadPart` (the manifest
|
|
trailer — no 5MB minimum applies to the last part), so `
|
|
CompleteMultipartUpload` produces one final object with the same
|
|
content-then-trailer-then-footer shape regardless of size.
|
|
|
|
## Signing
|
|
|
|
- **Format:** OpenPGP detached-style trailer via `gpg --clearsign`
|
|
semantics — clear-sign the **canonical ordered attribute list** (sorted
|
|
by attribute key, including the content hash and any chunk descriptors),
|
|
producing a self-contained ASCII-armored block (plaintext + embedded
|
|
signature). The result is stored as a new FlowFile attribute, then
|
|
becomes the `clearsign` field of the JSON manifest trailer at pack time
|
|
(see [Packaged Artifact Format](#packaged-artifact-format)) — the JSON
|
|
wrapper carries packaging metadata (format version, chunk descriptors)
|
|
alongside the signed block, without altering the signed bytes themselves.
|
|
- **Why clear-sign / OpenPGP over a bespoke signature scheme:** an operator
|
|
on the far side of the one-way diode boundary can verify an artifact by
|
|
hand with stock GnuPG and a public key — no custom SDK needed. This
|
|
matters because that boundary already has a manual-operator recovery
|
|
model (missing-chunk reports carried by hand, per Phat Files);
|
|
OpenPGP fits that operational reality directly.
|
|
- **Where signing runs:** in-process, via Bouncy Castle's OpenPGP API (not
|
|
shelling out to the `gpg` CLI per FlowFile — too slow at throughput). No
|
|
external signing service — this was evaluated and explicitly rejected in
|
|
favor of keeping the architecture self-contained within NiFi. The private
|
|
key is loaded via a Controller Service (same pattern as the existing
|
|
`AwsCredentialsProviderService`), plain in-JVM key material protected by
|
|
NiFi's sensitive-property encryption and OS-level file permissions.
|
|
- **Non-repudiation trade-off, stated explicitly:** non-repudiation depends
|
|
on the private key being exclusively controlled by one identity. Moving
|
|
signing in-process (vs. a network-isolated external service) means the
|
|
key now lives on the same host as ingestion, so custody depends on
|
|
host-level protections rather than network isolation. Mitigations: only
|
|
the NiFi instances actually responsible for packing/export have the
|
|
signing Controller Service configured (not every instance in the
|
|
deployment); NiFi's provenance repository gives an audit trail of which
|
|
flow/processor invoked signing and when, partially substituting for the
|
|
audit log a dedicated signing service would give for free. This is not a
|
|
one-way door — the Controller Service abstraction means a PKCS11/HSM
|
|
token could be swapped in later as a stronger key-custody option without
|
|
changing processor logic, if ever required.
|
|
- **Algorithm:** Ed25519 preferred for speed and small signature size at
|
|
throughput, unless a compliance requirement (e.g. FIPS mode) forces
|
|
RSA/ECDSA instead — not yet confirmed as a constraint.
|
|
- **Tamper-detection vs. non-repudiation:** a symmetric HMAC alternative
|
|
was considered and rejected. HMAC would detect unauthorized modification
|
|
(any party without the shared secret can't forge a valid tag) but cannot
|
|
prove authorship to a third party, since every verifier necessarily holds
|
|
the same secret and could in principle have produced it themselves. The
|
|
stated requirement is non-repudiation, which needs an asymmetric scheme.
|
|
|
|
## Storage Backend Handover
|
|
|
|
Final storage backend is chosen by size, separate from the ingest-time
|
|
chunking decision (which is uniform, no threshold — see Architecture):
|
|
large files land in S3, small files land on a local/shared XFS filesystem.
|
|
This split exists because S3's per-request overhead and cost make it a
|
|
poor fit for very high volumes of tiny objects; XFS is cheaper and faster
|
|
for that case. **Open question, not yet resolved:** if S3 can be shown to
|
|
serve small files efficiently enough (e.g. via batching, a different
|
|
bucket/storage class, or acceptable cost at the actual small-file volume
|
|
expected), the XFS tier could be dropped and S3 used uniformly — this
|
|
needs validating against real small-file throughput numbers before
|
|
implementation, see [Open Items](#open-items-for-implementation-planning).
|
|
The JSON-trailer packaged artifact format is shared by both backends, but
|
|
S3 additionally supports an existing metadata-based mechanism for small
|
|
attribute sets that avoids the trailer entirely — see below.
|
|
|
|
### S3 path
|
|
|
|
**Attribute handover on S3 follows an existing, established handover
|
|
interface specification already used elsewhere in this environment — this
|
|
design integrates with it rather than replacing it.** Under that spec:
|
|
|
|
- Attributes are written as S3 user object metadata (`x-amz-meta-*`
|
|
headers) when small enough to fit. A regular expression (defined by that
|
|
existing spec, not this one) identifies which user metadata keys get
|
|
turned into NiFi attributes on read — so consuming the attributes is a
|
|
single `HeadObject` call, no content touch, not even a suffix-range read.
|
|
- A flag (also part of that existing spec) records whether the attribute
|
|
set exceeded S3's ~2KB user-metadata limit. When it does, the object
|
|
falls back to **this design's** JSON manifest trailer (Packaged Artifact
|
|
Format) instead of metadata.
|
|
- The exact regex pattern and flag key/tag name live in the existing
|
|
handover interface spec document, not reproduced here — see
|
|
[Open Items](#open-items-for-implementation-planning).
|
|
|
|
So for S3 specifically, the JSON trailer is the **fallback path** (oversized
|
|
attribute sets), not the only mechanism — small-attribute objects never get
|
|
a trailer at all. Filesystem/XFS delivery, which has no equivalent native
|
|
metadata mechanism, always uses the JSON trailer.
|
|
|
|
```
|
|
NiFi (per chunk FlowFile, content already in repo from ingest)
|
|
|
|
|
| PutObject (single read of the content-repo claim --
|
|
v the only disk touch on the delivery side)
|
|
S3 staging: staging/<generation-id>/<chunk-index>
|
|
|
|
|
| (all N chunks staged; external service detects
|
|
v completeness via catalog DB or S3 listing)
|
|
UploadPartCopy x N -- server-side, zero bytes touch NiFi or the
|
|
| external service
|
|
v
|
|
Attributes fit in 2KB?
|
|
yes -> PutObjectMetadata / CompleteMultipartUpload with attributes
|
|
as x-amz-meta-* headers (no trailer)
|
|
no -> + UploadPart (JSON manifest trailer), oversized flag set
|
|
v
|
|
Final object at <key>
|
|
|
|
|
v
|
|
write durable completion marker + upsert catalog DB row
|
|
|
|
|
v
|
|
delete staging/<generation-id>/* (cleanup)
|
|
```
|
|
|
|
### XFS path (small files, below the size threshold)
|
|
|
|
No staging/assembly needed — small files are single-chunk by construction
|
|
(Architecture), so NiFi writes the packaged object (content + JSON trailer
|
|
+ footer) directly to its final path in one `write()` call. The external
|
|
service still writes the durable completion marker (a sibling file, or an
|
|
xattr, depending on implementation preference) and upserts the catalog DB
|
|
row, same as the S3 path, so replay and reconciliation behave identically
|
|
regardless of backend.
|
|
|
|
Chunk upload failures for **plain S3/filesystem/HTTP delivery** are
|
|
ordinary NiFi FlowFile retry — Phat Files' resend/stall-report machinery is
|
|
reserved for the actual one-way diode hop, where there's no return channel;
|
|
applying it to destinations that do have bidirectional connectivity would
|
|
be unnecessary complexity.
|
|
|
|
## Catalog / Replay Database
|
|
|
|
One row per packed artifact, written alongside the durable completion
|
|
marker (same write-marker-then-upsert-row ordering as Section
|
|
"Architecture"). Indexed for date/category replay queries: which backend
|
|
(S3 or XFS), object location (S3 key or filesystem path), the clear-signed attribute list
|
|
(so filtering never touches the object), completion status, and
|
|
generation/chunk-set id for large files.
|
|
|
|
The database is explicitly a **rebuildable index, not a source of truth**
|
|
— the durable marker plus the trailer/footer on disk or S3 is authoritative,
|
|
and a reconciliation job can rebuild the catalog by scanning objects and
|
|
reading trailers if the database is ever lost, corrupted, or migrated.
|
|
Replay is: query by date/category → get object locations → re-inject as
|
|
FlowFiles via the same ingest path used for first-time delivery.
|
|
|
|
## Error Handling
|
|
|
|
| Condition | Handling |
|
|
|---|---|
|
|
| Checksum mismatch on chunk arrival | Route to failure, never staged |
|
|
| Resend vs. unrequested duplicate | Explicit flag on the FlowFile, checked against the durable completion marker (adopted from Phat Files) |
|
|
| Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. |
|
|
| Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists |
|
|
| Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch |
|
|
| Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers |
|
|
|
|
## Testing
|
|
|
|
Following the convention established in Phat Files: pure logic extracted
|
|
to package-private static methods, unit-testable without a live NiFi/S3
|
|
connection —
|
|
|
|
- Canonical attribute-list ordering and serialization (must be
|
|
deterministic for clear-sign to produce consistent output for the same
|
|
attribute set).
|
|
- Chunk-boundary computation from a content stream (byte offset math, ~1GB
|
|
target size).
|
|
- Footer encode/decode (magic + length) and suffix-range calculation.
|
|
- Missing-chunk diffing (expected range vs. staged contents) — adopted
|
|
from Phat Files.
|
|
- Resend-vs-duplicate decision logic — adopted from Phat Files.
|
|
|
|
End-to-end behavior (ingest → sign → chunk → stage → assemble → catalog
|
|
row → attribute-only read via suffix range → replay) verified against the
|
|
lab's MinIO instance, consistent with prior processors in this
|
|
environment.
|
|
|
|
## Open Items for Implementation Planning
|
|
|
|
- Exact NAR/module layout for the new ingest processor(s), Controller
|
|
Service (signing key), and the external post-NiFi service.
|
|
- Concrete canonical attribute-list serialization format (sorted
|
|
`key: value` lines vs. canonical JSON) — functionally equivalent, pick
|
|
one for implementation.
|
|
- Confirm whether Ed25519 is acceptable or a compliance constraint (e.g.
|
|
FIPS) forces RSA/ECDSA.
|
|
- Concrete default values: chunk size (default ~1GB, confirm whether it
|
|
should differ between diode-bound and non-diode-bound chunked delivery),
|
|
stall timeout, cleanup retention window.
|
|
- Catalog database technology selection (not yet specified).
|
|
- Whether HTTP delivery to receiving applications needs the same suffix-
|
|
range attribute-only read support, or always transfers the full object.
|
|
- **S3-vs-XFS size threshold.** Needs validating against real small-file
|
|
volume/throughput numbers — if S3 can serve small files efficiently
|
|
enough at the actual expected volume, the XFS tier could be dropped
|
|
entirely and S3 used as the sole backend, simplifying Storage Backend
|
|
Handover to one path instead of two.
|
|
- Exact JSON trailer schema beyond the `clearsign` field (format version,
|
|
chunk descriptor list — whether chunk descriptors are duplicated at the
|
|
JSON top level for easy programmatic access, or only present inside the
|
|
signed attribute list and require parsing the clearsign block to read).
|
|
- **Get the existing S3 handover interface spec document** referenced in
|
|
Storage Backend Handover: the exact regex used to map `x-amz-meta-*`
|
|
keys to NiFi attributes, and the exact flag/tag name and value convention
|
|
used to signal the >2KB fallback to the JSON trailer. This design assumes
|
|
that spec's behavior but doesn't reproduce its details.
|