Hackathon/GrandPlan
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Phat Files and FlowFile Packaging & Signing design specs

Browse Source 
Initial commit of NiFi prototyping design docs: the existing Phat Files
large-file transfer spec, the new packaging/signing/chunking design, and
the whiteboard sketch (grand_plan.jpg) it was reconciled against.
This commit is contained in:
wall
2026-07-28 14:17:11 +10:00
commit 4008f61ce4
3 changed files with 779 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md
+451
View File
@@ -0,0 +1,451 @@
# FlowFile Packaging & Signing — Design Spec
**Date:** 2026-07-28
**Status:** Draft — pending user review
## Purpose
NiFi is used to manage complex workflows with payloads ranging from tiny
files to TB scale. Two related problems need solving:
1. **Performance.** The current pack/unpack path (`MergeContent` +
`UnpackContent`) is disk-I/O bound under high throughput. Content lives on
vSAN storage, which is shared, virtualized IOPS across the vSphere
cluster — so the pain isn't any single slow disk, it's *contention*
across every VM sharing that storage. The current flow makes multiple
redundant passes over content on disk: an ingest write, a separate read
to compute a content hash, a read+write to merge/pack, and a read+write
to unpack. Each pass competes for the same shared IOPS pool. NiFi runs as
independent single-node instances (no clustering), each with its own
logical function, scaled horizontally by adding more instances — so
there's no cluster-wide load balancing to fall back on.
2. **Integrity & non-repudiation.** When a FlowFile (content + attributes)
is packed and exported outside NiFi's space, its attributes — which
already include a hash of the content, so content hashing is not part of
packing — need to be signed to prove they haven't been tampered with
and to prove who produced them, in a way that holds up even to a party
who didn't produce the artifact. Uses for the packed form: storage/
retrieval to/from S3, delivery to other applications via HTTP or
filesystem, and selective (date/category) replay from a catalog
database.
A **future** requirement extends this to TB-scale files that must be
broken into pieces, tracked, transited through NiFi (for routing and
content-assurance controls — all content must pass through NiFi), and
reconstructed at the destination — including across a **one-way network
boundary** — referred to throughout this doc as **MR**, the named relay
bridging the two NiFi host chains within the overall system referred to as
**SA** (see [Deployment Topology](#deployment-topology)) — with no return
channel for retransmission requests. Forward error correction for that hop
is assumed to be available via an existing HTTP service, with ~1GB as the
target chunk size fed to it.
## Non-goals
- Redesigning NiFi's clustering model. Instances remain independent,
single-node, horizontally scaled by function.
- Automated recovery signaling across the true one-way diode boundary — by
construction there is no return channel; recovery there is necessarily
operator-mediated (see [Error Handling](#error-handling)).
- Building or operating the forward-error-correction transport itself —
assumed to already exist as an HTTP service outside this design's scope.
- A general-purpose NiFi improvement library. This design covers exactly the
packaging, chunking, signing, delivery, and replay-indexing concerns
described above.
## Relationship to the "Phat Files" design
A separate, already-approved spec
(`2026-07-23-phat-files-design.md`) covers low-side-S3-to-high-side-S3
transfer of TB-scale files across a one-way boundary: fixed-size blocks,
resend-vs-duplicate handling, staging by block index, server-side assembly
via `UploadPartCopy`, stall monitoring, and cleanup. That spec is treated
here as a **source of proven concepts, not a fixed dependency** — this
design is not an extension of it and does not modify it. The following
ideas are adopted (generalized beyond S3-to-S3):
- Streaming byte-range reads; never buffer a whole file.
- Chunk descriptor attributes (index, total, size, checksum).
- Server-side assembly via `UploadPartCopy` from independently-staged
per-chunk objects, rather than a live multipart-upload session spanning
the transport.
- Explicit resend-vs-duplicate distinction via a flag, not inferred from
bytes.
- Stall detection + missing-chunk report, for hops with no return channel.
- Testing convention: pure logic extracted to statically-testable methods.
What's **changed**: packaging moves from stock FlowFile Package V3 to the
custom manifest-trailer format below (needed for signing and
attribute-independent reads); staging/monitoring/cleanup responsibilities
move to an external service rather than NiFi timer-driven processors;
completion tracking is dual-written to a durable marker *and* a catalog
database instead of an S3-only marker object.
## Architecture
### Deployment Topology
The whole system described in this doc is referred to as **SA**: a single
system containing two chains of independent, single-function NiFi hosts
linked by Site-to-Site (per the existing horizontal-scaling model — no
clustering, just chained instances), bridged in the middle by **MR** — the
named one-way relay this design has been built around (FEC via an HTTP
service, ~1GB chunks, per [Purpose](#purpose)). Content flows in one
direction through SA: external S3/S2S in, through the first chain, across
MR, through the second chain, external S3/S2S out. The Replay DB branches
off **before** MR, from node 1 of the *first* chain — i.e. at the true
ingest point into SA, not after delivery.
```
+------------------------------------ SA -------------------------------------+
| |
| S3 --\ /-- S3 |
| S2S --+-->[1]--S2S-->[2]--S2S-->[3]--HTTP-->[MR]--HTTP-->[1]--S2S-->[2]--S2S-->[3]--+-- S2S |
| | |
| "copy" |
| v |
| +-----------+ |
| | Replay DB | |
| +-----------+ |
| / \ |
| v v |
| S3 (large files) XFS (small files) |
+------------------------------------------------------------------------------------+
```
Chunking is applied uniformly at ingest (below) specifically so the same
chunked representation transits **both** the intra-chain S2S hops **and**
the MR relay — there's no separate chunking scheme per transport.
Hashing happens at this same ingest point — node 1 of the first chain, the
true entry into SA — consistent with the single-pass design below.
### Per-FlowFile Pipeline
```
INGEST (custom NiFi processor;
replaces ListenHTTP for HTTP,
or first-in-flow processor
right after the S2S Input Port)
|
v
Single streaming pass: write content to NiFi's
content repo (required — all content must transit
NiFi for routing/content-assurance) WHILE computing
the content hash and emitting a chunk boundary every
~1GB. No pre-ingest size check or path branching —
chunk count falls out of content length; a small file
is just the one-chunk case.
|
v
Ordered attribute list (incl. content hash, chunk
descriptors) is clear-signed in-process (Bouncy
Castle OpenPGP, key via Controller Service — see
Signing) and stored as a new attribute.
|
v
FlowFile(s) proceed through normal NiFi routing /
content-assurance flow logic.
|
v
DELIVERY: per-chunk content streamed once (single
content-repo read) to destination staging.
|
v
+-----------------------------------------------+
| DESTINATIONS |
| S3: stage per-chunk objects, assemble via |
| UploadPartCopy (no re-transfer) |
| filesystem: pwrite chunks at offset, preallocated|
| HTTP: deliver assembled object to receiving app |
| diode: chunks -> FEC HTTP service (~1GB each) |
+-----------------------------------------------+
| |
write marker upsert row
(durable, (fast index,
per-dest) rebuildable
| from markers)
v v
+-----------------------------------------------+
| EXTERNAL SERVICE (post-NiFi only): |
| assembly (UploadPartCopy), stall monitoring, |
| cleanup, catalog DB maintenance |
+-----------------------------------------------+
|
v
+-----------------------------------------------+
| CATALOG / REPLAY DATABASE |
+-----------------------------------------------+
```
A prior iteration of this design considered a sidecar process fronting
ingestion (to bypass NiFi's content repository entirely for large
payloads, via a claim-check pattern). That's rejected: **all content must
go through NiFi** for routing and content-assurance controls, which removes
the main justification for a front door outside the JVM. The external
service's role is narrowed to post-NiFi work that never needs to see raw
content: assembly, monitoring, cleanup, and catalog indexing.
## Packaged Artifact Format
This is the format used for filesystem/XFS delivery always, and for S3
delivery when the attribute set is too large for S3 user object metadata
(see [Storage Backend Handover](#storage-backend-handover)). Content and
its manifest travel as **one object**, not two, for end-to-end
loss protection: two separate objects (content + manifest) have independent
failure/lifecycle/replication paths that can diverge — e.g. content lands
but the manifest write fails, or a lifecycle rule reaps one and not the
other. A single completed write (one `PutObject`, or one completed
multipart upload) can't end up in that split state.
```
PACKAGED OBJECT (single PutObject, or single completed multipart upload)
+---------------------------------------------------------+
| CONTENT (1 object, or N ~1GB parts via multipart) |
+---------------------------------------------------------+
| MANIFEST TRAILER (final part -- direct UploadPart, not |
| UploadPartCopy, so it can be arbitrarily small) |
| = JSON document, e.g.: |
| { "format_version": ..., |
| "clearsign": "-----BEGIN PGP SIGNED MESSAGE----- |
| ...ordered attribute list... |
| -----BEGIN PGP SIGNATURE-----... |
| -----END PGP SIGNATURE-----" } |
| The clearsign field is self-contained and human- |
| verifiable via `gpg --verify` independent of the JSON |
| wrapper around it. |
+---------------------------------------------------------+
| FOOTER (fixed size, e.g. last 12 bytes: magic + 8-byte |
| big-endian trailer length) -- self-describing, storage- |
| agnostic; works identically on S3, filesystem, and HTTP |
+---------------------------------------------------------+
Optional S3 object tag (attr-size = trailer byte length) as an
accelerant: lets the catalog DB or a reconciliation scan jump straight
to the trailer's byte range without reading the footer first.
```
Manifest is written **after** content by construction — the ingest pass is
single-streaming, so the final content hash isn't known until content is
fully read, ruling out a header-first format (which would require either
buffering to know its size upfront, or a second pass to fix it up).
**Attribute-only reads** (replay filtering by date/category, signature
verification, routing) use a suffix-range read — `Range: bytes=-N` where
`N` is the trailer size (from the catalog DB, the S3 tag, or the footer) —
which never touches the (possibly TB-scale) content bytes and completes in
one round trip when the DB already knows the offset.
For chunked files, S3 multipart uploads mix `UploadPartCopy` parts (the
staged content chunks) with one ordinary `UploadPart` (the manifest
trailer — no 5MB minimum applies to the last part), so `
CompleteMultipartUpload` produces one final object with the same
content-then-trailer-then-footer shape regardless of size.
## Signing
- **Format:** OpenPGP detached-style trailer via `gpg --clearsign`
semantics — clear-sign the **canonical ordered attribute list** (sorted
by attribute key, including the content hash and any chunk descriptors),
producing a self-contained ASCII-armored block (plaintext + embedded
signature). The result is stored as a new FlowFile attribute, then
becomes the `clearsign` field of the JSON manifest trailer at pack time
(see [Packaged Artifact Format](#packaged-artifact-format)) — the JSON
wrapper carries packaging metadata (format version, chunk descriptors)
alongside the signed block, without altering the signed bytes themselves.
- **Why clear-sign / OpenPGP over a bespoke signature scheme:** an operator
on the far side of the one-way diode boundary can verify an artifact by
hand with stock GnuPG and a public key — no custom SDK needed. This
matters because that boundary already has a manual-operator recovery
model (missing-chunk reports carried by hand, per Phat Files);
OpenPGP fits that operational reality directly.
- **Where signing runs:** in-process, via Bouncy Castle's OpenPGP API (not
shelling out to the `gpg` CLI per FlowFile — too slow at throughput). No
external signing service — this was evaluated and explicitly rejected in
favor of keeping the architecture self-contained within NiFi. The private
key is loaded via a Controller Service (same pattern as the existing
`AwsCredentialsProviderService`), plain in-JVM key material protected by
NiFi's sensitive-property encryption and OS-level file permissions.
- **Non-repudiation trade-off, stated explicitly:** non-repudiation depends
on the private key being exclusively controlled by one identity. Moving
signing in-process (vs. a network-isolated external service) means the
key now lives on the same host as ingestion, so custody depends on
host-level protections rather than network isolation. Mitigations: only
the NiFi instances actually responsible for packing/export have the
signing Controller Service configured (not every instance in the
deployment); NiFi's provenance repository gives an audit trail of which
flow/processor invoked signing and when, partially substituting for the
audit log a dedicated signing service would give for free. This is not a
one-way door — the Controller Service abstraction means a PKCS11/HSM
token could be swapped in later as a stronger key-custody option without
changing processor logic, if ever required.
- **Algorithm:** Ed25519 preferred for speed and small signature size at
throughput, unless a compliance requirement (e.g. FIPS mode) forces
RSA/ECDSA instead — not yet confirmed as a constraint.
- **Tamper-detection vs. non-repudiation:** a symmetric HMAC alternative
was considered and rejected. HMAC would detect unauthorized modification
(any party without the shared secret can't forge a valid tag) but cannot
prove authorship to a third party, since every verifier necessarily holds
the same secret and could in principle have produced it themselves. The
stated requirement is non-repudiation, which needs an asymmetric scheme.
## Storage Backend Handover
Final storage backend is chosen by size, separate from the ingest-time
chunking decision (which is uniform, no threshold — see Architecture):
large files land in S3, small files land on a local/shared XFS filesystem.
This split exists because S3's per-request overhead and cost make it a
poor fit for very high volumes of tiny objects; XFS is cheaper and faster
for that case. **Open question, not yet resolved:** if S3 can be shown to
serve small files efficiently enough (e.g. via batching, a different
bucket/storage class, or acceptable cost at the actual small-file volume
expected), the XFS tier could be dropped and S3 used uniformly — this
needs validating against real small-file throughput numbers before
implementation, see [Open Items](#open-items-for-implementation-planning).
The JSON-trailer packaged artifact format is shared by both backends, but
S3 additionally supports an existing metadata-based mechanism for small
attribute sets that avoids the trailer entirely — see below.
### S3 path
**Attribute handover on S3 follows an existing, established handover
interface specification already used elsewhere in this environment — this
design integrates with it rather than replacing it.** Under that spec:
- Attributes are written as S3 user object metadata (`x-amz-meta-*`
headers) when small enough to fit. A regular expression (defined by that
existing spec, not this one) identifies which user metadata keys get
turned into NiFi attributes on read — so consuming the attributes is a
single `HeadObject` call, no content touch, not even a suffix-range read.
- A flag (also part of that existing spec) records whether the attribute
set exceeded S3's ~2KB user-metadata limit. When it does, the object
falls back to **this design's** JSON manifest trailer (Packaged Artifact
Format) instead of metadata.
- The exact regex pattern and flag key/tag name live in the existing
handover interface spec document, not reproduced here — see
[Open Items](#open-items-for-implementation-planning).
So for S3 specifically, the JSON trailer is the **fallback path** (oversized
attribute sets), not the only mechanism — small-attribute objects never get
a trailer at all. Filesystem/XFS delivery, which has no equivalent native
metadata mechanism, always uses the JSON trailer.
```
NiFi (per chunk FlowFile, content already in repo from ingest)
|
| PutObject (single read of the content-repo claim --
v the only disk touch on the delivery side)
S3 staging: staging/<generation-id>/<chunk-index>
|
| (all N chunks staged; external service detects
v completeness via catalog DB or S3 listing)
UploadPartCopy x N -- server-side, zero bytes touch NiFi or the
| external service
v
Attributes fit in 2KB?
yes -> PutObjectMetadata / CompleteMultipartUpload with attributes
as x-amz-meta-* headers (no trailer)
no -> + UploadPart (JSON manifest trailer), oversized flag set
v
Final object at <key>
|
v
write durable completion marker + upsert catalog DB row
|
v
delete staging/<generation-id>/* (cleanup)
```
### XFS path (small files, below the size threshold)
No staging/assembly needed — small files are single-chunk by construction
(Architecture), so NiFi writes the packaged object (content + JSON trailer
+ footer) directly to its final path in one `write()` call. The external
service still writes the durable completion marker (a sibling file, or an
xattr, depending on implementation preference) and upserts the catalog DB
row, same as the S3 path, so replay and reconciliation behave identically
regardless of backend.
Chunk upload failures for **plain S3/filesystem/HTTP delivery** are
ordinary NiFi FlowFile retry — Phat Files' resend/stall-report machinery is
reserved for the actual one-way diode hop, where there's no return channel;
applying it to destinations that do have bidirectional connectivity would
be unnecessary complexity.
## Catalog / Replay Database
One row per packed artifact, written alongside the durable completion
marker (same write-marker-then-upsert-row ordering as Section
"Architecture"). Indexed for date/category replay queries: which backend
(S3 or XFS), object location (S3 key or filesystem path), the clear-signed attribute list
(so filtering never touches the object), completion status, and
generation/chunk-set id for large files.
The database is explicitly a **rebuildable index, not a source of truth**
— the durable marker plus the trailer/footer on disk or S3 is authoritative,
and a reconciliation job can rebuild the catalog by scanning objects and
reading trailers if the database is ever lost, corrupted, or migrated.
Replay is: query by date/category → get object locations → re-inject as
FlowFiles via the same ingest path used for first-time delivery.
## Error Handling
| Condition | Handling |
|---|---|
| Checksum mismatch on chunk arrival | Route to failure, never staged |
| Resend vs. unrequested duplicate | Explicit flag on the FlowFile, checked against the durable completion marker (adopted from Phat Files) |
| Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. |
| Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists |
| Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch |
| Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers |
## Testing
Following the convention established in Phat Files: pure logic extracted
to package-private static methods, unit-testable without a live NiFi/S3
connection —
- Canonical attribute-list ordering and serialization (must be
deterministic for clear-sign to produce consistent output for the same
attribute set).
- Chunk-boundary computation from a content stream (byte offset math, ~1GB
target size).
- Footer encode/decode (magic + length) and suffix-range calculation.
- Missing-chunk diffing (expected range vs. staged contents) — adopted
from Phat Files.
- Resend-vs-duplicate decision logic — adopted from Phat Files.
End-to-end behavior (ingest → sign → chunk → stage → assemble → catalog
row → attribute-only read via suffix range → replay) verified against the
lab's MinIO instance, consistent with prior processors in this
environment.
## Open Items for Implementation Planning
- Exact NAR/module layout for the new ingest processor(s), Controller
Service (signing key), and the external post-NiFi service.
- Concrete canonical attribute-list serialization format (sorted
`key: value` lines vs. canonical JSON) — functionally equivalent, pick
one for implementation.
- Confirm whether Ed25519 is acceptable or a compliance constraint (e.g.
FIPS) forces RSA/ECDSA.
- Concrete default values: chunk size (default ~1GB, confirm whether it
should differ between diode-bound and non-diode-bound chunked delivery),
stall timeout, cleanup retention window.
- Catalog database technology selection (not yet specified).
- Whether HTTP delivery to receiving applications needs the same suffix-
range attribute-only read support, or always transfers the full object.
- **S3-vs-XFS size threshold.** Needs validating against real small-file
volume/throughput numbers — if S3 can serve small files efficiently
enough at the actual expected volume, the XFS tier could be dropped
entirely and S3 used as the sole backend, simplifying Storage Backend
Handover to one path instead of two.
- Exact JSON trailer schema beyond the `clearsign` field (format version,
chunk descriptor list — whether chunk descriptors are duplicated at the
JSON top level for easy programmatic access, or only present inside the
signed attribute list and require parsing the clearsign block to read).
- **Get the existing S3 handover interface spec document** referenced in
Storage Backend Handover: the exact regex used to map `x-amz-meta-*`
keys to NiFi attributes, and the exact flag/tag name and value convention
used to signal the >2KB fallback to the JSON trailer. This design assumes
that spec's behavior but doesn't reproduce its details.