Hackathon/GrandPlan
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Phat Files and FlowFile Packaging & Signing design specs

Browse Source 
Initial commit of NiFi prototyping design docs: the existing Phat Files
large-file transfer spec, the new packaging/signing/chunking design, and
the whiteboard sketch (grand_plan.jpg) it was reconciled against.
This commit is contained in:
wall
2026-07-28 14:17:11 +10:00
commit 4008f61ce4
3 changed files with 779 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
2026-07-23-phat-files-design.md
Executable
+328
View File
@@ -0,0 +1,328 @@
# Phat Files — Design Spec
**Date:** 2026-07-23
**Status:** Approved for implementation planning
## Purpose
Move very large (TB-scale) files from a low-side S3-compatible bucket to a
high-side S3-compatible bucket across a one-way boundary, with no automated
return channel. Files are split into fixed-size blocks so that transfers can
be resumed by resending only specific missing blocks, without ever needing to
restart or re-transmit a whole file.
The transport that actually carries a block from low side to high side is
intentionally out of scope for this spec — it could be a network relay
(NiFi Site-to-Site, HTTP push), physical removable media, or any other
one-way carriage mechanism. The design only assumes it's one-way. Packaging
a FlowFile (content + attributes) into something portable enough to survive
that transport, and reconstructing it on the other side, is handled by
NiFi's standard pack/unpack FlowFile processors (e.g. `MergeContent` using
the FlowFile Stream packaging format, paired with `UnpackContent` on the
receiving side) — not by anything custom in this project. `FetchS3ObjectBlock`
and `PutS3ObjectBlock` only ever deal with ordinary FlowFiles; whatever sits
between them for the boundary crossing is someone else's stock processor.
This is a cross-domain-style transfer: data flows low → high only. Any
recovery signal (which blocks are missing) must be produced on the high side
and manually carried back down to the low side by an operator — there is no
network path from high side back to low side.
## Non-goals
- Content-defined/delta chunking for re-syncing *changed* files. Only
fixed-size blocks and resend-of-missing-blocks are in scope.
- Automated high-to-low signaling of any kind (by design constraint, not an
oversight).
- Bidirectional transfer (high → low) — this spec covers one direction only.
A mirror-image flow could reuse the same processors in principle, but is
out of scope here.
## Architecture
```
[Low-side S3] <--Range GET--> [Low-side NiFi] --one-way transport--> [High-side NiFi] --PutObject/UploadPartCopy--> [High-side S3]
^ (mechanism deferred: |
| network relay, media, etc.) |
resend blocks via missing-block report
processor properties (JSON, written by
(static, EL/attribute, MonitorS3StagingStalls)
or file-driven) |
| |
+----------------- operator manually carries report file down -------------------------+
```
- **Low-side NiFi**: reads byte ranges of a file from low-side S3 via ranged
`GetObject` (streaming read, never buffers a whole file), splits into
fixed-size blocks, and hands each block FlowFile to a stock pack/unpack
pair (e.g. `MergeContent` + `UnpackContent`) that carries it one-way
across whatever transport is chosen for the boundary and reconstructs it
as a FlowFile again on the high side.
- **High-side NiFi**: receives blocks, stages each as its own object in
high-side S3 (streaming `PutObject`), and once a file's full block set is
present, assembles the real object server-side via `UploadPartCopy` (no
bytes re-transferred through NiFi) and cleans up staging objects.
- **No return channel.** Recovery is manual: the high side detects missing or
stalled blocks and writes a JSON report; an operator carries that report
down to the low side (per their existing cross-domain process) to drive a
targeted resend.
## Components
### Low side: `FetchS3ObjectBlock` (custom processor)
Reads a byte range of a source S3 object and emits it as a block FlowFile.
**Properties** (all Expression-Language-enabled against FlowFile attributes,
so each can be set statically, via `${attribute}`, or driven by an upstream
flow that parses a carried-over report file):
| Property | Description |
|---|---|
| Bucket | Source S3 bucket |
| Object Key | Source S3 object key |
| Block Size | Block size in bytes (configurable; determines byte-range math) |
| Block Numbers | `all`, or an explicit list/ranges, e.g. `0-4,7,10-12` |
| Resend | Boolean, default `false`. Marks the emitted blocks as a deliberate resend rather than an initial/routine transfer — see [Resend vs. duplicate](#resend-vs-duplicate) below. |
**Behavior on trigger:**
1. `HeadObject` to get file size and source ETag.
2. Compute total block count = `ceil(fileSize / blockSize)`.
3. For each requested block index, compute byte range
`[index * blockSize, min((index + 1) * blockSize, fileSize))`.
4. Issue a ranged `GetObject` per requested block (streaming, not buffered).
5. Emit one FlowFile per block with attributes:
- `phat.object.key`
- `phat.source.etag`
- `phat.block.index`
- `phat.block.total`
- `phat.block.size`
- `phat.block.checksum` (SHA-256 of the block's content)
- `phat.resend` (`true`/`false`, copied from the Resend property)
That's it — `FetchS3ObjectBlock` only produces ordinary FlowFiles (content +
attributes). Getting them across the boundary intact is the pack/unpack
pair's job, not this processor's (see Architecture above).
### Low side: report-driven resend (stock processors)
A small flow segment, built from stock NiFi processors, watches a drop
location for a carried-over missing-block report file:
1. `GetFile`/`FetchFile` picks up the report JSON at the watched path.
2. `EvaluateJsonPath` extracts `objectKey`, `bucket`, `sourceEtag`,
`blockSize`, and `missingBlocks` into FlowFile attributes, and sets
`resend = true` unconditionally — a report-driven trigger is by
definition always a resend, never an initial transfer.
3. Those attributes feed `FetchS3ObjectBlock`'s EL-bound properties
(including its `Resend` property), so the correct blocks are resent
automatically — no manual property editing required for the common case.
Manual override (typing values directly into `FetchS3ObjectBlock`'s
properties, including setting `Resend` to `true`) remains available for
ad hoc use.
### High side: `PutS3ObjectBlock` (custom processor)
Receives a block FlowFile and stages it.
**Behavior:**
1. Verify `phat.block.checksum` against the received content. On mismatch,
route to `failure` — never stage corrupted data.
2. If `phat.resend` is **not** `true`: check whether a completion marker
exists at `_phat/completed/<key>/<etag>` (see
[Completion markers](#completion-markers) below). If it does, this block
is an unrequested duplicate of an already-completed transfer of this
exact file version — discard immediately, do not stage. (See
[Resend vs. duplicate](#resend-vs-duplicate) below for why this check is
skipped when `phat.resend` is `true`.)
3. Otherwise, stream the block to
`staging/<key>/<etag>/<block-index>` via `PutObject`, and flag that
`<key>/<etag>` for re-assembly regardless of whether it was previously
marked complete.
Staging by block index makes out-of-order arrival a non-issue (each block
writes to its own path regardless of arrival order) and makes duplicates
within the same in-progress transfer idempotent (re-writing the same path is
harmless).
#### Resend vs. duplicate
Two situations look identical at the byte level — a block for a `<key>/<etag>`
whose transfer has already been marked complete (see
[Completion markers](#completion-markers) below) — but need opposite handling:
- **Unrequested duplicate**: the transport (or an operator) accidentally
re-sent a full, unflagged transfer after it already completed — e.g. a
retried batch job, or a stray retransmission at the transport layer. This
carries no new information and should be silently absorbed.
- **Requested resend**: an operator deliberately re-sent specific blocks —
either because a missing-block report said so, or because they have
independent reason to distrust the already-assembled object (e.g.
suspected corruption) — and wants it honored even if the destination
believes the transfer already completed.
The two are distinguished purely by intent, which only a human (or the
report-parsing flow acting on the human's behalf) can supply — nothing about
the bytes themselves says which one it is. That's what the `Resend` property
on `FetchS3ObjectBlock` is for: it's `false` by default (routine/initial
transfers), and the report-driven resend flow and manual ad hoc resends both
set it `true`. `PutS3ObjectBlock` trusts that flag: `phat.resend=true` always
stages the block and re-triggers `AssembleS3ObjectBlocks` for that
`<key>/<etag>`, even over a previously-completed one; `phat.resend=false` (or
absent) is subject to the duplicate-of-completed short-circuit.
This means a resend can correct or re-confirm an already-assembled object,
while a plain accidental replay of a finished transfer costs nothing.
#### Completion markers
Duplicate detection cannot rely on the final assembled object itself,
because the whole point of landing it at `<key>` is that something
downstream collects it — and typically moves or deletes it afterward. If
"transfer complete" were only recorded as a tag on that object, the record
would vanish the moment it's collected, and a late unrequested duplicate
block arriving after that point would find nothing to match against and get
staged as if it were a new transfer.
So completion is recorded independently: once `AssembleS3ObjectBlocks`
successfully assembles `<key>` for a given `<etag>`, it also writes a small,
separate marker object at `_phat/completed/<key>/<etag>` (empty or a tiny
JSON blob — content doesn't matter, only presence). This marker is never
touched by whatever collects the real deliverable, so duplicate detection in
`PutS3ObjectBlock` keeps working regardless of what happens to the object at
`<key>` afterward.
Markers are small (one per completed file *generation*, not per block), but
unlike staging blocks they are not deleted as part of normal completion —
left alone, they'd accumulate indefinitely. `CleanupCompletionMarkers` (see
below) TTL-expires them on a separate, typically much longer, retention
window than staging cleanup, since the tradeoff here is audit/duplicate-
protection value vs. storage, not "is this transfer abandoned."
### High side: `AssembleS3ObjectBlocks` (custom processor)
Triggered after each block lands (or on a schedule), including re-triggers
from a resend against an already-completed `<key>/<etag>` (see
[Resend vs. duplicate](#resend-vs-duplicate)). For the block's
`<key>/<etag>`, lists the staging prefix and compares the object count
against `phat.block.total`. If complete:
1. Assemble (or re-assemble) the final object at `<key>` via server-side
`UploadPartCopy` from the staged blocks (no bytes move back through
NiFi) — this overwrites any previously-assembled object for the same
`<key>/<etag>`, which is the intended effect of a resend.
2. Tag the final object with `x-phat-source-etag` = the source ETag it was
built from.
3. Write (or refresh the timestamp on) the completion marker at
`_phat/completed/<key>/<etag>` — see [Completion markers](#completion-markers).
4. Delete the staging prefix for that `<key>/<etag>`.
### High side: `MonitorS3StagingStalls` (custom processor)
Timer-driven. For each incomplete `<key>/<etag>` staging prefix, checks the
write time of the most recently staged block. If older than the configurable
**Stall Timeout**, raises an event (NiFi bulletin/log/FlowFile) and writes a
JSON missing-block report:
```json
{
"objectKey": "...",
"bucket": "...",
"sourceEtag": "...",
"blockSize": 104857600,
"missingBlocks": [3, 17, 42]
}
```
Missing block indices are computed directly by diffing the expected range
`0..total-1` against what's present in the staging prefix — no separate
tracking state needed.
**Constraint:** Stall Timeout must be configured shorter than the Cleanup
Retention Window below, so operators get a chance to notice and request a
resend before the data they've already received is reaped.
### High side: `CleanupStagingBlocks` (custom processor)
Timer-driven housekeeping, independent of alerting. Reaps any `<key>/<etag>`
staging prefix whose most recent write is older than the configurable
**Cleanup Retention Window**, so abandoned partial transfers don't grow
storage unbounded.
### High side: `CleanupCompletionMarkers` (custom processor)
Timer-driven housekeeping for [completion markers](#completion-markers).
Deletes any `_phat/completed/<key>/<etag>` marker older than the
configurable **Completion Marker Retention** window. This is a separate,
independently-configured window from the Cleanup Retention Window above —
staging cleanup is about not hoarding abandoned partial transfers (short
timescale), while marker retention is about how long duplicate-of-completed
protection stays effective after a transfer succeeds (can be a much longer
timescale, since markers are cheap). Once a marker expires, a genuinely late
unrequested duplicate for that generation would no longer be recognized as
one and would be staged — an accepted tradeoff of choosing a finite
retention window over "keep forever."
## Data Flow Summary
| Stage | Location | Mechanism |
|---|---|---|
| Read | Low-side S3 | Ranged `GetObject` (streaming) |
| Transport | Low → High | One-way, mechanism deferred (network relay, removable media, etc.) |
| Stage | High-side S3 | `PutObject` to `staging/<key>/<etag>/<block-index>` |
| Assemble | High-side S3 | Server-side `UploadPartCopy`, no re-transfer |
| Completion record | High-side S3 | Marker object at `_phat/completed/<key>/<etag>`, independent of the deliverable's lifecycle |
| Recovery signal | High side → operator → Low side | JSON report, manually carried |
## Error Handling
| Condition | Handling |
|---|---|
| Checksum mismatch on arrival | Route to `failure`, never staged |
| Unrequested duplicate, transfer already completed (`phat.resend=false`) | Detected via presence of `_phat/completed/<key>/<etag>` marker; discarded silently |
| Requested resend, transfer already completed (`phat.resend=true`) | Always staged; re-triggers `AssembleS3ObjectBlocks`, overwriting the previously-assembled object and refreshing its marker |
| Duplicate block, transfer in progress | Idempotent overwrite of the same staging path |
| Completion marker expired (beyond Completion Marker Retention) | Duplicate protection no longer applies for that generation; a late duplicate would be staged as if new (accepted tradeoff) |
| Out-of-order block arrival | No-op by construction (staging keyed by block index) |
| Transfer stalled (blocks stopped arriving) | `MonitorS3StagingStalls` raises an alert + JSON report with exact missing block indices, before the Cleanup Retention Window would reap it |
| Transfer abandoned beyond retention window | `CleanupStagingBlocks` reaps the staging prefix |
## Testing
Following this project family's existing convention (see
`nifi-custom-processors`'s `processor-design.md`): pure logic is extracted
into package-private static methods so it can be unit-tested without a live
NiFi/S3 connection:
- Byte-range computation from block size / index / file size
- Block-number list/range parsing (`all`, `0-4,7,10-12`)
- Staging path construction and completion-marker path construction
- Missing-block diffing (expected range vs. staged contents)
- Resend-vs-duplicate decision logic (`phat.resend` combined with marker
presence/absence)
End-to-end behavior (fetch → stage → assemble → marker write → cleanup →
stall alert → marker expiry) is verified against the lab's MinIO instance,
consistent with how prior processors in this environment were tested. Also
worth an explicit test: assemble a transfer, delete/move the final object
(simulating downstream collection) exactly as a real consumer would, then
confirm an unrequested duplicate block is still correctly discarded and a
flagged resend still correctly re-assembles.
## Open Items for Implementation Planning
- Exact NAR/module layout (likely mirrors `nifi-custom-processors`: separate
low-side and high-side processor modules, or one bundle with both, deployed
selectively per NiFi instance).
- Controller service reuse for AWS credentials (existing
`AwsCredentialsProviderService` pattern from `nifi-custom-processors`
should carry over directly).
- Concrete default values for Block Size, Stall Timeout, Cleanup Retention
Window, and Completion Marker Retention.
- **Transport mechanism selection.** Deliberately deferred (see Purpose).
Whatever is chosen — NiFi Site-to-Site, HTTP push, or physical media —
must be genuinely one-way (note that standard S2S involves a bidirectional
peer/site-info handshake even when FlowFile data only moves one direction,
so it may not qualify as-is if zero bidirectional connectivity is
required). Packaging/unpackaging the FlowFile for that transport is
handled by stock NiFi processors (e.g. `MergeContent` + `UnpackContent`),
not by anything custom in this project.
docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md
+451
View File
@@ -0,0 +1,451 @@
# FlowFile Packaging & Signing — Design Spec
**Date:** 2026-07-28
**Status:** Draft — pending user review
## Purpose
NiFi is used to manage complex workflows with payloads ranging from tiny
files to TB scale. Two related problems need solving:
1. **Performance.** The current pack/unpack path (`MergeContent` +
`UnpackContent`) is disk-I/O bound under high throughput. Content lives on
vSAN storage, which is shared, virtualized IOPS across the vSphere
cluster — so the pain isn't any single slow disk, it's *contention*
across every VM sharing that storage. The current flow makes multiple
redundant passes over content on disk: an ingest write, a separate read
to compute a content hash, a read+write to merge/pack, and a read+write
to unpack. Each pass competes for the same shared IOPS pool. NiFi runs as
independent single-node instances (no clustering), each with its own
logical function, scaled horizontally by adding more instances — so
there's no cluster-wide load balancing to fall back on.
2. **Integrity & non-repudiation.** When a FlowFile (content + attributes)
is packed and exported outside NiFi's space, its attributes — which
already include a hash of the content, so content hashing is not part of
packing — need to be signed to prove they haven't been tampered with
and to prove who produced them, in a way that holds up even to a party
who didn't produce the artifact. Uses for the packed form: storage/
retrieval to/from S3, delivery to other applications via HTTP or
filesystem, and selective (date/category) replay from a catalog
database.
A **future** requirement extends this to TB-scale files that must be
broken into pieces, tracked, transited through NiFi (for routing and
content-assurance controls — all content must pass through NiFi), and
reconstructed at the destination — including across a **one-way network
boundary** — referred to throughout this doc as **MR**, the named relay
bridging the two NiFi host chains within the overall system referred to as
**SA** (see [Deployment Topology](#deployment-topology)) — with no return
channel for retransmission requests. Forward error correction for that hop
is assumed to be available via an existing HTTP service, with ~1GB as the
target chunk size fed to it.
## Non-goals
- Redesigning NiFi's clustering model. Instances remain independent,
single-node, horizontally scaled by function.
- Automated recovery signaling across the true one-way diode boundary — by
construction there is no return channel; recovery there is necessarily
operator-mediated (see [Error Handling](#error-handling)).
- Building or operating the forward-error-correction transport itself —
assumed to already exist as an HTTP service outside this design's scope.
- A general-purpose NiFi improvement library. This design covers exactly the
packaging, chunking, signing, delivery, and replay-indexing concerns
described above.
## Relationship to the "Phat Files" design
A separate, already-approved spec
(`2026-07-23-phat-files-design.md`) covers low-side-S3-to-high-side-S3
transfer of TB-scale files across a one-way boundary: fixed-size blocks,
resend-vs-duplicate handling, staging by block index, server-side assembly
via `UploadPartCopy`, stall monitoring, and cleanup. That spec is treated
here as a **source of proven concepts, not a fixed dependency** — this
design is not an extension of it and does not modify it. The following
ideas are adopted (generalized beyond S3-to-S3):
- Streaming byte-range reads; never buffer a whole file.
- Chunk descriptor attributes (index, total, size, checksum).
- Server-side assembly via `UploadPartCopy` from independently-staged
per-chunk objects, rather than a live multipart-upload session spanning
the transport.
- Explicit resend-vs-duplicate distinction via a flag, not inferred from
bytes.
- Stall detection + missing-chunk report, for hops with no return channel.
- Testing convention: pure logic extracted to statically-testable methods.
What's **changed**: packaging moves from stock FlowFile Package V3 to the
custom manifest-trailer format below (needed for signing and
attribute-independent reads); staging/monitoring/cleanup responsibilities
move to an external service rather than NiFi timer-driven processors;
completion tracking is dual-written to a durable marker *and* a catalog
database instead of an S3-only marker object.
## Architecture
### Deployment Topology
The whole system described in this doc is referred to as **SA**: a single
system containing two chains of independent, single-function NiFi hosts
linked by Site-to-Site (per the existing horizontal-scaling model — no
clustering, just chained instances), bridged in the middle by **MR** — the
named one-way relay this design has been built around (FEC via an HTTP
service, ~1GB chunks, per [Purpose](#purpose)). Content flows in one
direction through SA: external S3/S2S in, through the first chain, across
MR, through the second chain, external S3/S2S out. The Replay DB branches
off **before** MR, from node 1 of the *first* chain — i.e. at the true
ingest point into SA, not after delivery.
```
+------------------------------------ SA -------------------------------------+
| |
| S3 --\ /-- S3 |
| S2S --+-->[1]--S2S-->[2]--S2S-->[3]--HTTP-->[MR]--HTTP-->[1]--S2S-->[2]--S2S-->[3]--+-- S2S |
| | |
| "copy" |
| v |
| +-----------+ |
| | Replay DB | |
| +-----------+ |
| / \ |
| v v |
| S3 (large files) XFS (small files) |
+------------------------------------------------------------------------------------+
```
Chunking is applied uniformly at ingest (below) specifically so the same
chunked representation transits **both** the intra-chain S2S hops **and**
the MR relay — there's no separate chunking scheme per transport.
Hashing happens at this same ingest point — node 1 of the first chain, the
true entry into SA — consistent with the single-pass design below.
### Per-FlowFile Pipeline
```
INGEST (custom NiFi processor;
replaces ListenHTTP for HTTP,
or first-in-flow processor
right after the S2S Input Port)
|
v
Single streaming pass: write content to NiFi's
content repo (required — all content must transit
NiFi for routing/content-assurance) WHILE computing
the content hash and emitting a chunk boundary every
~1GB. No pre-ingest size check or path branching —
chunk count falls out of content length; a small file
is just the one-chunk case.
|
v
Ordered attribute list (incl. content hash, chunk
descriptors) is clear-signed in-process (Bouncy
Castle OpenPGP, key via Controller Service — see
Signing) and stored as a new attribute.
|
v
FlowFile(s) proceed through normal NiFi routing /
content-assurance flow logic.
|
v
DELIVERY: per-chunk content streamed once (single
content-repo read) to destination staging.
|
v
+-----------------------------------------------+
| DESTINATIONS |
| S3: stage per-chunk objects, assemble via |
| UploadPartCopy (no re-transfer) |
| filesystem: pwrite chunks at offset, preallocated|
| HTTP: deliver assembled object to receiving app |
| diode: chunks -> FEC HTTP service (~1GB each) |
+-----------------------------------------------+
| |
write marker upsert row
(durable, (fast index,
per-dest) rebuildable
| from markers)
v v
+-----------------------------------------------+
| EXTERNAL SERVICE (post-NiFi only): |
| assembly (UploadPartCopy), stall monitoring, |
| cleanup, catalog DB maintenance |
+-----------------------------------------------+
|
v
+-----------------------------------------------+
| CATALOG / REPLAY DATABASE |
+-----------------------------------------------+
```
A prior iteration of this design considered a sidecar process fronting
ingestion (to bypass NiFi's content repository entirely for large
payloads, via a claim-check pattern). That's rejected: **all content must
go through NiFi** for routing and content-assurance controls, which removes
the main justification for a front door outside the JVM. The external
service's role is narrowed to post-NiFi work that never needs to see raw
content: assembly, monitoring, cleanup, and catalog indexing.
## Packaged Artifact Format
This is the format used for filesystem/XFS delivery always, and for S3
delivery when the attribute set is too large for S3 user object metadata
(see [Storage Backend Handover](#storage-backend-handover)). Content and
its manifest travel as **one object**, not two, for end-to-end
loss protection: two separate objects (content + manifest) have independent
failure/lifecycle/replication paths that can diverge — e.g. content lands
but the manifest write fails, or a lifecycle rule reaps one and not the
other. A single completed write (one `PutObject`, or one completed
multipart upload) can't end up in that split state.
```
PACKAGED OBJECT (single PutObject, or single completed multipart upload)
+---------------------------------------------------------+
| CONTENT (1 object, or N ~1GB parts via multipart) |
+---------------------------------------------------------+
| MANIFEST TRAILER (final part -- direct UploadPart, not |
| UploadPartCopy, so it can be arbitrarily small) |
| = JSON document, e.g.: |
| { "format_version": ..., |
| "clearsign": "-----BEGIN PGP SIGNED MESSAGE----- |
| ...ordered attribute list... |
| -----BEGIN PGP SIGNATURE-----... |
| -----END PGP SIGNATURE-----" } |
| The clearsign field is self-contained and human- |
| verifiable via `gpg --verify` independent of the JSON |
| wrapper around it. |
+---------------------------------------------------------+
| FOOTER (fixed size, e.g. last 12 bytes: magic + 8-byte |
| big-endian trailer length) -- self-describing, storage- |
| agnostic; works identically on S3, filesystem, and HTTP |
+---------------------------------------------------------+
Optional S3 object tag (attr-size = trailer byte length) as an
accelerant: lets the catalog DB or a reconciliation scan jump straight
to the trailer's byte range without reading the footer first.
```
Manifest is written **after** content by construction — the ingest pass is
single-streaming, so the final content hash isn't known until content is
fully read, ruling out a header-first format (which would require either
buffering to know its size upfront, or a second pass to fix it up).
**Attribute-only reads** (replay filtering by date/category, signature
verification, routing) use a suffix-range read — `Range: bytes=-N` where
`N` is the trailer size (from the catalog DB, the S3 tag, or the footer) —
which never touches the (possibly TB-scale) content bytes and completes in
one round trip when the DB already knows the offset.
For chunked files, S3 multipart uploads mix `UploadPartCopy` parts (the
staged content chunks) with one ordinary `UploadPart` (the manifest
trailer — no 5MB minimum applies to the last part), so `
CompleteMultipartUpload` produces one final object with the same
content-then-trailer-then-footer shape regardless of size.
## Signing
- **Format:** OpenPGP detached-style trailer via `gpg --clearsign`
semantics — clear-sign the **canonical ordered attribute list** (sorted
by attribute key, including the content hash and any chunk descriptors),
producing a self-contained ASCII-armored block (plaintext + embedded
signature). The result is stored as a new FlowFile attribute, then
becomes the `clearsign` field of the JSON manifest trailer at pack time
(see [Packaged Artifact Format](#packaged-artifact-format)) — the JSON
wrapper carries packaging metadata (format version, chunk descriptors)
alongside the signed block, without altering the signed bytes themselves.
- **Why clear-sign / OpenPGP over a bespoke signature scheme:** an operator
on the far side of the one-way diode boundary can verify an artifact by
hand with stock GnuPG and a public key — no custom SDK needed. This
matters because that boundary already has a manual-operator recovery
model (missing-chunk reports carried by hand, per Phat Files);
OpenPGP fits that operational reality directly.
- **Where signing runs:** in-process, via Bouncy Castle's OpenPGP API (not
shelling out to the `gpg` CLI per FlowFile — too slow at throughput). No
external signing service — this was evaluated and explicitly rejected in
favor of keeping the architecture self-contained within NiFi. The private
key is loaded via a Controller Service (same pattern as the existing
`AwsCredentialsProviderService`), plain in-JVM key material protected by
NiFi's sensitive-property encryption and OS-level file permissions.
- **Non-repudiation trade-off, stated explicitly:** non-repudiation depends
on the private key being exclusively controlled by one identity. Moving
signing in-process (vs. a network-isolated external service) means the
key now lives on the same host as ingestion, so custody depends on
host-level protections rather than network isolation. Mitigations: only
the NiFi instances actually responsible for packing/export have the
signing Controller Service configured (not every instance in the
deployment); NiFi's provenance repository gives an audit trail of which
flow/processor invoked signing and when, partially substituting for the
audit log a dedicated signing service would give for free. This is not a
one-way door — the Controller Service abstraction means a PKCS11/HSM
token could be swapped in later as a stronger key-custody option without
changing processor logic, if ever required.
- **Algorithm:** Ed25519 preferred for speed and small signature size at
throughput, unless a compliance requirement (e.g. FIPS mode) forces
RSA/ECDSA instead — not yet confirmed as a constraint.
- **Tamper-detection vs. non-repudiation:** a symmetric HMAC alternative
was considered and rejected. HMAC would detect unauthorized modification
(any party without the shared secret can't forge a valid tag) but cannot
prove authorship to a third party, since every verifier necessarily holds
the same secret and could in principle have produced it themselves. The
stated requirement is non-repudiation, which needs an asymmetric scheme.
## Storage Backend Handover
Final storage backend is chosen by size, separate from the ingest-time
chunking decision (which is uniform, no threshold — see Architecture):
large files land in S3, small files land on a local/shared XFS filesystem.
This split exists because S3's per-request overhead and cost make it a
poor fit for very high volumes of tiny objects; XFS is cheaper and faster
for that case. **Open question, not yet resolved:** if S3 can be shown to
serve small files efficiently enough (e.g. via batching, a different
bucket/storage class, or acceptable cost at the actual small-file volume
expected), the XFS tier could be dropped and S3 used uniformly — this
needs validating against real small-file throughput numbers before
implementation, see [Open Items](#open-items-for-implementation-planning).
The JSON-trailer packaged artifact format is shared by both backends, but
S3 additionally supports an existing metadata-based mechanism for small
attribute sets that avoids the trailer entirely — see below.
### S3 path
**Attribute handover on S3 follows an existing, established handover
interface specification already used elsewhere in this environment — this
design integrates with it rather than replacing it.** Under that spec:
- Attributes are written as S3 user object metadata (`x-amz-meta-*`
headers) when small enough to fit. A regular expression (defined by that
existing spec, not this one) identifies which user metadata keys get
turned into NiFi attributes on read — so consuming the attributes is a
single `HeadObject` call, no content touch, not even a suffix-range read.
- A flag (also part of that existing spec) records whether the attribute
set exceeded S3's ~2KB user-metadata limit. When it does, the object
falls back to **this design's** JSON manifest trailer (Packaged Artifact
Format) instead of metadata.
- The exact regex pattern and flag key/tag name live in the existing
handover interface spec document, not reproduced here — see
[Open Items](#open-items-for-implementation-planning).
So for S3 specifically, the JSON trailer is the **fallback path** (oversized
attribute sets), not the only mechanism — small-attribute objects never get
a trailer at all. Filesystem/XFS delivery, which has no equivalent native
metadata mechanism, always uses the JSON trailer.
```
NiFi (per chunk FlowFile, content already in repo from ingest)
|
| PutObject (single read of the content-repo claim --
v the only disk touch on the delivery side)
S3 staging: staging/<generation-id>/<chunk-index>
|
| (all N chunks staged; external service detects
v completeness via catalog DB or S3 listing)
UploadPartCopy x N -- server-side, zero bytes touch NiFi or the
| external service
v
Attributes fit in 2KB?
yes -> PutObjectMetadata / CompleteMultipartUpload with attributes
as x-amz-meta-* headers (no trailer)
no -> + UploadPart (JSON manifest trailer), oversized flag set
v
Final object at <key>
|
v
write durable completion marker + upsert catalog DB row
|
v
delete staging/<generation-id>/* (cleanup)
```
### XFS path (small files, below the size threshold)
No staging/assembly needed — small files are single-chunk by construction
(Architecture), so NiFi writes the packaged object (content + JSON trailer
+ footer) directly to its final path in one `write()` call. The external
service still writes the durable completion marker (a sibling file, or an
xattr, depending on implementation preference) and upserts the catalog DB
row, same as the S3 path, so replay and reconciliation behave identically
regardless of backend.
Chunk upload failures for **plain S3/filesystem/HTTP delivery** are
ordinary NiFi FlowFile retry — Phat Files' resend/stall-report machinery is
reserved for the actual one-way diode hop, where there's no return channel;
applying it to destinations that do have bidirectional connectivity would
be unnecessary complexity.
## Catalog / Replay Database
One row per packed artifact, written alongside the durable completion
marker (same write-marker-then-upsert-row ordering as Section
"Architecture"). Indexed for date/category replay queries: which backend
(S3 or XFS), object location (S3 key or filesystem path), the clear-signed attribute list
(so filtering never touches the object), completion status, and
generation/chunk-set id for large files.
The database is explicitly a **rebuildable index, not a source of truth**
— the durable marker plus the trailer/footer on disk or S3 is authoritative,
and a reconciliation job can rebuild the catalog by scanning objects and
reading trailers if the database is ever lost, corrupted, or migrated.
Replay is: query by date/category → get object locations → re-inject as
FlowFiles via the same ingest path used for first-time delivery.
## Error Handling
| Condition | Handling |
|---|---|
| Checksum mismatch on chunk arrival | Route to failure, never staged |
| Resend vs. unrequested duplicate | Explicit flag on the FlowFile, checked against the durable completion marker (adopted from Phat Files) |
| Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. |
| Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists |
| Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch |
| Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers |
## Testing
Following the convention established in Phat Files: pure logic extracted
to package-private static methods, unit-testable without a live NiFi/S3
connection —
- Canonical attribute-list ordering and serialization (must be
deterministic for clear-sign to produce consistent output for the same
attribute set).
- Chunk-boundary computation from a content stream (byte offset math, ~1GB
target size).
- Footer encode/decode (magic + length) and suffix-range calculation.
- Missing-chunk diffing (expected range vs. staged contents) — adopted
from Phat Files.
- Resend-vs-duplicate decision logic — adopted from Phat Files.
End-to-end behavior (ingest → sign → chunk → stage → assemble → catalog
row → attribute-only read via suffix range → replay) verified against the
lab's MinIO instance, consistent with prior processors in this
environment.
## Open Items for Implementation Planning
- Exact NAR/module layout for the new ingest processor(s), Controller
Service (signing key), and the external post-NiFi service.
- Concrete canonical attribute-list serialization format (sorted
`key: value` lines vs. canonical JSON) — functionally equivalent, pick
one for implementation.
- Confirm whether Ed25519 is acceptable or a compliance constraint (e.g.
FIPS) forces RSA/ECDSA.
- Concrete default values: chunk size (default ~1GB, confirm whether it
should differ between diode-bound and non-diode-bound chunked delivery),
stall timeout, cleanup retention window.
- Catalog database technology selection (not yet specified).
- Whether HTTP delivery to receiving applications needs the same suffix-
range attribute-only read support, or always transfers the full object.
- **S3-vs-XFS size threshold.** Needs validating against real small-file
volume/throughput numbers — if S3 can serve small files efficiently
enough at the actual expected volume, the XFS tier could be dropped
entirely and S3 used as the sole backend, simplifying Storage Backend
Handover to one path instead of two.
- Exact JSON trailer schema beyond the `clearsign` field (format version,
chunk descriptor list — whether chunk descriptors are duplicated at the
JSON top level for easy programmatic access, or only present inside the
signed attribute list and require parsing the clearsign block to read).
- **Get the existing S3 handover interface spec document** referenced in
Storage Backend Handover: the exact regex used to map `x-amz-meta-*`
keys to NiFi attributes, and the exact flag/tag name and value convention
used to signal the >2KB fallback to the JSON trailer. This design assumes
that spec's behavior but doesn't reproduce its details.
grand_plan.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.2 MiB