commit 4008f61ce4222b542cc831a673a9a8e26106f16d Author: wall Date: Tue Jul 28 14:17:11 2026 +1000 Add Phat Files and FlowFile Packaging & Signing design specs Initial commit of NiFi prototyping design docs: the existing Phat Files large-file transfer spec, the new packaging/signing/chunking design, and the whiteboard sketch (grand_plan.jpg) it was reconciled against. diff --git a/2026-07-23-phat-files-design.md b/2026-07-23-phat-files-design.md new file mode 100755 index 0000000..adcffbe --- /dev/null +++ b/2026-07-23-phat-files-design.md @@ -0,0 +1,328 @@ +# Phat Files — Design Spec + +**Date:** 2026-07-23 +**Status:** Approved for implementation planning + +## Purpose + +Move very large (TB-scale) files from a low-side S3-compatible bucket to a +high-side S3-compatible bucket across a one-way boundary, with no automated +return channel. Files are split into fixed-size blocks so that transfers can +be resumed by resending only specific missing blocks, without ever needing to +restart or re-transmit a whole file. + +The transport that actually carries a block from low side to high side is +intentionally out of scope for this spec — it could be a network relay +(NiFi Site-to-Site, HTTP push), physical removable media, or any other +one-way carriage mechanism. The design only assumes it's one-way. Packaging +a FlowFile (content + attributes) into something portable enough to survive +that transport, and reconstructing it on the other side, is handled by +NiFi's standard pack/unpack FlowFile processors (e.g. `MergeContent` using +the FlowFile Stream packaging format, paired with `UnpackContent` on the +receiving side) — not by anything custom in this project. `FetchS3ObjectBlock` +and `PutS3ObjectBlock` only ever deal with ordinary FlowFiles; whatever sits +between them for the boundary crossing is someone else's stock processor. + +This is a cross-domain-style transfer: data flows low → high only. Any +recovery signal (which blocks are missing) must be produced on the high side +and manually carried back down to the low side by an operator — there is no +network path from high side back to low side. + +## Non-goals + +- Content-defined/delta chunking for re-syncing *changed* files. Only + fixed-size blocks and resend-of-missing-blocks are in scope. +- Automated high-to-low signaling of any kind (by design constraint, not an + oversight). +- Bidirectional transfer (high → low) — this spec covers one direction only. + A mirror-image flow could reuse the same processors in principle, but is + out of scope here. + +## Architecture + +``` +[Low-side S3] <--Range GET--> [Low-side NiFi] --one-way transport--> [High-side NiFi] --PutObject/UploadPartCopy--> [High-side S3] + ^ (mechanism deferred: | + | network relay, media, etc.) | + resend blocks via missing-block report + processor properties (JSON, written by + (static, EL/attribute, MonitorS3StagingStalls) + or file-driven) | + | | + +----------------- operator manually carries report file down -------------------------+ +``` + +- **Low-side NiFi**: reads byte ranges of a file from low-side S3 via ranged + `GetObject` (streaming read, never buffers a whole file), splits into + fixed-size blocks, and hands each block FlowFile to a stock pack/unpack + pair (e.g. `MergeContent` + `UnpackContent`) that carries it one-way + across whatever transport is chosen for the boundary and reconstructs it + as a FlowFile again on the high side. +- **High-side NiFi**: receives blocks, stages each as its own object in + high-side S3 (streaming `PutObject`), and once a file's full block set is + present, assembles the real object server-side via `UploadPartCopy` (no + bytes re-transferred through NiFi) and cleans up staging objects. +- **No return channel.** Recovery is manual: the high side detects missing or + stalled blocks and writes a JSON report; an operator carries that report + down to the low side (per their existing cross-domain process) to drive a + targeted resend. + +## Components + +### Low side: `FetchS3ObjectBlock` (custom processor) + +Reads a byte range of a source S3 object and emits it as a block FlowFile. + +**Properties** (all Expression-Language-enabled against FlowFile attributes, +so each can be set statically, via `${attribute}`, or driven by an upstream +flow that parses a carried-over report file): + +| Property | Description | +|---|---| +| Bucket | Source S3 bucket | +| Object Key | Source S3 object key | +| Block Size | Block size in bytes (configurable; determines byte-range math) | +| Block Numbers | `all`, or an explicit list/ranges, e.g. `0-4,7,10-12` | +| Resend | Boolean, default `false`. Marks the emitted blocks as a deliberate resend rather than an initial/routine transfer — see [Resend vs. duplicate](#resend-vs-duplicate) below. | + +**Behavior on trigger:** +1. `HeadObject` to get file size and source ETag. +2. Compute total block count = `ceil(fileSize / blockSize)`. +3. For each requested block index, compute byte range + `[index * blockSize, min((index + 1) * blockSize, fileSize))`. +4. Issue a ranged `GetObject` per requested block (streaming, not buffered). +5. Emit one FlowFile per block with attributes: + - `phat.object.key` + - `phat.source.etag` + - `phat.block.index` + - `phat.block.total` + - `phat.block.size` + - `phat.block.checksum` (SHA-256 of the block's content) + - `phat.resend` (`true`/`false`, copied from the Resend property) + +That's it — `FetchS3ObjectBlock` only produces ordinary FlowFiles (content + +attributes). Getting them across the boundary intact is the pack/unpack +pair's job, not this processor's (see Architecture above). + +### Low side: report-driven resend (stock processors) + +A small flow segment, built from stock NiFi processors, watches a drop +location for a carried-over missing-block report file: + +1. `GetFile`/`FetchFile` picks up the report JSON at the watched path. +2. `EvaluateJsonPath` extracts `objectKey`, `bucket`, `sourceEtag`, + `blockSize`, and `missingBlocks` into FlowFile attributes, and sets + `resend = true` unconditionally — a report-driven trigger is by + definition always a resend, never an initial transfer. +3. Those attributes feed `FetchS3ObjectBlock`'s EL-bound properties + (including its `Resend` property), so the correct blocks are resent + automatically — no manual property editing required for the common case. + Manual override (typing values directly into `FetchS3ObjectBlock`'s + properties, including setting `Resend` to `true`) remains available for + ad hoc use. + +### High side: `PutS3ObjectBlock` (custom processor) + +Receives a block FlowFile and stages it. + +**Behavior:** +1. Verify `phat.block.checksum` against the received content. On mismatch, + route to `failure` — never stage corrupted data. +2. If `phat.resend` is **not** `true`: check whether a completion marker + exists at `_phat/completed//` (see + [Completion markers](#completion-markers) below). If it does, this block + is an unrequested duplicate of an already-completed transfer of this + exact file version — discard immediately, do not stage. (See + [Resend vs. duplicate](#resend-vs-duplicate) below for why this check is + skipped when `phat.resend` is `true`.) +3. Otherwise, stream the block to + `staging///` via `PutObject`, and flag that + `/` for re-assembly regardless of whether it was previously + marked complete. + +Staging by block index makes out-of-order arrival a non-issue (each block +writes to its own path regardless of arrival order) and makes duplicates +within the same in-progress transfer idempotent (re-writing the same path is +harmless). + +#### Resend vs. duplicate + +Two situations look identical at the byte level — a block for a `/` +whose transfer has already been marked complete (see +[Completion markers](#completion-markers) below) — but need opposite handling: + +- **Unrequested duplicate**: the transport (or an operator) accidentally + re-sent a full, unflagged transfer after it already completed — e.g. a + retried batch job, or a stray retransmission at the transport layer. This + carries no new information and should be silently absorbed. +- **Requested resend**: an operator deliberately re-sent specific blocks — + either because a missing-block report said so, or because they have + independent reason to distrust the already-assembled object (e.g. + suspected corruption) — and wants it honored even if the destination + believes the transfer already completed. + +The two are distinguished purely by intent, which only a human (or the +report-parsing flow acting on the human's behalf) can supply — nothing about +the bytes themselves says which one it is. That's what the `Resend` property +on `FetchS3ObjectBlock` is for: it's `false` by default (routine/initial +transfers), and the report-driven resend flow and manual ad hoc resends both +set it `true`. `PutS3ObjectBlock` trusts that flag: `phat.resend=true` always +stages the block and re-triggers `AssembleS3ObjectBlocks` for that +`/`, even over a previously-completed one; `phat.resend=false` (or +absent) is subject to the duplicate-of-completed short-circuit. + +This means a resend can correct or re-confirm an already-assembled object, +while a plain accidental replay of a finished transfer costs nothing. + +#### Completion markers + +Duplicate detection cannot rely on the final assembled object itself, +because the whole point of landing it at `` is that something +downstream collects it — and typically moves or deletes it afterward. If +"transfer complete" were only recorded as a tag on that object, the record +would vanish the moment it's collected, and a late unrequested duplicate +block arriving after that point would find nothing to match against and get +staged as if it were a new transfer. + +So completion is recorded independently: once `AssembleS3ObjectBlocks` +successfully assembles `` for a given ``, it also writes a small, +separate marker object at `_phat/completed//` (empty or a tiny +JSON blob — content doesn't matter, only presence). This marker is never +touched by whatever collects the real deliverable, so duplicate detection in +`PutS3ObjectBlock` keeps working regardless of what happens to the object at +`` afterward. + +Markers are small (one per completed file *generation*, not per block), but +unlike staging blocks they are not deleted as part of normal completion — +left alone, they'd accumulate indefinitely. `CleanupCompletionMarkers` (see +below) TTL-expires them on a separate, typically much longer, retention +window than staging cleanup, since the tradeoff here is audit/duplicate- +protection value vs. storage, not "is this transfer abandoned." + +### High side: `AssembleS3ObjectBlocks` (custom processor) + +Triggered after each block lands (or on a schedule), including re-triggers +from a resend against an already-completed `/` (see +[Resend vs. duplicate](#resend-vs-duplicate)). For the block's +`/`, lists the staging prefix and compares the object count +against `phat.block.total`. If complete: +1. Assemble (or re-assemble) the final object at `` via server-side + `UploadPartCopy` from the staged blocks (no bytes move back through + NiFi) — this overwrites any previously-assembled object for the same + `/`, which is the intended effect of a resend. +2. Tag the final object with `x-phat-source-etag` = the source ETag it was + built from. +3. Write (or refresh the timestamp on) the completion marker at + `_phat/completed//` — see [Completion markers](#completion-markers). +4. Delete the staging prefix for that `/`. + +### High side: `MonitorS3StagingStalls` (custom processor) + +Timer-driven. For each incomplete `/` staging prefix, checks the +write time of the most recently staged block. If older than the configurable +**Stall Timeout**, raises an event (NiFi bulletin/log/FlowFile) and writes a +JSON missing-block report: + +```json +{ + "objectKey": "...", + "bucket": "...", + "sourceEtag": "...", + "blockSize": 104857600, + "missingBlocks": [3, 17, 42] +} +``` + +Missing block indices are computed directly by diffing the expected range +`0..total-1` against what's present in the staging prefix — no separate +tracking state needed. + +**Constraint:** Stall Timeout must be configured shorter than the Cleanup +Retention Window below, so operators get a chance to notice and request a +resend before the data they've already received is reaped. + +### High side: `CleanupStagingBlocks` (custom processor) + +Timer-driven housekeeping, independent of alerting. Reaps any `/` +staging prefix whose most recent write is older than the configurable +**Cleanup Retention Window**, so abandoned partial transfers don't grow +storage unbounded. + +### High side: `CleanupCompletionMarkers` (custom processor) + +Timer-driven housekeeping for [completion markers](#completion-markers). +Deletes any `_phat/completed//` marker older than the +configurable **Completion Marker Retention** window. This is a separate, +independently-configured window from the Cleanup Retention Window above — +staging cleanup is about not hoarding abandoned partial transfers (short +timescale), while marker retention is about how long duplicate-of-completed +protection stays effective after a transfer succeeds (can be a much longer +timescale, since markers are cheap). Once a marker expires, a genuinely late +unrequested duplicate for that generation would no longer be recognized as +one and would be staged — an accepted tradeoff of choosing a finite +retention window over "keep forever." + +## Data Flow Summary + +| Stage | Location | Mechanism | +|---|---|---| +| Read | Low-side S3 | Ranged `GetObject` (streaming) | +| Transport | Low → High | One-way, mechanism deferred (network relay, removable media, etc.) | +| Stage | High-side S3 | `PutObject` to `staging///` | +| Assemble | High-side S3 | Server-side `UploadPartCopy`, no re-transfer | +| Completion record | High-side S3 | Marker object at `_phat/completed//`, independent of the deliverable's lifecycle | +| Recovery signal | High side → operator → Low side | JSON report, manually carried | + +## Error Handling + +| Condition | Handling | +|---|---| +| Checksum mismatch on arrival | Route to `failure`, never staged | +| Unrequested duplicate, transfer already completed (`phat.resend=false`) | Detected via presence of `_phat/completed//` marker; discarded silently | +| Requested resend, transfer already completed (`phat.resend=true`) | Always staged; re-triggers `AssembleS3ObjectBlocks`, overwriting the previously-assembled object and refreshing its marker | +| Duplicate block, transfer in progress | Idempotent overwrite of the same staging path | +| Completion marker expired (beyond Completion Marker Retention) | Duplicate protection no longer applies for that generation; a late duplicate would be staged as if new (accepted tradeoff) | +| Out-of-order block arrival | No-op by construction (staging keyed by block index) | +| Transfer stalled (blocks stopped arriving) | `MonitorS3StagingStalls` raises an alert + JSON report with exact missing block indices, before the Cleanup Retention Window would reap it | +| Transfer abandoned beyond retention window | `CleanupStagingBlocks` reaps the staging prefix | + +## Testing + +Following this project family's existing convention (see +`nifi-custom-processors`'s `processor-design.md`): pure logic is extracted +into package-private static methods so it can be unit-tested without a live +NiFi/S3 connection: + +- Byte-range computation from block size / index / file size +- Block-number list/range parsing (`all`, `0-4,7,10-12`) +- Staging path construction and completion-marker path construction +- Missing-block diffing (expected range vs. staged contents) +- Resend-vs-duplicate decision logic (`phat.resend` combined with marker + presence/absence) + +End-to-end behavior (fetch → stage → assemble → marker write → cleanup → +stall alert → marker expiry) is verified against the lab's MinIO instance, +consistent with how prior processors in this environment were tested. Also +worth an explicit test: assemble a transfer, delete/move the final object +(simulating downstream collection) exactly as a real consumer would, then +confirm an unrequested duplicate block is still correctly discarded and a +flagged resend still correctly re-assembles. + +## Open Items for Implementation Planning + +- Exact NAR/module layout (likely mirrors `nifi-custom-processors`: separate + low-side and high-side processor modules, or one bundle with both, deployed + selectively per NiFi instance). +- Controller service reuse for AWS credentials (existing + `AwsCredentialsProviderService` pattern from `nifi-custom-processors` + should carry over directly). +- Concrete default values for Block Size, Stall Timeout, Cleanup Retention + Window, and Completion Marker Retention. +- **Transport mechanism selection.** Deliberately deferred (see Purpose). + Whatever is chosen — NiFi Site-to-Site, HTTP push, or physical media — + must be genuinely one-way (note that standard S2S involves a bidirectional + peer/site-info handshake even when FlowFile data only moves one direction, + so it may not qualify as-is if zero bidirectional connectivity is + required). Packaging/unpackaging the FlowFile for that transport is + handled by stock NiFi processors (e.g. `MergeContent` + `UnpackContent`), + not by anything custom in this project. diff --git a/docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md b/docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md new file mode 100644 index 0000000..ed85e9f --- /dev/null +++ b/docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md @@ -0,0 +1,451 @@ +# FlowFile Packaging & Signing — Design Spec + +**Date:** 2026-07-28 +**Status:** Draft — pending user review + +## Purpose + +NiFi is used to manage complex workflows with payloads ranging from tiny +files to TB scale. Two related problems need solving: + +1. **Performance.** The current pack/unpack path (`MergeContent` + + `UnpackContent`) is disk-I/O bound under high throughput. Content lives on + vSAN storage, which is shared, virtualized IOPS across the vSphere + cluster — so the pain isn't any single slow disk, it's *contention* + across every VM sharing that storage. The current flow makes multiple + redundant passes over content on disk: an ingest write, a separate read + to compute a content hash, a read+write to merge/pack, and a read+write + to unpack. Each pass competes for the same shared IOPS pool. NiFi runs as + independent single-node instances (no clustering), each with its own + logical function, scaled horizontally by adding more instances — so + there's no cluster-wide load balancing to fall back on. + +2. **Integrity & non-repudiation.** When a FlowFile (content + attributes) + is packed and exported outside NiFi's space, its attributes — which + already include a hash of the content, so content hashing is not part of + packing — need to be signed to prove they haven't been tampered with + and to prove who produced them, in a way that holds up even to a party + who didn't produce the artifact. Uses for the packed form: storage/ + retrieval to/from S3, delivery to other applications via HTTP or + filesystem, and selective (date/category) replay from a catalog + database. + +A **future** requirement extends this to TB-scale files that must be +broken into pieces, tracked, transited through NiFi (for routing and +content-assurance controls — all content must pass through NiFi), and +reconstructed at the destination — including across a **one-way network +boundary** — referred to throughout this doc as **MR**, the named relay +bridging the two NiFi host chains within the overall system referred to as +**SA** (see [Deployment Topology](#deployment-topology)) — with no return +channel for retransmission requests. Forward error correction for that hop +is assumed to be available via an existing HTTP service, with ~1GB as the +target chunk size fed to it. + +## Non-goals + +- Redesigning NiFi's clustering model. Instances remain independent, + single-node, horizontally scaled by function. +- Automated recovery signaling across the true one-way diode boundary — by + construction there is no return channel; recovery there is necessarily + operator-mediated (see [Error Handling](#error-handling)). +- Building or operating the forward-error-correction transport itself — + assumed to already exist as an HTTP service outside this design's scope. +- A general-purpose NiFi improvement library. This design covers exactly the + packaging, chunking, signing, delivery, and replay-indexing concerns + described above. + +## Relationship to the "Phat Files" design + +A separate, already-approved spec +(`2026-07-23-phat-files-design.md`) covers low-side-S3-to-high-side-S3 +transfer of TB-scale files across a one-way boundary: fixed-size blocks, +resend-vs-duplicate handling, staging by block index, server-side assembly +via `UploadPartCopy`, stall monitoring, and cleanup. That spec is treated +here as a **source of proven concepts, not a fixed dependency** — this +design is not an extension of it and does not modify it. The following +ideas are adopted (generalized beyond S3-to-S3): + +- Streaming byte-range reads; never buffer a whole file. +- Chunk descriptor attributes (index, total, size, checksum). +- Server-side assembly via `UploadPartCopy` from independently-staged + per-chunk objects, rather than a live multipart-upload session spanning + the transport. +- Explicit resend-vs-duplicate distinction via a flag, not inferred from + bytes. +- Stall detection + missing-chunk report, for hops with no return channel. +- Testing convention: pure logic extracted to statically-testable methods. + +What's **changed**: packaging moves from stock FlowFile Package V3 to the +custom manifest-trailer format below (needed for signing and +attribute-independent reads); staging/monitoring/cleanup responsibilities +move to an external service rather than NiFi timer-driven processors; +completion tracking is dual-written to a durable marker *and* a catalog +database instead of an S3-only marker object. + +## Architecture + +### Deployment Topology + +The whole system described in this doc is referred to as **SA**: a single +system containing two chains of independent, single-function NiFi hosts +linked by Site-to-Site (per the existing horizontal-scaling model — no +clustering, just chained instances), bridged in the middle by **MR** — the +named one-way relay this design has been built around (FEC via an HTTP +service, ~1GB chunks, per [Purpose](#purpose)). Content flows in one +direction through SA: external S3/S2S in, through the first chain, across +MR, through the second chain, external S3/S2S out. The Replay DB branches +off **before** MR, from node 1 of the *first* chain — i.e. at the true +ingest point into SA, not after delivery. + +``` + +------------------------------------ SA -------------------------------------+ + | | + | S3 --\ /-- S3 | + | S2S --+-->[1]--S2S-->[2]--S2S-->[3]--HTTP-->[MR]--HTTP-->[1]--S2S-->[2]--S2S-->[3]--+-- S2S | + | | | + | "copy" | + | v | + | +-----------+ | + | | Replay DB | | + | +-----------+ | + | / \ | + | v v | + | S3 (large files) XFS (small files) | + +------------------------------------------------------------------------------------+ +``` + +Chunking is applied uniformly at ingest (below) specifically so the same +chunked representation transits **both** the intra-chain S2S hops **and** +the MR relay — there's no separate chunking scheme per transport. +Hashing happens at this same ingest point — node 1 of the first chain, the +true entry into SA — consistent with the single-pass design below. + +### Per-FlowFile Pipeline + +``` + INGEST (custom NiFi processor; + replaces ListenHTTP for HTTP, + or first-in-flow processor + right after the S2S Input Port) + | + v + Single streaming pass: write content to NiFi's + content repo (required — all content must transit + NiFi for routing/content-assurance) WHILE computing + the content hash and emitting a chunk boundary every + ~1GB. No pre-ingest size check or path branching — + chunk count falls out of content length; a small file + is just the one-chunk case. + | + v + Ordered attribute list (incl. content hash, chunk + descriptors) is clear-signed in-process (Bouncy + Castle OpenPGP, key via Controller Service — see + Signing) and stored as a new attribute. + | + v + FlowFile(s) proceed through normal NiFi routing / + content-assurance flow logic. + | + v + DELIVERY: per-chunk content streamed once (single + content-repo read) to destination staging. + | + v + +-----------------------------------------------+ + | DESTINATIONS | + | S3: stage per-chunk objects, assemble via | + | UploadPartCopy (no re-transfer) | + | filesystem: pwrite chunks at offset, preallocated| + | HTTP: deliver assembled object to receiving app | + | diode: chunks -> FEC HTTP service (~1GB each) | + +-----------------------------------------------+ + | | + write marker upsert row + (durable, (fast index, + per-dest) rebuildable + | from markers) + v v + +-----------------------------------------------+ + | EXTERNAL SERVICE (post-NiFi only): | + | assembly (UploadPartCopy), stall monitoring, | + | cleanup, catalog DB maintenance | + +-----------------------------------------------+ + | + v + +-----------------------------------------------+ + | CATALOG / REPLAY DATABASE | + +-----------------------------------------------+ +``` + +A prior iteration of this design considered a sidecar process fronting +ingestion (to bypass NiFi's content repository entirely for large +payloads, via a claim-check pattern). That's rejected: **all content must +go through NiFi** for routing and content-assurance controls, which removes +the main justification for a front door outside the JVM. The external +service's role is narrowed to post-NiFi work that never needs to see raw +content: assembly, monitoring, cleanup, and catalog indexing. + +## Packaged Artifact Format + +This is the format used for filesystem/XFS delivery always, and for S3 +delivery when the attribute set is too large for S3 user object metadata +(see [Storage Backend Handover](#storage-backend-handover)). Content and +its manifest travel as **one object**, not two, for end-to-end +loss protection: two separate objects (content + manifest) have independent +failure/lifecycle/replication paths that can diverge — e.g. content lands +but the manifest write fails, or a lifecycle rule reaps one and not the +other. A single completed write (one `PutObject`, or one completed +multipart upload) can't end up in that split state. + +``` + PACKAGED OBJECT (single PutObject, or single completed multipart upload) + +---------------------------------------------------------+ + | CONTENT (1 object, or N ~1GB parts via multipart) | + +---------------------------------------------------------+ + | MANIFEST TRAILER (final part -- direct UploadPart, not | + | UploadPartCopy, so it can be arbitrarily small) | + | = JSON document, e.g.: | + | { "format_version": ..., | + | "clearsign": "-----BEGIN PGP SIGNED MESSAGE----- | + | ...ordered attribute list... | + | -----BEGIN PGP SIGNATURE-----... | + | -----END PGP SIGNATURE-----" } | + | The clearsign field is self-contained and human- | + | verifiable via `gpg --verify` independent of the JSON | + | wrapper around it. | + +---------------------------------------------------------+ + | FOOTER (fixed size, e.g. last 12 bytes: magic + 8-byte | + | big-endian trailer length) -- self-describing, storage- | + | agnostic; works identically on S3, filesystem, and HTTP | + +---------------------------------------------------------+ + + Optional S3 object tag (attr-size = trailer byte length) as an + accelerant: lets the catalog DB or a reconciliation scan jump straight + to the trailer's byte range without reading the footer first. +``` + +Manifest is written **after** content by construction — the ingest pass is +single-streaming, so the final content hash isn't known until content is +fully read, ruling out a header-first format (which would require either +buffering to know its size upfront, or a second pass to fix it up). + +**Attribute-only reads** (replay filtering by date/category, signature +verification, routing) use a suffix-range read — `Range: bytes=-N` where +`N` is the trailer size (from the catalog DB, the S3 tag, or the footer) — +which never touches the (possibly TB-scale) content bytes and completes in +one round trip when the DB already knows the offset. + +For chunked files, S3 multipart uploads mix `UploadPartCopy` parts (the +staged content chunks) with one ordinary `UploadPart` (the manifest +trailer — no 5MB minimum applies to the last part), so ` +CompleteMultipartUpload` produces one final object with the same +content-then-trailer-then-footer shape regardless of size. + +## Signing + +- **Format:** OpenPGP detached-style trailer via `gpg --clearsign` + semantics — clear-sign the **canonical ordered attribute list** (sorted + by attribute key, including the content hash and any chunk descriptors), + producing a self-contained ASCII-armored block (plaintext + embedded + signature). The result is stored as a new FlowFile attribute, then + becomes the `clearsign` field of the JSON manifest trailer at pack time + (see [Packaged Artifact Format](#packaged-artifact-format)) — the JSON + wrapper carries packaging metadata (format version, chunk descriptors) + alongside the signed block, without altering the signed bytes themselves. +- **Why clear-sign / OpenPGP over a bespoke signature scheme:** an operator + on the far side of the one-way diode boundary can verify an artifact by + hand with stock GnuPG and a public key — no custom SDK needed. This + matters because that boundary already has a manual-operator recovery + model (missing-chunk reports carried by hand, per Phat Files); + OpenPGP fits that operational reality directly. +- **Where signing runs:** in-process, via Bouncy Castle's OpenPGP API (not + shelling out to the `gpg` CLI per FlowFile — too slow at throughput). No + external signing service — this was evaluated and explicitly rejected in + favor of keeping the architecture self-contained within NiFi. The private + key is loaded via a Controller Service (same pattern as the existing + `AwsCredentialsProviderService`), plain in-JVM key material protected by + NiFi's sensitive-property encryption and OS-level file permissions. +- **Non-repudiation trade-off, stated explicitly:** non-repudiation depends + on the private key being exclusively controlled by one identity. Moving + signing in-process (vs. a network-isolated external service) means the + key now lives on the same host as ingestion, so custody depends on + host-level protections rather than network isolation. Mitigations: only + the NiFi instances actually responsible for packing/export have the + signing Controller Service configured (not every instance in the + deployment); NiFi's provenance repository gives an audit trail of which + flow/processor invoked signing and when, partially substituting for the + audit log a dedicated signing service would give for free. This is not a + one-way door — the Controller Service abstraction means a PKCS11/HSM + token could be swapped in later as a stronger key-custody option without + changing processor logic, if ever required. +- **Algorithm:** Ed25519 preferred for speed and small signature size at + throughput, unless a compliance requirement (e.g. FIPS mode) forces + RSA/ECDSA instead — not yet confirmed as a constraint. +- **Tamper-detection vs. non-repudiation:** a symmetric HMAC alternative + was considered and rejected. HMAC would detect unauthorized modification + (any party without the shared secret can't forge a valid tag) but cannot + prove authorship to a third party, since every verifier necessarily holds + the same secret and could in principle have produced it themselves. The + stated requirement is non-repudiation, which needs an asymmetric scheme. + +## Storage Backend Handover + +Final storage backend is chosen by size, separate from the ingest-time +chunking decision (which is uniform, no threshold — see Architecture): +large files land in S3, small files land on a local/shared XFS filesystem. +This split exists because S3's per-request overhead and cost make it a +poor fit for very high volumes of tiny objects; XFS is cheaper and faster +for that case. **Open question, not yet resolved:** if S3 can be shown to +serve small files efficiently enough (e.g. via batching, a different +bucket/storage class, or acceptable cost at the actual small-file volume +expected), the XFS tier could be dropped and S3 used uniformly — this +needs validating against real small-file throughput numbers before +implementation, see [Open Items](#open-items-for-implementation-planning). +The JSON-trailer packaged artifact format is shared by both backends, but +S3 additionally supports an existing metadata-based mechanism for small +attribute sets that avoids the trailer entirely — see below. + +### S3 path + +**Attribute handover on S3 follows an existing, established handover +interface specification already used elsewhere in this environment — this +design integrates with it rather than replacing it.** Under that spec: + +- Attributes are written as S3 user object metadata (`x-amz-meta-*` + headers) when small enough to fit. A regular expression (defined by that + existing spec, not this one) identifies which user metadata keys get + turned into NiFi attributes on read — so consuming the attributes is a + single `HeadObject` call, no content touch, not even a suffix-range read. +- A flag (also part of that existing spec) records whether the attribute + set exceeded S3's ~2KB user-metadata limit. When it does, the object + falls back to **this design's** JSON manifest trailer (Packaged Artifact + Format) instead of metadata. +- The exact regex pattern and flag key/tag name live in the existing + handover interface spec document, not reproduced here — see + [Open Items](#open-items-for-implementation-planning). + +So for S3 specifically, the JSON trailer is the **fallback path** (oversized +attribute sets), not the only mechanism — small-attribute objects never get +a trailer at all. Filesystem/XFS delivery, which has no equivalent native +metadata mechanism, always uses the JSON trailer. + +``` + NiFi (per chunk FlowFile, content already in repo from ingest) + | + | PutObject (single read of the content-repo claim -- + v the only disk touch on the delivery side) + S3 staging: staging// + | + | (all N chunks staged; external service detects + v completeness via catalog DB or S3 listing) + UploadPartCopy x N -- server-side, zero bytes touch NiFi or the + | external service + v + Attributes fit in 2KB? + yes -> PutObjectMetadata / CompleteMultipartUpload with attributes + as x-amz-meta-* headers (no trailer) + no -> + UploadPart (JSON manifest trailer), oversized flag set + v + Final object at + | + v + write durable completion marker + upsert catalog DB row + | + v + delete staging//* (cleanup) +``` + +### XFS path (small files, below the size threshold) + +No staging/assembly needed — small files are single-chunk by construction +(Architecture), so NiFi writes the packaged object (content + JSON trailer ++ footer) directly to its final path in one `write()` call. The external +service still writes the durable completion marker (a sibling file, or an +xattr, depending on implementation preference) and upserts the catalog DB +row, same as the S3 path, so replay and reconciliation behave identically +regardless of backend. + +Chunk upload failures for **plain S3/filesystem/HTTP delivery** are +ordinary NiFi FlowFile retry — Phat Files' resend/stall-report machinery is +reserved for the actual one-way diode hop, where there's no return channel; +applying it to destinations that do have bidirectional connectivity would +be unnecessary complexity. + +## Catalog / Replay Database + +One row per packed artifact, written alongside the durable completion +marker (same write-marker-then-upsert-row ordering as Section +"Architecture"). Indexed for date/category replay queries: which backend +(S3 or XFS), object location (S3 key or filesystem path), the clear-signed attribute list +(so filtering never touches the object), completion status, and +generation/chunk-set id for large files. + +The database is explicitly a **rebuildable index, not a source of truth** +— the durable marker plus the trailer/footer on disk or S3 is authoritative, +and a reconciliation job can rebuild the catalog by scanning objects and +reading trailers if the database is ever lost, corrupted, or migrated. +Replay is: query by date/category → get object locations → re-inject as +FlowFiles via the same ingest path used for first-time delivery. + +## Error Handling + +| Condition | Handling | +|---|---| +| Checksum mismatch on chunk arrival | Route to failure, never staged | +| Resend vs. unrequested duplicate | Explicit flag on the FlowFile, checked against the durable completion marker (adopted from Phat Files) | +| Diode hop: chunks stop arriving | No return channel by design; stall detection produces a missing-chunk report an operator carries back manually (adopted from Phat Files). Loss tolerance in transit is handled by the external FEC HTTP service, out of scope here. | +| Non-diode destination delivery failure (S3/filesystem/HTTP) | Ordinary NiFi FlowFile retry — no resend/duplicate machinery needed, a return channel exists | +| Signature or trailer corruption on read | Rejected outright — never treated as a valid artifact for replay or delivery, same handling as a checksum mismatch | +| Catalog DB unavailable or inconsistent | Non-blocking for delivery (DB is an index, not authoritative); reconciliation job rebuilds it from durable markers/trailers | + +## Testing + +Following the convention established in Phat Files: pure logic extracted +to package-private static methods, unit-testable without a live NiFi/S3 +connection — + +- Canonical attribute-list ordering and serialization (must be + deterministic for clear-sign to produce consistent output for the same + attribute set). +- Chunk-boundary computation from a content stream (byte offset math, ~1GB + target size). +- Footer encode/decode (magic + length) and suffix-range calculation. +- Missing-chunk diffing (expected range vs. staged contents) — adopted + from Phat Files. +- Resend-vs-duplicate decision logic — adopted from Phat Files. + +End-to-end behavior (ingest → sign → chunk → stage → assemble → catalog +row → attribute-only read via suffix range → replay) verified against the +lab's MinIO instance, consistent with prior processors in this +environment. + +## Open Items for Implementation Planning + +- Exact NAR/module layout for the new ingest processor(s), Controller + Service (signing key), and the external post-NiFi service. +- Concrete canonical attribute-list serialization format (sorted + `key: value` lines vs. canonical JSON) — functionally equivalent, pick + one for implementation. +- Confirm whether Ed25519 is acceptable or a compliance constraint (e.g. + FIPS) forces RSA/ECDSA. +- Concrete default values: chunk size (default ~1GB, confirm whether it + should differ between diode-bound and non-diode-bound chunked delivery), + stall timeout, cleanup retention window. +- Catalog database technology selection (not yet specified). +- Whether HTTP delivery to receiving applications needs the same suffix- + range attribute-only read support, or always transfers the full object. +- **S3-vs-XFS size threshold.** Needs validating against real small-file + volume/throughput numbers — if S3 can serve small files efficiently + enough at the actual expected volume, the XFS tier could be dropped + entirely and S3 used as the sole backend, simplifying Storage Backend + Handover to one path instead of two. +- Exact JSON trailer schema beyond the `clearsign` field (format version, + chunk descriptor list — whether chunk descriptors are duplicated at the + JSON top level for easy programmatic access, or only present inside the + signed attribute list and require parsing the clearsign block to read). +- **Get the existing S3 handover interface spec document** referenced in + Storage Backend Handover: the exact regex used to map `x-amz-meta-*` + keys to NiFi attributes, and the exact flag/tag name and value convention + used to signal the >2KB fallback to the JSON trailer. This design assumes + that spec's behavior but doesn't reproduce its details. diff --git a/grand_plan.jpg b/grand_plan.jpg new file mode 100644 index 0000000..93d0518 Binary files /dev/null and b/grand_plan.jpg differ