Files

T

wall 45f4dc733d Add CLAUDE.md, project memory notes, and token_count

CLAUDE.md points future sessions at .claude/memory/ for accumulated
project context (purpose, SA/MR topology, packaging/signing decisions,
open external dependencies) not fully captured elsewhere in the repo.

2026-07-28 14:27:05 +10:00

1.9 KiB

Raw Blame History

CLAUDE.md

Project notes for the NiFi prototyping repo — improving NiFi workflow performance, packaging/signing of FlowFiles, and large-file (TB-scale) transfer across a one-way network boundary.

Before starting work

Read everything in .claude/memory/ first — it holds accumulated notes, terminology, and context (some of it non-obvious from the code/docs alone) that this file only summarizes. Treat it as living project memory: update it as significant new decisions or context emerge, don't just accumulate.

Key documents

2026-07-23-phat-files-design.md — approved spec for chunked TB-scale S3-to-S3 transfer across a one-way boundary. Source of proven concepts (chunking, resend/duplicate handling, server-side assembly) reused by the packaging/signing design below — not a dependency of it.
docs/superpowers/specs/2026-07-28-flowfile-packaging-signing-design.md — design for FlowFile packaging (content + JSON trailer), in-process OpenPGP signing, uniform chunking, and S3/XFS storage backend handling. Status: draft, pending user review.
grand_plan.jpg — whiteboard sketch of the SA/MR deployment topology the design above was reconciled against.

Terminology

SA — the overall system: two chains of independent, single-function NiFi hosts (linked via Site-to-Site within each chain) bridged by MR.
MR — the one-way relay/diode boundary between the two NiFi host chains. No return channel; forward error correction via an HTTP service, ~1GB target chunk size.

Environment

On-prem VMs, vSAN-backed storage — shared IOPS across the vSphere cluster, so contention (not per-VM disk speed) is the bottleneck.
NiFi runs as independent single-node instances (no clustering), each with a distinct logical function, scaled horizontally.

Remote

https://gitea-5001.taild45a1b.ts.net/Hackathon/GrandPlan (branch main) — credentials supplied out-of-band, not stored in this repo.

1.9 KiB Raw Blame History