Design Principles
These principles explain why NPipeline is built the way it is. Understanding them helps you make changes that are consistent with the existing codebase.
Streaming-First
Data flows item-by-item through IAsyncEnumerable<T>. Nothing is buffered in memory unless the user explicitly opts in (the node restart replay window, batching, in-memory streams).
Why: Pipelines often process datasets larger than available memory. Streaming lets you process a 10GB CSV file with constant memory usage.
Implication for contributors: Never collect an entire stream into a list as a convenience. If a feature requires buffering, make it opt-in, hold only what it needs, and bound it with backpressure rather than an error. For an example, see NodeRestartOptions.MaxReplayWindow: node restart holds only the items between the checkpoint and the read head, and stops reading its input when the window is full.
Lazy Evaluation
Source nodes don't produce items until the downstream consumer requests them. Transforms don't execute until enumerated. This is the natural behavior of IAsyncEnumerable<T> - items are pulled, not pushed.
Why: Lazy evaluation enables backpressure for free. A slow sink automatically slows down the source without explicit flow control logic.
Implication for contributors: Don't eagerly enumerate streams in execution strategies. Even node restart reads its input lazily and holds only the items it might replay. Use await foreach and yield results as IAsyncEnumerable<T>.
Type Safety at the Graph Level
Node connections are validated at build time through typed handles (SourceNodeHandle<TOut>, TransformNodeHandle<TIn, TOut>, SinkNodeHandle<TIn>). The C# compiler prevents connecting nodes with incompatible types.
Why: Runtime type mismatches in data pipelines are notoriously hard to debug. Compile-time checking eliminates an entire class of errors.
Implication for contributors: When adding new node types, ensure the builder method returns a strongly-typed handle. Use generic constraints to enforce type compatibility in Connect() overloads.
Immutable Configuration
All configuration records (PipelineResilienceOptions, ItemRetryOptions, NodeRestartOptions, CircuitBreakerOptions, LineageOptions, and others) are sealed record types. They're modified using with expressions, never mutation.
Why: Immutability prevents configuration changes during execution from creating race conditions. It also enables safe sharing across threads.
Implication for contributors: New configuration should be sealed record with init-only properties or positional parameters. Provide a Default static property. Include a Validate() method when constraints exist.
Fail-Fast Defaults
The default resilience policy carries out the node's resilience options exactly and adds no rules of its own. Under the Default optimization profile, a transform retries a transient item failure three times with exponential backoff, and a permanent failure fails at once. Under HighThroughput, nothing is retried. In either profile, nothing is skipped, dead-lettered, or restarted until you opt in.
Why: Silent data loss is worse than a loud failure. In data pipelines, a dropped item that nobody notices can corrupt downstream reports, analytics, or databases.
Implication for contributors: New features should default to the strictest (safest) behavior. Permissive behavior requires explicit opt-in. Retrying transient item failures is the only recovery that's on by default, and it never drops an item: when the retries run out, the node fails.
Zero-Allocation Hot Paths
The item processing loop - the code path that runs once per item per node - avoids heap allocations. This means no LINQ in hot paths, no closures that capture variables, no boxing of value types, and ValueTask<T> where appropriate.
Why: Pipeline throughput is directly limited by GC pressure. At millions of items per second, even one allocation per item causes significant GC overhead.
Implication for contributors: Roslyn analyzers (LinqInHotPathsAnalyzer, AnonymousObjectAllocationAnalyzer) enforce this. Profile with BenchmarkDotNet before and after changes to hot paths.
Compiled Factories Over Reflection
Node instantiation uses compiled expression trees cached in a ConcurrentDictionary<Type, Func<INode>>. This eliminates per-invocation reflection overhead after the first call.
Why: Activator.CreateInstance is slow. Compiled expressions are as fast as direct constructor calls after the initial compilation cost.
Implication for contributors: See Node Instantiation for details. Node types must have a parameterless constructor (enforced by the NodeParameterlessConstructorAnalyzer) unless using DI.
Extension Points Over Modification
New behavior is added by implementing interfaces (IExecutionStrategy, IResiliencePolicy, IDeadLetterSink, IObservabilitySurface, ILineage), not by modifying existing classes.
Why: Open/closed principle. The core runtime is stable; extensions add capabilities without risking regressions.
Implication for contributors: Before modifying an existing class, check if the behavior can be added through an existing extension point. If not, consider adding a new extension point rather than embedding the behavior.
Separation of Construction and Execution
Pipeline definition (PipelineBuilder) is separate from pipeline execution (PipelineRunner). The builder produces an immutable PipelineGraph; the runner executes it.
Why: This separation enables execution plan caching (same graph, different runs), graph validation before execution, and potential future features like graph serialization or visualization.
Implication for contributors: Don't mix construction logic with execution logic. The builder should never depend on execution classes, and execution code should only read from the immutable graph.
Next Steps
- Execution Model - how these principles manifest in the runner
- Node Instantiation - compiled factories in detail
- Coding Conventions - practical rules derived from these principles