What Next?

You've installed NPipeline, built your first pipeline, and understand the mental model. Where you go from here depends on what you're trying to accomplish.

I Want To

Read or write files (CSV, JSON, Parquet, Excel)

Install a connector package and use its pre-built source/sink nodes.

→ Connectors - choose the right connector for your file format

Connect to a database (PostgreSQL, SQL Server, MongoDB, etc.)

Database connectors provide source and sink nodes for reading/writing directly.

→ Connectors - database connectors section

Process messages from Kafka, RabbitMQ, or Azure Service Bus

Message queue connectors give you streaming sources that consume messages in real time.

→ Connectors - messaging connectors section

Handle errors without crashing the pipeline

NPipeline has built-in retry strategies, dead-letter queues, and circuit breakers.

→ Error Handling

Run nodes in parallel for higher throughput

The parallelism extension lets you process items concurrently with configurable thread safety.

→ Parallel Execution → Optimization Profiles - choose HighThroughput for maximum performance

Split data into multiple paths (branching or conditional routing)

Use branch/tap for unconditional fan-out, and RouteNode for predicate-based routing.

→ Branching and Merging → Routing with RouteNode

Combine data from multiple sources (joins)

Join nodes merge data from two or more input streams based on keys or time windows.

→ Joins and Lookups

Group or aggregate data

Aggregate nodes and grouping strategies let you compute summaries over streams.

→ Aggregation

Process large files without running out of memory

NPipeline streams by default, but understanding the buffering trade-offs helps with very large datasets.

→ Streaming Large Datasets

Use dependency injection

Integrate NPipeline with Microsoft.Extensions.DependencyInjection for automatic node resolution.

→ Dependency Injection

Test my pipelines

In-memory sources and sinks, plus a test harness, make pipeline testing straightforward.

→ Testing Pipelines

Understand what's available as a package

See every extension, connector, and storage provider NPipeline offers.

→ Extensions

Optimize performance

Techniques for reducing allocations, caching execution plans, and choosing the right patterns.

→ Performance Best Practices

Next Steps

• Defining Pipelines - fluent builder vs class-based approaches
• Lambda Nodes - inline nodes without separate classes
• Custom Nodes - writing your own source, transform, and sink nodes
• Reference: Glossary - definitions for all NPipeline terminology