Dead-Letter Queues

When a resilience policy returns ResilienceDecision.DeadLetter, the failed item is routed to a dead-letter sink rather than being discarded or crashing the pipeline. This preserves failed items for debugging, reprocessing, or alerting - without blocking the pipeline from continuing.

When Items Are Dead-Lettered

An item reaches the dead-letter sink when:

1. A resilience policy explicitly returns ResilienceDecision.DeadLetter.
2. A retry policy exhausts its maximum retries and its exhaustedDecision is DeadLetter (the default for the fluent builder's .Retry() method).
3. The default policy decides the failure, it doesn't retry the item, and the node's options set OnItemFailure = ItemFailureAction.DeadLetter.

// After 3 failed retries, the item goes to dead-letter
ResiliencePolicyBuilder
    .ForNode<MyTransform, Order>()
    .On<TimeoutException>().Retry(maxRetries: 3)  // exhausted → DeadLetter
    .Build();

If no dead-letter sink is configured when an item is dead-lettered, the node fails with DeadLetterSinkNotConfiguredException (NP0424), and the original failure is its inner exception. NPipeline never drops the item silently. Add a sink with builder.AddDeadLetterSink(...), or have the policy return Skip to discard the item.

If a transform's options set OnItemFailure = ItemFailureAction.DeadLetter and no dead-letter sink is configured, the run fails with DeadLetterSinkNotConfiguredException before any node starts.

The DeadLetterEnvelope

Every dead-lettered item is wrapped in a DeadLetterEnvelope that captures full failure context:

public sealed record DeadLetterEnvelope(
    object Item,                        // The item that failed
    Exception Error,                    // The exception that caused the failure
    NodeFailureAttribution Attribution  // Where the failure originated and was decided
);

The NodeFailureAttribution record provides traceability:

public sealed record NodeFailureAttribution(
    string OriginNodeId,       // Node where the exception was thrown
    string DecisionNodeId,     // Node where the error handling decision was made
    Guid OriginPipelineId,     // Pipeline containing the origin node
    Guid DecisionPipelineId,   // Pipeline containing the decision node
    Guid? RunId = null,        // Pipeline run identifier
    Guid? CorrelationId = null,// Item-level correlation ID
    int RetryCount = 0         // Number of retry attempts before dead-lettering
);

The IDeadLetterSink Interface

Implement IDeadLetterSink to control where dead-lettered items go:

public interface IDeadLetterSink
{
    Task HandleAsync(
        DeadLetterEnvelope envelope,
        PipelineContext context,
        CancellationToken cancellationToken);
}

Built-In: BoundedInMemoryDeadLetterSink

NPipeline ships a memory-bounded dead-letter sink for development and testing:

using NPipeline.ErrorHandling;

// Default: holds up to 1000 items
var sink = new BoundedInMemoryDeadLetterSink();

// Custom capacity for high-volume scenarios
var highCapacity = new BoundedInMemoryDeadLetterSink(capacity: 5000);

// Reduced capacity for memory-constrained environments
var constrained = new BoundedInMemoryDeadLetterSink(capacity: 100);

Behavior:

• Items are stored in a thread-safe ConcurrentQueue.
• When capacity is reached, the next dead-letter attempt throws an InvalidOperationException, failing the pipeline.
• Access collected items via the Items property (IReadOnlyCollection<DeadLetterEnvelope>).

Register it on the builder:

builder.AddDeadLetterSink(new BoundedInMemoryDeadLetterSink());

Or register by type (resolved via DI):

builder.AddDeadLetterSink<BoundedInMemoryDeadLetterSink>();

Custom Dead-Letter Sinks

For production, implement IDeadLetterSink to route failures to durable storage:

public class DatabaseDeadLetterSink : IDeadLetterSink
{
    private readonly IDbConnection _connection;

    public DatabaseDeadLetterSink(IDbConnection connection)
    {
        _connection = connection;
    }

    public async Task HandleAsync(
        DeadLetterEnvelope envelope,
        PipelineContext context,
        CancellationToken cancellationToken)
    {
        await _connection.ExecuteAsync(
            "INSERT INTO dead_letters (item, error, origin_node, retry_count, timestamp) " +
            "VALUES (@Item, @Error, @Origin, @Retries, @Timestamp)",
            new
            {
                Item = JsonSerializer.Serialize(envelope.Item),
                Error = envelope.Error.ToString(),
                Origin = envelope.Attribution.OriginNodeId,
                Retries = envelope.Attribution.RetryCount,
                Timestamp = DateTime.UtcNow
            });
    }
}

Other common implementations:

• File-based: Write JSON lines to a dead-letter file for batch reprocessing.
• Message queue: Publish to a separate queue for retry workflows.
• Alerting: Send to a monitoring system when items start dead-lettering.

Inspecting Dead-Lettered Items

With BoundedInMemoryDeadLetterSink, inspect failures after pipeline execution:

var deadLetterSink = new BoundedInMemoryDeadLetterSink();
builder.AddDeadLetterSink(deadLetterSink);

// ... run pipeline ...

foreach (var envelope in deadLetterSink.Items)
{
    Console.WriteLine($"Failed item from node '{envelope.Attribution.OriginNodeId}':");
    Console.WriteLine($"  Error: {envelope.Error.Message}");
    Console.WriteLine($"  Retry attempts: {envelope.Attribution.RetryCount}");
    Console.WriteLine($"  Item: {envelope.Item}");
}

Complete Example

public class ResilientOrderPipeline : IPipelineDefinition
{
    public void Define(PipelineBuilder builder, PipelineContext context)
    {
        // Wait between retries: exponential backoff from 200 ms up to 30 s, with full jitter.
        // The Default optimization profile already starts from these options; setting them keeps the example explicit.
        builder.WithResilience(options => options with { ItemRetry = ItemRetryOptions.Default });

        // Dead-letter sink captures failures for inspection
        builder.AddDeadLetterSink(new BoundedInMemoryDeadLetterSink(capacity: 500));

        // Policy: retry transient errors, dead-letter validation errors
        var policy = ResiliencePolicyBuilder
            .ForNode<ValidateOrder, Order>()
            .On<HttpRequestException>().Retry(3)
            .On<ValidationException>().DeadLetter()
            .OnAny().Fail()
            .Build();

        builder.AddResiliencePolicy(policy);

        var source = builder.AddSource<OrderSource, Order>("orders");
        var validate = builder.AddTransform<ValidateOrder, Order, Order>("validate");
        var save = builder.AddSink<OrderSink, Order>("save");

        builder.Connect(source, validate);
        builder.Connect(validate, save);
    }
}

In this pipeline:

• The policy decides every failure in ValidateOrder. ItemRetry supplies the wait between retries.
• The policy retries HTTP errors up to 3 times, waiting with exponential backoff between attempts.
• When the retries run out, the policy dead-letters the item, because Retry(n) dead-letters after its last retry.
• Validation errors are dead-lettered immediately (no retry).
• Any other exception fails the pipeline.

Next Steps

• Node restart and the replay window - restart a failed transform from its checkpoint
• Retry Strategies - configure delays between retry attempts
• Resilience Policies - customize which items get dead-lettered