Skip to content

Kafka Connector ​

The NPipeline.Connectors.Kafka package consumes Kafka topics as a member of a consumer group and produces to topics, with at-least-once delivery by default and exactly-once transactions when asked. Serialization, settlement, undeserializable messages and failed writes work as in every message-queue connector; see Message Queues: Shared Behaviour.

Installation ​

bash
dotnet add package NPipeline.Connectors.Kafka

Dependencies: Confluent.Kafka, and Confluent.SchemaRegistry for the Avro and Protobuf serializers.

Consuming ​

csharp
var orders = KafkaConnector.Source<Order>("kafka:9092", "orders", groupId: "billing",
    o => o with { AutoOffsetReset = AutoOffsetReset.Earliest });

builder.AddSource(orders, "orders");

Each message is a KafkaMessage<T> with its Body, Key (UTF-8 text), Topic, Partition, Offset, Timestamp, Headers and IsTombstone. MessageId is topic/partition/offset, which is unique, and Metadata holds the same values (Topic, Partition, Offset, Timestamp, Key) with each header as Header.<name>.

Offsets ​

Acknowledging a message stores its offset, and the consumer commits stored offsets every CommitInterval (5 seconds) and when it closes. Offsets commit in order: a partition's committed offset only moves past a message once every earlier message of that partition is settled, so a message still being written, or one that failed, is never skipped by a later acknowledgement. What is committed is the next offset to read, so a restart continues after the last acknowledged message.

RejectAsync(requeue: false) moves past a message. RejectAsync(requeue: true) holds the partition's commits at it, since Kafka cannot redeliver one message: a restart reads it, and what follows, again.

When the read ends, the consumer stays until the messages it handed on are settled (up to SettleTimeout), then commits and leaves the group, so its partitions move to another member at once rather than after a session timeout.

Source options ​

OptionDefaultDescription
Topic, GroupIdrequiredThe topic, and the consumer group whose committed offsets a read starts from
AutoOffsetResetLatestWhere a group with no committed offset starts: Latest (Kafka's default) or Earliest
GroupInstanceIdnullA static member id, so a restarted member keeps its partitions without a rebalance
IsolationLevellibrdkafka's, read-committedWhether transactional messages are read only once committed
CommitInterval5 sHow often stored offsets are committed
SkipTombstonestrueTombstones (null values) are passed over; with false they are handed on with a default body and IsTombstone set
RowErrorHandler, RawExcerptLengthfail, 256See Messages that don't deserialize; Skip moves past the message
PollTimeout100 msHow long a poll waits before polling again
ResilienceKafkaConnectorResilience.DefaultHow a failed poll is retried; see Resilience
SettleTimeout30 sHow long the consumer waits for handed-on messages to be settled after the read ends

Producing ​

csharp
var invoices = KafkaConnector.Sink<Invoice>("kafka:9092", "invoices", o => o with { KeySelector = invoice => invoice.CustomerId });
builder.AddSink(invoices.Acknowledging(), "invoices");

The key decides the partition, and so the order: messages with one key keep their order. With no KeySelector, a message read from Kafka keeps its key, and any other message has none, so librdkafka spreads them over the partitions.

The sink produces each batch of BatchSize messages and waits for all their deliveries; written through Acknowledging(), each source message is acknowledged once the broker has it.

Sink options ​

OptionDefaultDescription
TopicrequiredThe topic
KeySelectornullChooses each message's key from its body
CopyHeadersfalseWhether a message read from Kafka keeps its headers
AcksAllHow many replicas must have a message before the broker acknowledges it
EnableIdempotencetrueThe producer's own retries never duplicate a message
Linger, Compression5 ms, nonelibrdkafka's batching delay and codec; CompressionType.Lz4 or Zstd suits busy topics
DeliveryTimeout2 minHow long librdkafka keeps retrying a message before it fails
BatchSize, BatchLinger1,000, 10 msMessages whose deliveries are awaited together, and the longest a batch waits to fill
FailedMessagesFailSee Failed writes
TransactionalId, TransactionTimeoutnone, 30 sExactly-once; see below

Exactly-once ​

With a TransactionalId, each batch is one transaction that also commits the offsets of the Kafka messages it came from, so reading, transforming and writing are exactly-once: a batch and its source offsets commit or abort together.

csharp
var orders = KafkaConnector.Source<Order>("kafka:9092", "orders", "billing");
var invoices = KafkaConnector.Sink<Invoice>("kafka:9092", "invoices", o => o with { TransactionalId = "billing-1" });

Each producer instance needs its own id. A failed batch aborts its transaction and fails the write (so FailedMessages must be Fail), and a restart reads the batch's messages again. Consumers of the output should use read-committed isolation, librdkafka's default.

Connections and security ​

Both options records share the client settings:

OptionDescription
BootstrapServersThe brokers, host:port separated by commas
ClientIdThe client id the brokers see
SecurityProtocol, SaslMechanism, SaslUsername, SaslPasswordSecurity; SASL with Plain or SCRAM needs the user name and password
ClientSettingsAny other librdkafka setting by name, such as ssl.ca.location, applied last

Tuning with ClientSettings ​

The options cover what most pipelines change. For the rest, set the librdkafka property directly. These are the ones earlier versions of the connector exposed as properties:

librdkafka settingApplies toDefaultEffect
fetch.min.bytes, fetch.max.bytesSource1, 52428800The least and most data a fetch returns; raise the minimum to trade latency for throughput
max.poll.interval.msSource300000How long the consumer may go without polling before the group removes it
session.timeout.msSource45000How long the broker waits for a heartbeat before it moves the member's partitions
batch.sizeSink1000000The producer's batch size in bytes (this is separate from the sink's BatchSize, which counts messages)
message.max.bytesSink1000000The largest message the producer accepts
retry.backoff.ms, retry.backoff.max.msSink100, 1000The first and longest delay between librdkafka's produce retries
csharp
var orders = KafkaConnector.Source<Order>("kafka:9092", "orders", "billing", o => o with
{
    ClientSettings = new Dictionary<string, string> { ["fetch.min.bytes"] = "65536", ["max.poll.interval.ms"] = "600000" },
});

Serialization ​

JSON by default; see Serialization. Avro and Protobuf use a schema registry:

SerializerFormatBest for
JsonMessageSerializer (default)JSON, no schemaSimple messages, debugging
AvroMessageSerializerAvroSchema evolution, compact encoding
ProtobufMessageSerializerProtobufCross-language use, compact encoding
csharp
var registry = new SchemaRegistryConfiguration { Url = "http://registry:8081" };
var orders = KafkaConnector.Source<OrderRecord>("kafka:9092", "orders", "billing", o => o with { Serializer = new AvroMessageSerializer(registry) });
SchemaRegistryConfiguration optionDefaultDescription
UrlrequiredThe registry's address
BasicAuthUsername, BasicAuthPasswordnullBasic authentication
EnableSslfalseUse TLS to reach the registry
RequestTimeoutMs30000How long a registry call may take
SchemaCacheCapacity1000The most schemas cached locally
AutoRegisterSchemastrueRegister a schema the registry doesn't have yet
SubjectNameStrategyTopicTopic, Record or TopicRecord

Schemas are registered and looked up under a subject derived from the topic. Under the default subject name strategy the value subject is <topic>-value, so sinks writing different types to different topics each get their own subject. SubjectNameStrategy and AutoRegisterSchemas apply to both serializers. To use another format or registry, implement IMessageSerializer; it receives the topic as MessageContext.Destination, and IsKey tells a key from a body.

Resilience ​

The source retries a failed poll with NResilience. KafkaConnectorResilience.Default:

  • Makes up to four attempts per poll (three retries), with exponential backoff and full jitter from 100 ms up to 30 s.
  • Retries errors Kafka reports as retriable: a lost broker connection, a timeout, a leader or coordinator that moved, a rebalance, too few in-sync replicas, an exceeded max.poll.interval.ms. Quota throttling takes the longer backoff.
  • Doesn't retry a fatal error, an authorization failure, or any other error.

Derive a policy to change it (KafkaConnectorResilience.Default with { Attempts = 6 }), or turn retries off with Resilience.None. KafkaConnectorResilience.IsRetriable(Error) exposes the classification.

The sink doesn't retry: librdkafka retries every failed produce until DeliveryTimeout, and the idempotent producer removes the duplicates those retries would cause. A retry above librdkafka would be a new record the producer could not recognize, so a message whose delivery timed out but still reached the broker would be written twice.

Dead letters ​

KafkaDeadLetterSink is a pipeline dead-letter sink that produces failed items to a topic, with the error in x-dead-letter-* headers. A MessageFailure (a message that didn't deserialize) is produced with its original value and key, so it can be replayed:

csharp
using var producer = new ProducerBuilder<byte[]?, byte[]>(new ProducerConfig { BootstrapServers = "kafka:9092" }).Build();
builder.AddDeadLetterSink(new KafkaDeadLetterSink(producer, "orders-dead-letters"));

Best practices ​

  1. Keep Acks.All and EnableIdempotence (the defaults) for durability.
  2. Give each logical consumer its own GroupId. Members of one group share a topic's partitions, so run more members, up to the partition count, to read in parallel.
  3. Use Avro or Protobuf with a schema registry when the schema will evolve.
  4. Use CompressionType.Lz4 or Zstd on busy topics, and keep Linger at 5 to 50 ms so small messages batch.
  5. Choose keys deliberately. Messages with one key keep their order, and a hot key overloads one partition.
  6. Use exactly-once only where duplicates matter. A transaction per batch costs more than at-least-once delivery.
  7. Watch the metrics (Metrics) for rows read and written, row errors and settlements.

Next Steps ​

Released under the MIT License.