Skip to main content
Version: 6.0-SNAPSHOT

Apache Pulsar Connector

The Pulsar connector allows you to stream, filter, and transform events between Hazelcast clusters and Pulsar.

Apache Pulsar is a popular distributed streaming and messaging system. Data in Pulsar is structured as topics and each topic consists of one or more partitions, stored in the Pulsar cluster. Each topic is assigned to one tenant and namespace.

Installing the Connector​

This connector is included in the full distribution of Hazelcast.

If you're using the slim distribution, you must add the hazelcast-jet-pulsar module to your member's classpath.

Permissions​

Enterprise EditionThis feature requires Hazelcast Enterprise, a licensed edition of the Hazelcast Platform.

If security is enabled, your clients may need permissions to use this connector. For details, see Securing Jobs.

Configuration Options​

To read from Pulsar as a source you must provide at least Schema, which Pulsar uses for given topic:

java
Pipeline p = Pipeline.reate();
p.readFrom(PulsarSources.pulsarConsumerBuilder(json(MyPojo.class))
.connectionSupplier(() -> PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build())
.topic("myTopic")
.build())
.withNativeTimestamps(0)
.writeTo(Sinks.logger());

Each member of Hazelcast cluster will create a Pulsar Consumer with the same subscription name and subscription type "Shared", meaning that the reading will be spread across Hazelcast cluster.

You can also use PulsarSources#pulsarConsumer or PulsarSources#pulsarReader, which are the same as in old version from hazelcast-jet-contrib. We kept those for easy migration, but please be aware, that they will be removed at some point.

Reader options​

PulsarSources.pulsarReaderBuilder() returns a PulsarReaderBuilder, which has the following configuration methods:

MethodDescription
topicThe Pulsar topic or topics to read from.
connectionSupplierA function that supplies the Pulsar client to use. Use this or dataConnectionRef, not both.
dataConnectionRefThe data connection to use for connecting to Pulsar. Use this or connectionSupplier, not both.
projectionFnConverts each Pulsar message to the item the source emits.
identityProjectionFnSets the projection function to return the Message object unchanged, and disables message pooling to make this safe.
disableMessagePoolingDisables Pulsar's message pooling, which is on by default. Disable it if your projection function returns the Message object itself rather than a value extracted from it, because a pooled message's underlying buffer can be reused after the source has processed it.
readerCustomizerA function that configures the underlying Pulsar ReaderBuilder directly, for options this builder does not expose.
readerConfigPulsar reader configuration properties, applied directly to the underlying ReaderBuilder.
preferredLocalParallelismThe preferred number of parallel reader processors on each member.
initialOffsetThe message to start reading from. After the job restores a snapshot, the source resumes from the snapshotted offset instead of this value.

When used as a sink, then the only requirement is again the Schema of target topic:

java
Pipeline p = Pipeline.create();
p.readFrom(Sources.files("home/logs"))
.map(line -> LogParser.parse(line))
.map(log -> new LogEntry(log))
.writeTo(PulsarSinks.builder(json(LogEntry.class))
.connectionSupplier(() -> PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build())
.topic("logs")
.build());`

Sink options​

PulsarSinks.builder() returns a PulsarSinkBuilder, which has the following configuration methods:

MethodDescription
topicThe Pulsar topic to publish to.
connectionSupplierA function that supplies the Pulsar client to use. Use this or dataConnectionRef, not both.
httpServiceUrlThe URL of Pulsar's HTTP service. The sink uses it to check whether deduplication is enabled on the broker before starting an EXACTLY_ONCE job. See Transactional Guarantees.
dataConnectionRefThe data connection to use for connecting to Pulsar. Use this or connectionSupplier, not both. It also lets the sink perform the deduplication check described in Transactional Guarantees.
producerConfigPulsar producer configuration properties, applied directly to the underlying ProducerBuilder.
extractKeyFnExtracts the message key from each emitted item.
extractValueFnExtracts the message value from each emitted item.
extractPropertiesFnExtracts the message properties from each emitted item.
extractTimestampFnExtracts the event timestamp from each emitted item.
transactionTimeoutThe timeout for transactions, used when the job has a processing guarantee.
useTransactionsEnables or disables exactly-once behavior for this sink independently of the job's processing guarantee. Defaults to true. Set it to false to use at-least-once for this sink even when the job's guarantee is EXACTLY_ONCE.
preferredLocalParallelismThe preferred number of parallel producer processors on each member. Defaults to four.
producerCustomizerA function that configures the underlying Pulsar ProducerBuilder directly, for options this builder does not expose.

Fault Tolerance​

One of the most important features of using Pulsar as a source is that it's possible to replay data, which enables fault-tolerance. If the job has a processing guarantee configured, Hazelcast will periodically save the current offsets internally and then replay from the saved offset when the job is restarted. In this mode, Hazelcast will commit messages, keeping track of their offsets.

If the processing guarantees are disabled, the source will start reading from earliest messages in the topic.

Pulsar source is a stateful source—it remembers the last processed offset. To prevent data loss when using at-least-once or exactly-once job guarantees and Pulsar consumer configured to start from the latest offset, enable setRequireSnapshotBeforeProcessing in the job configuration. For more information, see Fault Tolerance.

Transactional Guarantees​

As a sink, the Pulsar connector provides exactly-once guarantees at the cost of using Pulsar transactions, which must be enabled on the broker and in DataConnecion or the provided PulsarClient. Hazelcast commits the produced records messages each snapshot is completed. This greatly increases the latency because consumers see the records only after they are committed.

If you use at-least-once guarantee or user calls useTransactions(false) in sink builder, records are visible immediately, but in the case of a failure some records could be duplicated. You can also configure jobs in exactly-once mode and decrease the guarantee just for a particular Pulsar sink.

Enable deduplication on the Pulsar sink topic. In rare cases, the Pulsar client retries message delivery, for example when a network issue prevents the client from receiving an acknowledgment.

When the job's processing guarantee is EXACTLY_ONCE, the sink checks whether deduplication is enabled on the broker before it starts. If deduplication is disabled, the sink fails job initialization with an error, because exactly-once delivery requires it.

If the sink has neither a data connection nor an httpServiceUrl configured, it cannot perform this check. In that case, it logs a warning and proceeds instead of failing. Configure one of these options to enforce the check.

Version Compatibility​

The Pulsar sink and source are based on version 5.x while keeping support of 4.x.