Apache Pulsar Connector
The Pulsar connector allows you to stream, filter, and transform events between Hazelcast clusters and Pulsar.
Apache Pulsar is a popular distributed streaming and messaging system. Data in Pulsar is structured as topics and each topic consists of one or more partitions, stored in the Pulsar cluster. Each topic is assigned to one tenant and namespace.
Installing the Connector
This connector is included in the full distribution of Hazelcast.
If you're using the slim distribution, you must add the hazelcast-jet-pulsar module to your member's classpath.
Permissions
Enterprise EditionThis feature requires Hazelcast Enterprise, a licensed edition of the Hazelcast Platform.If security is enabled, your clients may need permissions to use this connector. For details, see Securing Jobs.
Configuration Options
To read from Pulsar as a source you must provide at least Schema, which Pulsar uses for given topic:
Pipeline p = Pipeline.reate();
p.readFrom(PulsarSources.pulsarConsumerBuilder(json(MyPojo.class))
.connectionSupplier(() -> PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build())
.topic("myTopic")
.build())
.withNativeTimestamps(0)
.writeTo(Sinks.logger());
Each member of Hazelcast cluster will create a Pulsar Consumer with the same subscription name and subscription type "Shared", meaning that the reading will be spread across Hazelcast cluster.
You can also use PulsarSources#pulsarConsumer or PulsarSources#pulsarReader, which are the same as in old version from hazelcast-jet-contrib. We kept those for easy migration, but please be aware, that they will be removed at some point.
Reader options
PulsarSources.pulsarReaderBuilder() returns a PulsarReaderBuilder, which has the following configuration methods:
| Method | Description |
|---|---|
topic | The Pulsar topic or topics to read from. |
connection | A function that supplies the Pulsar client to use. Use this or data, not both. |
data | The data connection to use for connecting to Pulsar. Use this or connection, not both. |
projection | Converts each Pulsar message to the item the source emits. |
identity | Sets the projection function to return the Message object unchanged, and disables message pooling to make this safe. |
disable | Disables Pulsar's message pooling, which is on by default. Disable it if your projection function returns the Message object itself rather than a value extracted from it, because a pooled message's underlying buffer can be reused after the source has processed it. |
reader | A function that configures the underlying Pulsar Reader directly, for options this builder does not expose. |
reader | Pulsar reader configuration properties, applied directly to the underlying Reader. |
preferred | The preferred number of parallel reader processors on each member. |
initial | The message to start reading from. After the job restores a snapshot, the source resumes from the snapshotted offset instead of this value. |
When used as a sink, then the only requirement is again the Schema of target topic:
Pipeline p = Pipeline.create();
p.readFrom(Sources.files("home/logs"))
.map(line -> LogParser.parse(line))
.map(log -> new LogEntry(log))
.writeTo(PulsarSinks.builder(json(LogEntry.class))
.connectionSupplier(() -> PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build())
.topic("logs")
.build());`
Sink options
PulsarSinks.builder() returns a PulsarSinkBuilder, which has the following configuration methods:
| Method | Description |
|---|---|
topic | The Pulsar topic to publish to. |
connection | A function that supplies the Pulsar client to use. Use this or data, not both. |
http | The URL of Pulsar's HTTP service. The sink uses it to check whether deduplication is enabled on the broker before starting an EXACTLY_ job. See Transactional Guarantees. |
data | The data connection to use for connecting to Pulsar. Use this or connection, not both. It also lets the sink perform the deduplication check described in Transactional Guarantees. |
producer | Pulsar producer configuration properties, applied directly to the underlying Producer. |
extract | Extracts the message key from each emitted item. |
extract | Extracts the message value from each emitted item. |
extract | Extracts the message properties from each emitted item. |
extract | Extracts the event timestamp from each emitted item. |
transaction | The timeout for transactions, used when the job has a processing guarantee. |
use | Enables or disables exactly-once behavior for this sink independently of the job's processing guarantee. Defaults to true. Set it to false to use at-least-once for this sink even when the job's guarantee is EXACTLY_. |
preferred | The preferred number of parallel producer processors on each member. Defaults to four. |
producer | A function that configures the underlying Pulsar Producer directly, for options this builder does not expose. |
Fault Tolerance
One of the most important features of using Pulsar as a source is that it's possible to replay data, which enables fault-tolerance. If the job has a processing guarantee configured, Hazelcast will periodically save the current offsets internally and then replay from the saved offset when the job is restarted. In this mode, Hazelcast will commit messages, keeping track of their offsets.
If the processing guarantees are disabled, the source will start reading from earliest messages in the topic.
Pulsar source is a stateful source—it remembers the last processed offset. To prevent data loss when using at-least-once or exactly-once job guarantees and Pulsar consumer configured to start from the latest offset, enable setRequireSnapshotBeforeProcessing in the job configuration. For more information, see Fault Tolerance.
Transactional Guarantees
As a sink, the Pulsar connector provides exactly-once guarantees at the cost of using Pulsar transactions, which must be enabled on the broker and in DataConnecion or the provided PulsarClient. Hazelcast commits the produced records messages each snapshot is completed. This greatly increases the latency because consumers see the records only after they are committed.
If you use at-least-once guarantee or user calls useTransactions(false) in sink builder, records are visible immediately, but in the case of a failure some records could be duplicated. You can also configure jobs in exactly-once mode and decrease the guarantee just for a particular Pulsar sink.
Enable deduplication on the Pulsar sink topic. In rare cases, the Pulsar client retries message delivery, for example when a network issue prevents the client from receiving an acknowledgment.
When the job's processing guarantee is EXACTLY_ONCE, the sink checks whether deduplication is enabled on the broker before it starts. If deduplication is disabled, the sink fails job initialization with an error, because exactly-once delivery requires it.
If the sink has neither a data connection nor an httpServiceUrl configured, it cannot perform this check. In that case, it logs a warning and proceeds instead of failing. Configure one of these options to enforce the check.
Version Compatibility
The Pulsar sink and source are based on version 5.x while keeping support of 4.x.