org.apache.flink.streaming.connectors.kafka.internals

类 AbstractPartitionDiscoverer

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer

直接已知子类:

KafkaPartitionDiscoverer

@Internal
public abstract class AbstractPartitionDiscoverer
extends Object

Base class for all partition discoverers.

This partition discoverer base class implements the logic around bookkeeping discovered partitions, and using the information to determine whether or not there are new partitions that the consumer subtask should subscribe to.

Subclass implementations should simply implement the logic of using the version-specific Kafka clients to fetch topic and partition metadata.

Since Kafka clients are generally not thread-safe, partition discoverers should not be concurrently accessed. The only exception for this would be the wakeup() call, which allows the discoverer to be interrupted during a discoverPartitions() call.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	AbstractPartitionDiscoverer.ClosedException Thrown if this discoverer was used to discover partitions after it was closed.
static class	AbstractPartitionDiscoverer.WakeupException Signaling exception to indicate that an actual Kafka call was interrupted.

构造器概要

构造器

构造器和说明
AbstractPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor, int indexOfThisSubtask, int numParallelSubtasks)

方法概要

限定符和类型	方法和说明
void	close() Closes the partition discoverer, cleaning up all Kafka connections.
protected abstract void	closeConnections() Close all established connections.
List<KafkaTopicPartition>	discoverPartitions() Execute a partition discovery attempt for this subtask.
protected abstract List<KafkaTopicPartition>	getAllPartitionsForTopics(List<String> topics) Fetch the list of all partitions for a specific topics list from Kafka.
protected abstract List<String>	getAllTopics() Fetch the list of all topics from Kafka.
protected abstract void	initializeConnections() Establish the required connections in order to fetch topics and partitions metadata.
void	open() Opens the partition discoverer, initializing all required Kafka connections.
boolean	setAndCheckDiscoveredPartition(KafkaTopicPartition partition) Sets a partition as discovered.
void	wakeup() Interrupt an in-progress discovery attempt by throwing a AbstractPartitionDiscoverer.WakeupException.
protected abstract void	wakeupConnections() Attempt to eagerly wakeup from blocking calls to Kafka in getAllTopics() and getAllPartitionsForTopics(List).

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

AbstractPartitionDiscoverer

public AbstractPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor,
                                   int indexOfThisSubtask,
                                   int numParallelSubtasks)

方法详细资料

open

public void open()
          throws Exception

Opens the partition discoverer, initializing all required Kafka connections.

NOTE: thread-safety is not guaranteed.

抛出:

Exception

close

public void close()
           throws Exception

Closes the partition discoverer, cleaning up all Kafka connections.

NOTE: thread-safety is not guaranteed.

抛出:

Exception

wakeup

public void wakeup()

Interrupt an in-progress discovery attempt by throwing a AbstractPartitionDiscoverer.WakeupException. If no attempt is in progress, the immediate next attempt will throw a AbstractPartitionDiscoverer.WakeupException.

This method can be called concurrently from a different thread.

discoverPartitions

public List<KafkaTopicPartition> discoverPartitions()
                                             throws AbstractPartitionDiscoverer.WakeupException,
                                                    AbstractPartitionDiscoverer.ClosedException

Execute a partition discovery attempt for this subtask. This method lets the partition discoverer update what partitions it has discovered so far.

返回:

List of discovered new partitions that this subtask should subscribe to.

抛出:

AbstractPartitionDiscoverer.WakeupException

AbstractPartitionDiscoverer.ClosedException

setAndCheckDiscoveredPartition

public boolean setAndCheckDiscoveredPartition(KafkaTopicPartition partition)

Sets a partition as discovered. Partitions are considered as new if its partition id is larger than all partition ids previously seen for the topic it belongs to. Therefore, for a set of discovered partitions, the order that this method is invoked with each partition is important.

If the partition is indeed newly discovered, this method also returns whether the new partition should be subscribed by this subtask.

参数:

partition - the partition to set and check

返回:

true, if the partition wasn't seen before and should be subscribed by this subtask; false otherwise

initializeConnections

protected abstract void initializeConnections()
                                       throws Exception

Establish the required connections in order to fetch topics and partitions metadata.

抛出:

Exception

wakeupConnections

protected abstract void wakeupConnections()

Attempt to eagerly wakeup from blocking calls to Kafka in getAllTopics() and getAllPartitionsForTopics(List).

If the invocation indeed results in interrupting an actual blocking Kafka call, the implementations of getAllTopics() and getAllPartitionsForTopics(List) are responsible of throwing a AbstractPartitionDiscoverer.WakeupException.

closeConnections

protected abstract void closeConnections()
                                  throws Exception

Close all established connections.

抛出:

Exception

getAllTopics

protected abstract List<String> getAllTopics()
                                      throws AbstractPartitionDiscoverer.WakeupException

Fetch the list of all topics from Kafka.

抛出:

AbstractPartitionDiscoverer.WakeupException

getAllPartitionsForTopics

protected abstract List<KafkaTopicPartition> getAllPartitionsForTopics(List<String> topics)
                                                                throws AbstractPartitionDiscoverer.WakeupException

Fetch the list of all partitions for a specific topics list from Kafka.

抛出:

AbstractPartitionDiscoverer.WakeupException