org.apache.flink.streaming.connectors.kafka

类 FlinkKafkaConsumerBase<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

类型参数:

T - The type of records produced by this data source

所有已实现的接口:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<T>, org.apache.flink.streaming.api.functions.source.SourceFunction<T>

直接已知子类:

FlinkKafkaConsumer

@Internal
public abstract class FlinkKafkaConsumerBase<T>
extends org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>
implements org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Base class of all Flink Kafka Consumer data sources. This implements the common behavior across all Kafka versions.

The Kafka version specific behavior is defined mainly in the specific subclasses of the AbstractFetcher.

另请参阅:

序列化表格

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

字段概要

字段

限定符和类型	字段和说明
protected KafkaDeserializationSchema<T>	deserializer The schema to convert between Kafka's byte messages, and Flink's objects.
static String	KEY_DISABLE_METRICS Boolean configuration key to disable metrics tracking
static String	KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS Configuration key to define the consumer's partition discovery interval, in milliseconds.
protected static org.slf4j.Logger	LOG
static int	MAX_NUM_PENDING_CHECKPOINTS The maximum number of pending non-committed checkpoints to track, to avoid memory leaks.
static long	PARTITION_DISCOVERY_DISABLED The default interval to execute partition discovery, in milliseconds (Long.MIN_VALUE, i.e. disabled by default).

构造器概要

构造器

构造器和说明
FlinkKafkaConsumerBase(List<String> topics, Pattern topicPattern, KafkaDeserializationSchema<T> deserializer, long discoveryIntervalMillis, boolean useMetrics) Base constructor.

方法概要

限定符和类型	方法和说明
protected static void	adjustAutoCommitConfig(Properties properties, OffsetCommitMode offsetCommitMode) Make sure that auto commit is disabled when our offset commit mode is ON_CHECKPOINTS.
FlinkKafkaConsumerBase<T>	assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T> assigner) 已过时。 Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.
FlinkKafkaConsumerBase<T>	assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T> assigner) 已过时。 Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.
FlinkKafkaConsumerBase<T>	assignTimestampsAndWatermarks(org.apache.flink.api.common.eventtime.WatermarkStrategy<T> watermarkStrategy) Sets the given WatermarkStrategy on this consumer.
void	cancel()
void	close()
protected abstract AbstractFetcher<T,?>	createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> subscribedPartitionsToStartOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext, OffsetCommitMode offsetCommitMode, org.apache.flink.metrics.MetricGroup kafkaMetricGroup, boolean useMetrics) Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the data, and emits it into the data streams.
protected abstract AbstractPartitionDiscoverer	createPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor, int indexOfThisSubtask, int numParallelSubtasks) Creates the partition discoverer that is used to find new partitions for this subtask.
FlinkKafkaConsumerBase<T>	disableFilterRestoredPartitionsWithSubscribedTopics() By default, when restoring from a checkpoint / savepoint, the consumer always ignores restored partitions that are no longer associated with the current specified topics or topic pattern to subscribe to.
protected abstract Map<KafkaTopicPartition,Long>	fetchOffsetsWithTimestamp(Collection<KafkaTopicPartition> partitions, long timestamp)
boolean	getEnableCommitOnCheckpoints()
protected abstract boolean	getIsAutoCommitEnabled()
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
void	open(org.apache.flink.configuration.Configuration configuration)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext)
FlinkKafkaConsumerBase<T>	setCommitOffsetsOnCheckpoints(boolean commitOnCheckpoints) Specifies whether or not the consumer should commit offsets back to Kafka on checkpoints.
FlinkKafkaConsumerBase<T>	setStartFromEarliest() Specifies the consumer to start reading from the earliest offset for all partitions.
FlinkKafkaConsumerBase<T>	setStartFromGroupOffsets() Specifies the consumer to start reading from any committed group offsets found in Zookeeper / Kafka brokers.
FlinkKafkaConsumerBase<T>	setStartFromLatest() Specifies the consumer to start reading from the latest offset for all partitions.
FlinkKafkaConsumerBase<T>	setStartFromSpecificOffsets(Map<KafkaTopicPartition,Long> specificStartupOffsets) Specifies the consumer to start reading partitions from specific offsets, set independently for each partition.
FlinkKafkaConsumerBase<T>	setStartFromTimestamp(long startupOffsetsTimestamp) Specifies the consumer to start reading partitions from a specified timestamp.
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

LOG

protected static final org.slf4j.Logger LOG

MAX_NUM_PENDING_CHECKPOINTS

public static final int MAX_NUM_PENDING_CHECKPOINTS

The maximum number of pending non-committed checkpoints to track, to avoid memory leaks.

另请参阅:

常量字段值

PARTITION_DISCOVERY_DISABLED

public static final long PARTITION_DISCOVERY_DISABLED

The default interval to execute partition discovery, in milliseconds (Long.MIN_VALUE, i.e. disabled by default).

另请参阅:

常量字段值

KEY_DISABLE_METRICS

public static final String KEY_DISABLE_METRICS

Boolean configuration key to disable metrics tracking. *

另请参阅:

常量字段值

KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS

public static final String KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS

Configuration key to define the consumer's partition discovery interval, in milliseconds.

另请参阅:

常量字段值

deserializer

protected final KafkaDeserializationSchema<T> deserializer

The schema to convert between Kafka's byte messages, and Flink's objects.

构造器详细资料

FlinkKafkaConsumerBase

public FlinkKafkaConsumerBase(List<String> topics,
                              Pattern topicPattern,
                              KafkaDeserializationSchema<T> deserializer,
                              long discoveryIntervalMillis,
                              boolean useMetrics)

Base constructor.

参数:

topics - fixed list of topics to subscribe to (null, if using topic pattern)

topicPattern - the topic pattern to subscribe to (null, if using fixed topics)

deserializer - The deserializer to turn raw byte messages into Java/Scala objects.

discoveryIntervalMillis - the topic / partition discovery interval, in milliseconds (0 if discovery is disabled).

方法详细资料

adjustAutoCommitConfig

protected static void adjustAutoCommitConfig(Properties properties,
                                             OffsetCommitMode offsetCommitMode)

Make sure that auto commit is disabled when our offset commit mode is ON_CHECKPOINTS. This overwrites whatever setting the user configured in the properties.

参数:

properties - - Kafka configuration properties to be adjusted

offsetCommitMode - offset commit mode

assignTimestampsAndWatermarks

public FlinkKafkaConsumerBase<T> assignTimestampsAndWatermarks(org.apache.flink.api.common.eventtime.WatermarkStrategy<T> watermarkStrategy)

Sets the given WatermarkStrategy on this consumer. These will be used to assign timestamps to records and generates watermarks to signal event time progress.

Running timestamp extractors / watermark generators directly inside the Kafka source (which you can do by using this method), per Kafka partition, allows users to let them exploit the per-partition characteristics.

When a subtask of a FlinkKafkaConsumer source reads multiple Kafka partitions, the streams from the partitions are unioned in a "first come first serve" fashion. Per-partition characteristics are usually lost that way. For example, if the timestamps are strictly ascending per Kafka partition, they will not be strictly ascending in the resulting Flink DataStream, if the parallel source subtask reads more than one partition.

Common watermark generation patterns can be found as static methods in the WatermarkStrategy class.

返回:

The consumer object, to allow function chaining.

assignTimestampsAndWatermarks

@Deprecated
public FlinkKafkaConsumerBase<T> assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T> assigner)

已过时。 Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.

Specifies an AssignerWithPunctuatedWatermarks to emit watermarks in a punctuated manner. The watermark extractor will run per Kafka partition, watermarks will be merged across partitions in the same way as in the Flink runtime, when streams are merged.

When a subtask of a FlinkKafkaConsumer source reads multiple Kafka partitions, the streams from the partitions are unioned in a "first come first serve" fashion. Per-partition characteristics are usually lost that way. For example, if the timestamps are strictly ascending per Kafka partition, they will not be strictly ascending in the resulting Flink DataStream, if the parallel source subtask reads more than one partition.

Running timestamp extractors / watermark generators directly inside the Kafka source, per Kafka partition, allows users to let them exploit the per-partition characteristics.

Note: One can use either an AssignerWithPunctuatedWatermarks or an AssignerWithPeriodicWatermarks, not both at the same time.

This method uses the deprecated watermark generator interfaces. Please switch to assignTimestampsAndWatermarks(WatermarkStrategy) to use the new interfaces instead. The new interfaces support watermark idleness and no longer need to differentiate between "periodic" and "punctuated" watermarks.

参数:

assigner - The timestamp assigner / watermark generator to use.

返回:

The consumer object, to allow function chaining.

assignTimestampsAndWatermarks

@Deprecated
public FlinkKafkaConsumerBase<T> assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T> assigner)

已过时。 Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.

Specifies an AssignerWithPunctuatedWatermarks to emit watermarks in a punctuated manner. The watermark extractor will run per Kafka partition, watermarks will be merged across partitions in the same way as in the Flink runtime, when streams are merged.

When a subtask of a FlinkKafkaConsumer source reads multiple Kafka partitions, the streams from the partitions are unioned in a "first come first serve" fashion. Per-partition characteristics are usually lost that way. For example, if the timestamps are strictly ascending per Kafka partition, they will not be strictly ascending in the resulting Flink DataStream, if the parallel source subtask reads more that one partition.

Running timestamp extractors / watermark generators directly inside the Kafka source, per Kafka partition, allows users to let them exploit the per-partition characteristics.

Note: One can use either an AssignerWithPunctuatedWatermarks or an AssignerWithPeriodicWatermarks, not both at the same time.

This method uses the deprecated watermark generator interfaces. Please switch to assignTimestampsAndWatermarks(WatermarkStrategy) to use the new interfaces instead. The new interfaces support watermark idleness and no longer need to differentiate between "periodic" and "punctuated" watermarks.

参数:

assigner - The timestamp assigner / watermark generator to use.

返回:

The consumer object, to allow function chaining.

setCommitOffsetsOnCheckpoints

public FlinkKafkaConsumerBase<T> setCommitOffsetsOnCheckpoints(boolean commitOnCheckpoints)

Specifies whether or not the consumer should commit offsets back to Kafka on checkpoints.

This setting will only have effect if checkpointing is enabled for the job. If checkpointing isn't enabled, only the "auto.commit.enable" (for 0.8) / "enable.auto.commit" (for 0.9+) property settings will be used.

返回:

The consumer object, to allow function chaining.

setStartFromEarliest

public FlinkKafkaConsumerBase<T> setStartFromEarliest()

Specifies the consumer to start reading from the earliest offset for all partitions. This lets the consumer ignore any committed group offsets in Zookeeper / Kafka brokers.

This method does not affect where partitions are read from when the consumer is restored from a checkpoint or savepoint. When the consumer is restored from a checkpoint or savepoint, only the offsets in the restored state will be used.

返回:

The consumer object, to allow function chaining.

setStartFromLatest

public FlinkKafkaConsumerBase<T> setStartFromLatest()

Specifies the consumer to start reading from the latest offset for all partitions. This lets the consumer ignore any committed group offsets in Zookeeper / Kafka brokers.

This method does not affect where partitions are read from when the consumer is restored from a checkpoint or savepoint. When the consumer is restored from a checkpoint or savepoint, only the offsets in the restored state will be used.

返回:

The consumer object, to allow function chaining.

setStartFromTimestamp

public FlinkKafkaConsumerBase<T> setStartFromTimestamp(long startupOffsetsTimestamp)

Specifies the consumer to start reading partitions from a specified timestamp. The specified timestamp must be before the current timestamp. This lets the consumer ignore any committed group offsets in Zookeeper / Kafka brokers.

The consumer will look up the earliest offset whose timestamp is greater than or equal to the specific timestamp from Kafka. If there's no such offset, the consumer will use the latest offset to read data from kafka.

This method does not affect where partitions are read from when the consumer is restored from a checkpoint or savepoint. When the consumer is restored from a checkpoint or savepoint, only the offsets in the restored state will be used.

参数:

startupOffsetsTimestamp - timestamp for the startup offsets, as milliseconds from epoch.

返回:

The consumer object, to allow function chaining.

setStartFromGroupOffsets

public FlinkKafkaConsumerBase<T> setStartFromGroupOffsets()

Specifies the consumer to start reading from any committed group offsets found in Zookeeper / Kafka brokers. The "group.id" property must be set in the configuration properties. If no offset can be found for a partition, the behaviour in "auto.offset.reset" set in the configuration properties will be used for the partition.

This method does not affect where partitions are read from when the consumer is restored from a checkpoint or savepoint. When the consumer is restored from a checkpoint or savepoint, only the offsets in the restored state will be used.

返回:

The consumer object, to allow function chaining.

setStartFromSpecificOffsets

public FlinkKafkaConsumerBase<T> setStartFromSpecificOffsets(Map<KafkaTopicPartition,Long> specificStartupOffsets)

Specifies the consumer to start reading partitions from specific offsets, set independently for each partition. The specified offset should be the offset of the next record that will be read from partitions. This lets the consumer ignore any committed group offsets in Zookeeper / Kafka brokers.

If the provided map of offsets contains entries whose KafkaTopicPartition is not subscribed by the consumer, the entry will be ignored. If the consumer subscribes to a partition that does not exist in the provided map of offsets, the consumer will fallback to the default group offset behaviour (see setStartFromGroupOffsets()) for that particular partition.

If the specified offset for a partition is invalid, or the behaviour for that partition is defaulted to group offsets but still no group offset could be found for it, then the "auto.offset.reset" behaviour set in the configuration properties will be used for the partition

This method does not affect where partitions are read from when the consumer is restored from a checkpoint or savepoint. When the consumer is restored from a checkpoint or savepoint, only the offsets in the restored state will be used.

返回:

The consumer object, to allow function chaining.

disableFilterRestoredPartitionsWithSubscribedTopics

public FlinkKafkaConsumerBase<T> disableFilterRestoredPartitionsWithSubscribedTopics()

By default, when restoring from a checkpoint / savepoint, the consumer always ignores restored partitions that are no longer associated with the current specified topics or topic pattern to subscribe to.

This method configures the consumer to not filter the restored partitions, therefore always attempting to consume whatever partition was present in the previous execution regardless of the specified topics to subscribe to in the current execution.

返回:

The consumer object, to allow function chaining.

open

public void open(org.apache.flink.configuration.Configuration configuration)
          throws Exception

指定者:

open 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

open 在类中 org.apache.flink.api.common.functions.AbstractRichFunction

抛出:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext)
         throws Exception

指定者:

run 在接口中 org.apache.flink.streaming.api.functions.source.SourceFunction<T>

抛出:

Exception

cancel

public void cancel()

指定者:

cancel 在接口中 org.apache.flink.streaming.api.functions.source.SourceFunction<T>

close

public void close()
           throws Exception

指定者:

close 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

close 在类中 org.apache.flink.api.common.functions.AbstractRichFunction

抛出:

Exception

initializeState

public final void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                           throws Exception

指定者:

initializeState 在接口中 org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

抛出:

Exception

snapshotState

public final void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                         throws Exception

指定者:

snapshotState 在接口中 org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

抛出:

Exception

notifyCheckpointComplete

public final void notifyCheckpointComplete(long checkpointId)
                                    throws Exception

指定者:

notifyCheckpointComplete 在接口中 org.apache.flink.api.common.state.CheckpointListener

抛出:

Exception

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

指定者:

notifyCheckpointAborted 在接口中 org.apache.flink.api.common.state.CheckpointListener

createFetcher

protected abstract AbstractFetcher<T,?> createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                                                      Map<KafkaTopicPartition,Long> subscribedPartitionsToStartOffsets,
                                                      org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy,
                                                      org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext,
                                                      OffsetCommitMode offsetCommitMode,
                                                      org.apache.flink.metrics.MetricGroup kafkaMetricGroup,
                                                      boolean useMetrics)
                                               throws Exception

Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the data, and emits it into the data streams.

参数:

sourceContext - The source context to emit data to.

subscribedPartitionsToStartOffsets - The set of partitions that this subtask should handle, with their start offsets.

watermarkStrategy - Optional, a serialized WatermarkStrategy.

runtimeContext - The task's runtime context.

返回:

The instantiated fetcher

抛出:

Exception - The method should forward exceptions

createPartitionDiscoverer

protected abstract AbstractPartitionDiscoverer createPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor,
                                                                         int indexOfThisSubtask,
                                                                         int numParallelSubtasks)

Creates the partition discoverer that is used to find new partitions for this subtask.

参数:

topicsDescriptor - Descriptor that describes whether we are discovering partitions for fixed topics or a topic pattern.

indexOfThisSubtask - The index of this consumer subtask.

numParallelSubtasks - The total number of parallel consumer subtasks.

返回:

The instantiated partition discoverer

getIsAutoCommitEnabled

protected abstract boolean getIsAutoCommitEnabled()

fetchOffsetsWithTimestamp

protected abstract Map<KafkaTopicPartition,Long> fetchOffsetsWithTimestamp(Collection<KafkaTopicPartition> partitions,
                                                                           long timestamp)

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

指定者:

getProducedType 在接口中 org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

getEnableCommitOnCheckpoints

@VisibleForTesting
public boolean getEnableCommitOnCheckpoints()