org.apache.flink.streaming.connectors.kafka.table

类 KafkaDynamicSource

java.lang.Object

org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSource

所有已实现的接口:

org.apache.flink.table.connector.source.abilities.SupportsReadingMetadata, org.apache.flink.table.connector.source.abilities.SupportsWatermarkPushDown, org.apache.flink.table.connector.source.DynamicTableSource, org.apache.flink.table.connector.source.ScanTableSource

@Internal
public class KafkaDynamicSource
extends Object
implements org.apache.flink.table.connector.source.ScanTableSource, org.apache.flink.table.connector.source.abilities.SupportsReadingMetadata, org.apache.flink.table.connector.source.abilities.SupportsWatermarkPushDown

A version-agnostic Kafka ScanTableSource.

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.table.connector.source.ScanTableSource

org.apache.flink.table.connector.source.ScanTableSource.ScanContext, org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider

从接口继承的嵌套类/接口 org.apache.flink.table.connector.source.DynamicTableSource

org.apache.flink.table.connector.source.DynamicTableSource.Context, org.apache.flink.table.connector.source.DynamicTableSource.DataStructureConverter

字段概要

字段

限定符和类型	字段和说明
protected BoundedMode	boundedMode The bounded mode for the contained consumer (default is an unbounded data stream).
protected long	boundedTimestampMillis The bounded timestamp to locate partition offsets; only relevant when bounded mode is BoundedMode.TIMESTAMP.
protected org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>>	keyDecodingFormat Optional format for decoding keys from Kafka.
protected String	keyPrefix Prefix that needs to be removed from fields when constructing the physical data type.
protected int[]	keyProjection Indices that determine the key fields and the target position in the produced row.
protected List<String>	metadataKeys Metadata that is appended at the end of a physical source row.
protected org.apache.flink.table.types.DataType	physicalDataType Data type to configure the formats.
protected org.apache.flink.table.types.DataType	producedDataType Data type that describes the final output of the source.
protected Properties	properties Properties for the Kafka consumer.
protected Map<KafkaTopicPartition,Long>	specificBoundedOffsets Specific end offsets; only relevant when bounded mode is BoundedMode.SPECIFIC_OFFSETS.
protected Map<KafkaTopicPartition,Long>	specificStartupOffsets Specific startup offsets; only relevant when startup mode is StartupMode.SPECIFIC_OFFSETS.
protected StartupMode	startupMode The startup mode for the contained consumer (default is StartupMode.GROUP_OFFSETS).
protected long	startupTimestampMillis The start timestamp to locate partition offsets; only relevant when startup mode is StartupMode.TIMESTAMP.
protected String	tableIdentifier
protected Pattern	topicPattern The Kafka topic pattern to consume.
protected List<String>	topics The Kafka topics to consume.
protected boolean	upsertMode Flag to determine source mode.
protected org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>>	valueDecodingFormat Format for decoding values from Kafka.
protected int[]	valueProjection Indices that determine the value fields and the target position in the produced row.
protected org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData>	watermarkStrategy Watermark strategy that is used to generate per-partition watermark.

构造器概要

构造器

构造器和说明

KafkaDynamicSource(org.apache.flink.table.types.DataType physicalDataType, org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> keyDecodingFormat, org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> valueDecodingFormat, int[] keyProjection, int[] valueProjection, String keyPrefix, List<String> topics, Pattern topicPattern, Properties properties, StartupMode startupMode, Map<KafkaTopicPartition,Long> specificStartupOffsets, long startupTimestampMillis, BoundedMode boundedMode, Map<KafkaTopicPartition,Long> specificBoundedOffsets, long boundedTimestampMillis, boolean upsertMode, String tableIdentifier)

方法概要

限定符和类型	方法和说明
void	applyReadableMetadata(List<String> metadataKeys, org.apache.flink.table.types.DataType producedDataType)
void	applyWatermark(org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy)
String	asSummaryString()
org.apache.flink.table.connector.source.DynamicTableSource	copy()
protected KafkaSource<org.apache.flink.table.data.RowData>	createKafkaSource(org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData> keyDeserialization, org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData> valueDeserialization, org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo)
boolean	equals(Object o)
org.apache.flink.table.connector.ChangelogMode	getChangelogMode()
org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider	getScanRuntimeProvider(org.apache.flink.table.connector.source.ScanTableSource.ScanContext context)
int	hashCode()
Map<String,org.apache.flink.table.types.DataType>	listReadableMetadata()
boolean	supportsMetadataProjection()

从类继承的方法 java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

字段详细资料

producedDataType

protected org.apache.flink.table.types.DataType producedDataType

Data type that describes the final output of the source.

metadataKeys

protected List<String> metadataKeys

Metadata that is appended at the end of a physical source row.

watermarkStrategy

@Nullable
protected org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy

Watermark strategy that is used to generate per-partition watermark.

physicalDataType

protected final org.apache.flink.table.types.DataType physicalDataType

Data type to configure the formats.

keyDecodingFormat

@Nullable
protected final org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> keyDecodingFormat

Optional format for decoding keys from Kafka.

valueDecodingFormat

protected final org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> valueDecodingFormat

Format for decoding values from Kafka.

keyProjection

protected final int[] keyProjection

Indices that determine the key fields and the target position in the produced row.

valueProjection

protected final int[] valueProjection

Indices that determine the value fields and the target position in the produced row.

keyPrefix

@Nullable
protected final String keyPrefix

Prefix that needs to be removed from fields when constructing the physical data type.

topics

protected final List<String> topics

The Kafka topics to consume.

topicPattern

protected final Pattern topicPattern

The Kafka topic pattern to consume.

properties

protected final Properties properties

Properties for the Kafka consumer.

startupMode

protected final StartupMode startupMode

The startup mode for the contained consumer (default is StartupMode.GROUP_OFFSETS).

specificStartupOffsets

protected final Map<KafkaTopicPartition,Long> specificStartupOffsets

Specific startup offsets; only relevant when startup mode is StartupMode.SPECIFIC_OFFSETS.

startupTimestampMillis

protected final long startupTimestampMillis

The start timestamp to locate partition offsets; only relevant when startup mode is StartupMode.TIMESTAMP.

boundedMode

protected final BoundedMode boundedMode

The bounded mode for the contained consumer (default is an unbounded data stream).

specificBoundedOffsets

protected final Map<KafkaTopicPartition,Long> specificBoundedOffsets

Specific end offsets; only relevant when bounded mode is BoundedMode.SPECIFIC_OFFSETS.

boundedTimestampMillis

protected final long boundedTimestampMillis

The bounded timestamp to locate partition offsets; only relevant when bounded mode is BoundedMode.TIMESTAMP.

upsertMode

protected final boolean upsertMode

Flag to determine source mode. In upsert mode, it will keep the tombstone message. *

tableIdentifier

protected final String tableIdentifier

构造器详细资料

KafkaDynamicSource

public KafkaDynamicSource(org.apache.flink.table.types.DataType physicalDataType,
                          @Nullable
                          org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> keyDecodingFormat,
                          org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> valueDecodingFormat,
                          int[] keyProjection,
                          int[] valueProjection,
                          @Nullable
                          String keyPrefix,
                          @Nullable
                          List<String> topics,
                          @Nullable
                          Pattern topicPattern,
                          Properties properties,
                          StartupMode startupMode,
                          Map<KafkaTopicPartition,Long> specificStartupOffsets,
                          long startupTimestampMillis,
                          BoundedMode boundedMode,
                          Map<KafkaTopicPartition,Long> specificBoundedOffsets,
                          long boundedTimestampMillis,
                          boolean upsertMode,
                          String tableIdentifier)

方法详细资料

getChangelogMode

public org.apache.flink.table.connector.ChangelogMode getChangelogMode()

指定者:

getChangelogMode 在接口中 org.apache.flink.table.connector.source.ScanTableSource

getScanRuntimeProvider

public org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider getScanRuntimeProvider(org.apache.flink.table.connector.source.ScanTableSource.ScanContext context)

指定者:

getScanRuntimeProvider 在接口中 org.apache.flink.table.connector.source.ScanTableSource

listReadableMetadata

public Map<String,org.apache.flink.table.types.DataType> listReadableMetadata()

指定者:

listReadableMetadata 在接口中 org.apache.flink.table.connector.source.abilities.SupportsReadingMetadata

applyReadableMetadata

public void applyReadableMetadata(List<String> metadataKeys,
                                  org.apache.flink.table.types.DataType producedDataType)

指定者:

applyReadableMetadata 在接口中 org.apache.flink.table.connector.source.abilities.SupportsReadingMetadata

supportsMetadataProjection

public boolean supportsMetadataProjection()

指定者:

supportsMetadataProjection 在接口中 org.apache.flink.table.connector.source.abilities.SupportsReadingMetadata

applyWatermark

public void applyWatermark(org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy)

指定者:

applyWatermark 在接口中 org.apache.flink.table.connector.source.abilities.SupportsWatermarkPushDown

copy

public org.apache.flink.table.connector.source.DynamicTableSource copy()

指定者:

copy 在接口中 org.apache.flink.table.connector.source.DynamicTableSource

asSummaryString

public String asSummaryString()

指定者:

asSummaryString 在接口中 org.apache.flink.table.connector.source.DynamicTableSource

equals

public boolean equals(Object o)

覆盖:

equals 在类中 Object

hashCode

public int hashCode()

覆盖:

hashCode 在类中 Object

createKafkaSource

protected KafkaSource<org.apache.flink.table.data.RowData> createKafkaSource(org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData> keyDeserialization,
                                                                             org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData> valueDeserialization,
                                                                             org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo)