org.apache.flink.streaming.connectors.kafka

类 FlinkKafkaProducerBase<IN>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase<IN>

类型参数:

IN - Type of the messages to write into Kafka.

所有已实现的接口:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<IN>

@Internal
public abstract class FlinkKafkaProducerBase<IN>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Flink Sink to produce data into a Kafka topic.

Please note that this producer provides at-least-once reliability guarantees when checkpoints are enabled and setFlushOnCheckpoint(true) is set. Otherwise, the producer doesn't provide any reliability guarantees.

另请参阅:

序列化表格

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.sink.SinkFunction

org.apache.flink.streaming.api.functions.sink.SinkFunction.Context

字段概要

字段

限定符和类型	字段和说明
protected Exception	asyncException Errors encountered in the async producer are stored here.
protected org.apache.kafka.clients.producer.Callback	callback The callback than handles error propagation or logging callbacks.
protected String	defaultTopicId The name of the default topic this producer is writing data to.
protected FlinkKafkaPartitioner<IN>	flinkKafkaPartitioner User-provided partitioner for assigning an object to a Kafka partition for each topic.
protected boolean	flushOnCheckpoint If true, the producer will wait until all outstanding records have been send to the broker.
static String	KEY_DISABLE_METRICS Configuration key for disabling the metrics reporting.
protected boolean	logFailuresOnly Flag indicating whether to accept failures (and log them), or to fail on failures.
protected long	pendingRecords Number of unacknowledged records.
protected org.apache.flink.util.SerializableObject	pendingRecordsLock Lock for accessing the pending records.
protected org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]>	producer KafkaProducer instance.
protected Properties	producerConfig User defined properties for the Producer.
protected KeyedSerializationSchema<IN>	schema (Serializable) SerializationSchema for turning objects used with Flink into. byte[] for Kafka.
protected Map<String,int[]>	topicPartitionsMap Partitions of each topic.

构造器概要

构造器

构造器和说明
FlinkKafkaProducerBase(String defaultTopicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, FlinkKafkaPartitioner<IN> customPartitioner) The main constructor for creating a FlinkKafkaProducer.

方法概要

限定符和类型	方法和说明
protected void	checkErroneous()
void	close()
protected abstract void	flush() Flush pending records.
protected <K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V>	getKafkaProducer(Properties props) Used for testing only.
protected static int[]	getPartitionsByTopic(String topic, org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> producer)
static Properties	getPropertiesFromBrokerList(String brokerList)
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	invoke(IN next, org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context) Called when new data arrives to the sink, and forwards it to Kafka.
protected long	numPendingRecords()
void	open(org.apache.flink.configuration.Configuration configuration) Initializes the connection to Kafka.
void	setFlushOnCheckpoint(boolean flush) If set to true, the Flink producer will wait for all outstanding messages in the Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
void	setLogFailuresOnly(boolean logFailuresOnly) Defines whether the producer should fail on errors, or only log them.
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext ctx)

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.streaming.api.functions.sink.SinkFunction

finish, invoke, writeWatermark

字段详细资料

KEY_DISABLE_METRICS

public static final String KEY_DISABLE_METRICS

Configuration key for disabling the metrics reporting.

另请参阅:

常量字段值

producerConfig

protected final Properties producerConfig

User defined properties for the Producer.

defaultTopicId

protected final String defaultTopicId

The name of the default topic this producer is writing data to.

schema

protected final KeyedSerializationSchema<IN> schema

(Serializable) SerializationSchema for turning objects used with Flink into. byte[] for Kafka.

flinkKafkaPartitioner

protected final FlinkKafkaPartitioner<IN> flinkKafkaPartitioner

User-provided partitioner for assigning an object to a Kafka partition for each topic.

topicPartitionsMap

protected final Map<String,int[]> topicPartitionsMap

Partitions of each topic.

logFailuresOnly

protected boolean logFailuresOnly

Flag indicating whether to accept failures (and log them), or to fail on failures.

flushOnCheckpoint

protected boolean flushOnCheckpoint

If true, the producer will wait until all outstanding records have been send to the broker.

producer

protected transient org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> producer

KafkaProducer instance.

callback

protected transient org.apache.kafka.clients.producer.Callback callback

The callback than handles error propagation or logging callbacks.

asyncException

protected transient volatile Exception asyncException

Errors encountered in the async producer are stored here.

pendingRecordsLock

protected final org.apache.flink.util.SerializableObject pendingRecordsLock

Lock for accessing the pending records.

pendingRecords

protected long pendingRecords

Number of unacknowledged records.

构造器详细资料

FlinkKafkaProducerBase

public FlinkKafkaProducerBase(String defaultTopicId,
                              KeyedSerializationSchema<IN> serializationSchema,
                              Properties producerConfig,
                              FlinkKafkaPartitioner<IN> customPartitioner)

The main constructor for creating a FlinkKafkaProducer.

参数:

defaultTopicId - The default topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. Passing null will use Kafka's partitioner.

方法详细资料

setLogFailuresOnly

public void setLogFailuresOnly(boolean logFailuresOnly)

Defines whether the producer should fail on errors, or only log them. If this is set to true, then exceptions will be only logged, if set to false, exceptions will be eventually thrown and cause the streaming program to fail (and enter recovery).

参数:

logFailuresOnly - The flag to indicate logging-only on exceptions.

setFlushOnCheckpoint

public void setFlushOnCheckpoint(boolean flush)

If set to true, the Flink producer will wait for all outstanding messages in the Kafka buffers to be acknowledged by the Kafka producer on a checkpoint. This way, the producer can guarantee that messages in the Kafka buffers are part of the checkpoint.

参数:

flush - Flag indicating the flushing mode (true = flush on checkpoint)

getKafkaProducer

@VisibleForTesting
protected <K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V> getKafkaProducer(Properties props)

Used for testing only.

open

public void open(org.apache.flink.configuration.Configuration configuration)
          throws Exception

Initializes the connection to Kafka.

指定者:

open 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

open 在类中 org.apache.flink.api.common.functions.AbstractRichFunction

抛出:

Exception

invoke

public void invoke(IN next,
                   org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
            throws Exception

Called when new data arrives to the sink, and forwards it to Kafka.

指定者:

invoke 在接口中 org.apache.flink.streaming.api.functions.sink.SinkFunction<IN>

参数:

next - The incoming data

抛出:

Exception

close

public void close()
           throws Exception

指定者:

close 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

close 在类中 org.apache.flink.api.common.functions.AbstractRichFunction

抛出:

Exception

flush

protected abstract void flush()

Flush pending records.

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

指定者:

initializeState 在接口中 org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

抛出:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext ctx)
                   throws Exception

指定者:

snapshotState 在接口中 org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

抛出:

Exception

checkErroneous

protected void checkErroneous()
                       throws Exception

抛出:

Exception

getPropertiesFromBrokerList

public static Properties getPropertiesFromBrokerList(String brokerList)

getPartitionsByTopic

protected static int[] getPartitionsByTopic(String topic,
                                            org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> producer)

numPendingRecords

@VisibleForTesting
protected long numPendingRecords()