org.apache.flink.connector.base.source.hybrid

类 HybridSource<T>

java.lang.Object

org.apache.flink.connector.base.source.hybrid.HybridSource<T>

所有已实现的接口:

Serializable, org.apache.flink.api.connector.source.Source<T,HybridSourceSplit,HybridSourceEnumeratorState>, org.apache.flink.api.connector.source.SourceReaderFactory<T,HybridSourceSplit>

@PublicEvolving
public class HybridSource<T>
extends Object
implements org.apache.flink.api.connector.source.Source<T,HybridSourceSplit,HybridSourceEnumeratorState>

Hybrid source that switches underlying sources based on configured source chain.

A simple example with FileSource and KafkaSource with fixed Kafka start position:

 FileSource<String> fileSource =
   FileSource.forRecordStreamFormat(new TextLineInputFormat(), Path.fromLocalFile(testDir)).build();
 KafkaSource<String> kafkaSource =
           KafkaSource.<String>builder()
                   .setBootstrapServers("localhost:9092")
                   .setGroupId("MyGroup")
                   .setTopics(Arrays.asList("quickstart-events"))
                   .setDeserializer(
                           KafkaRecordDeserializer.valueOnly(StringDeserializer.class))
                   .setStartingOffsets(OffsetsInitializer.earliest())
                   .build();
 HybridSource<String> hybridSource =
           HybridSource.builder(fileSource)
                   .addSource(kafkaSource)
                   .build();
 

A more complex example with Kafka start position derived from previous source:

 HybridSource<String> hybridSource =
     HybridSource.<String, StaticFileSplitEnumerator>builder(fileSource)
         .addSource(
             switchContext -> {
               StaticFileSplitEnumerator previousEnumerator =
                   switchContext.getPreviousEnumerator();
               // how to get timestamp depends on specific enumerator
               long timestamp = previousEnumerator.getEndTimestamp();
               OffsetsInitializer offsets =
                   OffsetsInitializer.timestamp(timestamp);
               KafkaSource<String> kafkaSource =
                   KafkaSource.<String>builder()
                       .setBootstrapServers("localhost:9092")
                       .setGroupId("MyGroup")
                       .setTopics(Arrays.asList("quickstart-events"))
                       .setDeserializer(
                           KafkaRecordDeserializer.valueOnly(StringDeserializer.class))
                       .setStartingOffsets(offsets)
                       .build();
               return kafkaSource;
             },
             Boundedness.CONTINUOUS_UNBOUNDED)
         .build();
 

另请参阅:

序列化表格

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	HybridSource.HybridSourceBuilder<T,EnumT extends org.apache.flink.api.connector.source.SplitEnumerator> Builder for HybridSource.
static interface	HybridSource.SourceFactory<T,SourceT extends org.apache.flink.api.connector.source.Source<T,?,?>,FromEnumT extends org.apache.flink.api.connector.source.SplitEnumerator> Factory for underlying sources of HybridSource.
static interface	HybridSource.SourceSwitchContext<EnumT> Context provided to source factory.

构造器概要

构造器

限定符	构造器和说明
protected	HybridSource(List<org.apache.flink.connector.base.source.hybrid.HybridSource.SourceListEntry> sources) Protected for subclass, use builder(Source) to construct source.

方法概要

限定符和类型	方法和说明
static <T,EnumT extends org.apache.flink.api.connector.source.SplitEnumerator> HybridSource.HybridSourceBuilder<T,EnumT>	builder(org.apache.flink.api.connector.source.Source<T,?,?> firstSource) Builder for HybridSource.
org.apache.flink.api.connector.source.SplitEnumerator<HybridSourceSplit,HybridSourceEnumeratorState>	createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<HybridSourceSplit> enumContext)
org.apache.flink.api.connector.source.SourceReader<T,HybridSourceSplit>	createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext)
org.apache.flink.api.connector.source.Boundedness	getBoundedness()
org.apache.flink.core.io.SimpleVersionedSerializer<HybridSourceEnumeratorState>	getEnumeratorCheckpointSerializer()
org.apache.flink.core.io.SimpleVersionedSerializer<HybridSourceSplit>	getSplitSerializer()
org.apache.flink.api.connector.source.SplitEnumerator<HybridSourceSplit,HybridSourceEnumeratorState>	restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<HybridSourceSplit> enumContext, HybridSourceEnumeratorState checkpoint)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

HybridSource

protected HybridSource(List<org.apache.flink.connector.base.source.hybrid.HybridSource.SourceListEntry> sources)

Protected for subclass, use builder(Source) to construct source.

方法详细资料

builder

public static <T,EnumT extends org.apache.flink.api.connector.source.SplitEnumerator> HybridSource.HybridSourceBuilder<T,EnumT> builder(org.apache.flink.api.connector.source.Source<T,?,?> firstSource)

Builder for HybridSource.

getBoundedness

public org.apache.flink.api.connector.source.Boundedness getBoundedness()

指定者:

getBoundedness 在接口中 org.apache.flink.api.connector.source.Source<T,HybridSourceSplit,HybridSourceEnumeratorState>

createReader

@Internal
public org.apache.flink.api.connector.source.SourceReader<T,HybridSourceSplit> createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext)
                                                                                               throws Exception

指定者:

createReader 在接口中 org.apache.flink.api.connector.source.SourceReaderFactory<T,HybridSourceSplit>

抛出:

Exception

createEnumerator

@Internal
public org.apache.flink.api.connector.source.SplitEnumerator<HybridSourceSplit,HybridSourceEnumeratorState> createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<HybridSourceSplit> enumContext)

指定者:

createEnumerator 在接口中 org.apache.flink.api.connector.source.Source<T,HybridSourceSplit,HybridSourceEnumeratorState>

restoreEnumerator

@Internal
public org.apache.flink.api.connector.source.SplitEnumerator<HybridSourceSplit,HybridSourceEnumeratorState> restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<HybridSourceSplit> enumContext,
                                                                                                                                        HybridSourceEnumeratorState checkpoint)
                                                                                                                                 throws Exception

指定者:

restoreEnumerator 在接口中 org.apache.flink.api.connector.source.Source<T,HybridSourceSplit,HybridSourceEnumeratorState>

抛出:

Exception

getSplitSerializer

@Internal
public org.apache.flink.core.io.SimpleVersionedSerializer<HybridSourceSplit> getSplitSerializer()

指定者:

getSplitSerializer 在接口中 org.apache.flink.api.connector.source.Source<T,HybridSourceSplit,HybridSourceEnumeratorState>

getEnumeratorCheckpointSerializer

@Internal
public org.apache.flink.core.io.SimpleVersionedSerializer<HybridSourceEnumeratorState> getEnumeratorCheckpointSerializer()

指定者:

getEnumeratorCheckpointSerializer 在接口中 org.apache.flink.api.connector.source.Source<T,HybridSourceSplit,HybridSourceEnumeratorState>