org.apache.flink.connector.base.source.reader.fetcher

类 SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

java.lang.Object

org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<E,SplitT>

直接已知子类:

SingleThreadFetcherManager

@Internal
public abstract class SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>
extends Object

A class responsible for starting the SplitFetcher and manage the life cycles of them. This class works with the SourceReaderBase.

The split fetcher manager could be used to support different threading models by implementing the addSplits(List) method differently. For example, a single thread split fetcher manager would only start a single fetcher and assign all the splits to it. A one-thread-per-split fetcher may spawn a new thread every time a new split is assigned.

字段概要

字段

限定符和类型	字段和说明
protected Map<Integer,SplitFetcher<E,SplitT>>	fetchers A map keeping track of all the split fetchers.

构造器概要

构造器

构造器和说明
SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration) Create a split fetcher manager.
SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration, java.util.function.Consumer<Collection<String>> splitFinishedHook) Create a split fetcher manager.

方法概要

限定符和类型	方法和说明
abstract void	addSplits(List<SplitT> splitsToAdd)
void	checkErrors()
void	close(long timeoutMs) Close the split fetcher manager.
protected SplitFetcher<E,SplitT>	createSplitFetcher() Synchronize method to ensure no fetcher is created after the split fetcher manager has closed.
int	getNumAliveFetchers()
boolean	maybeShutdownFinishedFetchers() Check and shutdown the fetchers that have completed their work.
void	pauseOrResumeSplits(Collection<String> splitIdsToPause, Collection<String> splitIdsToResume)
protected void	startFetcher(SplitFetcher<E,SplitT> fetcher)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

fetchers

protected final Map<Integer,SplitFetcher<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>> fetchers

A map keeping track of all the split fetchers.

构造器详细资料

SplitFetcherManager

public SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                           java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderFactory,
                           org.apache.flink.configuration.Configuration configuration)

Create a split fetcher manager.

参数:

elementsQueue - the queue that split readers will put elements into.

splitReaderFactory - a supplier that could be used to create split readers.

configuration - the configuration of this fetcher manager.

SplitFetcherManager

@VisibleForTesting
public SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                              java.util.function.Supplier<SplitReader<E,SplitT>> splitReaderFactory,
                                              org.apache.flink.configuration.Configuration configuration,
                                              java.util.function.Consumer<Collection<String>> splitFinishedHook)

Create a split fetcher manager.

参数:

elementsQueue - the queue that split readers will put elements into.

splitReaderFactory - a supplier that could be used to create split readers.

configuration - the configuration of this fetcher manager.

splitFinishedHook - Hook for handling finished splits in split fetchers.

方法详细资料

addSplits

public abstract void addSplits(List<SplitT> splitsToAdd)

pauseOrResumeSplits

public void pauseOrResumeSplits(Collection<String> splitIdsToPause,
                                Collection<String> splitIdsToResume)

startFetcher

protected void startFetcher(SplitFetcher<E,SplitT> fetcher)

createSplitFetcher

protected SplitFetcher<E,SplitT> createSplitFetcher()

Synchronize method to ensure no fetcher is created after the split fetcher manager has closed.

返回:

the created split fetcher.

抛出:

IllegalStateException - if the split fetcher manager has closed.

maybeShutdownFinishedFetchers

public boolean maybeShutdownFinishedFetchers()

Check and shutdown the fetchers that have completed their work.

返回:

true if all the fetchers have completed the work, false otherwise.

close

public void close(long timeoutMs)
           throws Exception

Close the split fetcher manager.

参数:

timeoutMs - the max time in milliseconds to wait.

抛出:

Exception - when failed to close the split fetcher manager.

checkErrors

public void checkErrors()

getNumAliveFetchers

@VisibleForTesting
public int getNumAliveFetchers()