【Flink原理和應用】：Flink的調度模式

原文：http://www.liaojiayi.com/Flink-Schedule-Mode/

Apache Flink內部提供了兩種調度模式，分別爲：

/** Schedule tasks lazy from the sources. Downstream tasks start once their input data are ready */
LAZY_FROM_SOURCES,

/** Schedules all tasks immediately. */
EAGER;

LAZY_FROM_SOURCES

LAZY_FROM_SOURCES的原理我們可以進入到scheduleLazy的方法裏看一下：

private CompletableFuture<Void> scheduleLazy(SlotProvider slotProvider) {

    final ArrayList<CompletableFuture<Void>> schedulingFutures = new ArrayList<>(numVerticesTotal);

    // take the vertices without inputs.
    for (ExecutionJobVertex ejv : verticesInCreationOrder) {
        if (ejv.getJobVertex().isInputVertex()) {
            final CompletableFuture<Void> schedulingJobVertexFuture = ejv.scheduleAll(
                slotProvider,
                allowQueuedScheduling,
                LocationPreferenceConstraint.ALL); // since it is an input vertex, the input based location preferences should be empty

            schedulingFutures.add(schedulingJobVertexFuture);
        }
    }

    return FutureUtils.waitForAll(schedulingFutures);
}

ExecutionJobVertex代表某個operation，如map。在這裏根據ExecutionGraph中ExecutionJobVertex的順序來依次初始化。當使用到當前task時，再去調度。

EAGER

EAGER的方式：

private CompletableFuture<Void> scheduleEager(SlotProvider slotProvider, final Time timeout) {
    checkState(state == JobStatus.RUNNING, "job is not running currently");

    // Important: reserve all the space we need up front.
    // that way we do not have any operation that can fail between allocating the slots
    // and adding them to the list. If we had a failure in between there, that would
    // cause the slots to get lost
    final boolean queued = allowQueuedScheduling;

    // collecting all the slots may resize and fail in that operation without slots getting lost
    final ArrayList<CompletableFuture<Execution>> allAllocationFutures = new ArrayList<>(getNumberOfExecutionJobVertices());

    // allocate the slots (obtain all their futures
    for (ExecutionJobVertex ejv : getVerticesTopologically()) {
        // these calls are not blocking, they only return futures
        Collection<CompletableFuture<Execution>> allocationFutures = ejv.allocateResourcesForAll(
            slotProvider,
            queued,
            LocationPreferenceConstraint.ALL,
            allocationTimeout);

        allAllocationFutures.addAll(allocationFutures);
    }

    // this future is complete once all slot futures are complete.
    // the future fails once one slot future fails.
    final ConjunctFuture<Collection<Execution>> allAllocationsFuture = FutureUtils.combineAll(allAllocationFutures);

    final CompletableFuture<Void> currentSchedulingFuture = allAllocationsFuture
        .thenAccept(
            (Collection<Execution> executionsToDeploy) -> {
                for (Execution execution : executionsToDeploy) {
                    try {
                        execution.deploy();
                    } catch (Throwable t) {
                        throw new CompletionException(
                            new FlinkException(
                                String.format("Could not deploy execution %s.", execution),
                                t));
                    }
                }
            })
        // Generate a more specific failure message for the eager scheduling
        .exceptionally(
            (Throwable throwable) -> {
                final Throwable strippedThrowable = ExceptionUtils.stripCompletionException(throwable);
                final Throwable resultThrowable;

                if (strippedThrowable instanceof TimeoutException) {
                    int numTotal = allAllocationsFuture.getNumFuturesTotal();
                    int numComplete = allAllocationsFuture.getNumFuturesCompleted();
                    String message = "Could not allocate all requires slots within timeout of " +
                        timeout + ". Slots required: " + numTotal + ", slots allocated: " + numComplete;

                    resultThrowable = new NoResourceAvailableException(message);
                } else {
                    resultThrowable = strippedThrowable;
                }

                throw new CompletionException(resultThrowable);
            });

    return currentSchedulingFuture;
}

如果採用EAGER方式的話，是先調用allocateResourcesForAll來分配資源，然後纔是把所有的task部署到對應的slot上。

不同點

這裏可以把LAZY_FROM_SOURCES理解爲，一個一個operator逐個完成，比較適合批處理模式，這種模式使得每一種operator都能最大限度的利用集羣資源。而EAGER模式比較適用於流式數據處理，因爲task正常情況下不存在退出結束的行爲。