How to build a cluster scheduler 項目簡介

Aim / Learning objectives

The aim of this project is to explore the resource allocation problem in data-intensive systems. Schedulers need to distribute the workload across a set of resources to achieve a balance between various quality of service measures and the cost of operating a system.

Background

The background to the resource allocation problem is explained in the paper on schedulers that is already in the list of core papers for the module.

Diptanu Gon Choudhury and Timothy Perrett. Designing cluster schedulers for internet-scale services. Communications of the ACM, 61(6):34–40, May 2018

The problem to tackle in this project is a simplified version of the general problem of scheduling workloads onto a set of infrastructure components (servers).

• 本篇博客是關於"如何編寫一個集羣調度器"系列博客的背景介紹。這一系列博客將圍繞該項目進行闡述，並以實例代碼的方式逐步地編寫一個集羣調度器。在前面的博客中，我爲大家轉發了關於Apache開源的兩種資源調度框架的文章，其中就提到了"Scheduler"這個重要的組件。在這裏就不介紹了，有興趣的小夥伴可以去查閱瞭解一下。本篇博客中，我爲大家推薦一篇論文 - “Designing cluster schedulers for internet-scale services”, 該論文主要講解了什麼是"cluster scheduler" 和如何設計scheduler；論文關於設計的部分是以實際應用中所遇到的問題爲前提和角度去闡述的，通讀該論文應該能對"cluster scheduler"能有個大致的瞭解。

University of St Andrews 的"Data-intensive System"課程的一個項目