本篇博客將詳細探討DStream模板下的RDD是如何被創建，然後被執行的。在開始敘述之前，先來思考幾個問題，本篇文章也就是基於此問題構建的。
1. RDD是誰產生的？
2. 如何產生RDD？
帶着這兩個問題開啓我們的探索之旅。
一：實戰WordCount源碼如下：
object WordCount { def main(args:Array[String]): Unit ={
val sparkConf = new SparkConf().setMaster("Master:7077").setAppName("WordCount")
val ssc = new StreamingContext(sparkConf,Seconds(1))
val lines = ssc.socketTextStream("Master",9999)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x,1)).reduceByKey(_+_)
wordCounts.print()
ssc.start()
ssc.awaitTermination()
}
}Dstream之間是有依賴關係。比如map操作，產生MappedDStream.
/** Return a new DStream by applying a function to all elements of this DStream. */def map[U: ClassTag](mapFunc: T => U): DStream[U] = ssc.withScope { new MappedDStream(this, context.sparkContext.clean(mapFunc))
}2. MappedDStream中的compute方法，會先獲取parent Dstream.然後基於其結果進行map操作，其中mapFunc就是我們傳入的業務邏輯。
private[streaming]class MappedDStream[T: ClassTag, U: ClassTag] (
parent: DStream[T],
mapFunc: T => U
) extends DStream[U](parent.ssc) {
override def dependencies: List[DStream[_]] = List(parent)
override def slideDuration: Duration = parent.slideDuration
override def compute(validTime: Time): Option[RDD[U]] = {
parent.getOrCompute(validTime).map(_.map[U](mapFunc))
}
}3. DStream: a) 每個DStream之間有依賴關係，除了第一個DStream是基於數據源產生，其他DStream均依賴於前面的DStream. b) DStream基於時間產生RDD。
* DStreams internally is characterized by a few basic properties:
* - A list of other DStreams that the DStream depends on
* - A time interval at which the DStream generates an RDD
* - A function that is used to generate an RDD after each time interval
*/
abstract class DStream[T: ClassTag] (
@transient private[streaming] var ssc: StreamingContext
) extends Serializable with Logging {至此，我們就知道了，RDD是DStream產生的，那麼DStream是如何產生RDD的呢？
DStream中的generatedRDDs的HashMap中每個Time都會產生一個RDD，而每個RDD都對應着一個Job，因爲此時的RDD就是整個DStream操作的時間間隔的最後一個RDD，而最後一個RDD和前面的RDD是有依賴關係。
// RDDs generated, marked as private[streaming] so that testsuites can access it@transientprivate[streaming] var generatedRDDs = new HashMap[Time, RDD[T]] ()
generatedRDDs是DStream的成員，說明DStream的實例中均有此成員，但是實質在運行的時候指抓住最後一個DStream的句柄。
generatedRDDs在哪裏被實例化的？搞清楚了這裏的HashMap在哪裏被實例化的話，就知道RDD是怎麼產生的。
1. DStream中的getOrCompute會根據時間生成RDD。
/**
* Get the RDD corresponding to the given time; either retrieve it from cache
* or compute-and-cache it.
*/private[streaming] final def getOrCompute(time: Time): Option[RDD[T]] = { // If RDD was already generated, then retrieve it from HashMap,
// or else compute the RDD
generatedRDDs.get(time).orElse { // Compute the RDD if time is valid (e.g. correct time in a sliding window)
// of RDD generation, else generate nothing.
if (isTimeValid(time)) {
val rddOption = createRDDWithLocalProperties(time, displayInnerRDDOps = false) { // Disable checks for existing output directories in jobs launched by the streaming
// scheduler, since we may need to write output to an existing directory during checkpoint
// recovery; see SPARK-4835 for more details. We need to have this call here because
// compute() might cause Spark jobs to be launched.
PairRDDFunctions.disableOutputSpecValidation.withValue(true) {//compute根據時間計算產生RDD
compute(time)
}
}//rddOption裏面有RDD生成的邏輯，然後生成的RDD，會put到generatedRDDs中
rddOption.foreach { case newRDD => // Register the generated RDD for caching and checkpointing
if (storageLevel != StorageLevel.NONE) {
newRDD.persist(storageLevel)
logDebug(s"Persisting RDD ${newRDD.id} for time $time to $storageLevel")
} if (checkpointDuration != null && (time - zeroTime).isMultipleOf(checkpointDuration)) {
newRDD.checkpoint()
logInfo(s"Marking RDD ${newRDD.id} for time $time for checkpointing")
}
generatedRDDs.put(time, newRDD)
}
rddOption
} else {
None
}
}
}2. 在ReceiverInputDStream中compute源碼如下：ReceiverInputDStream會生成計算鏈條中的首個RDD。後面的RDD就會依賴此RDD。
/**
* Generates RDDs with blocks received by the receiver of this stream. */override def compute(validTime: Time): Option[RDD[T]] = { val blockRDD = { if (validTime < graph.startTime) { // If this is called for any time before the start time of the context,
// then this returns an empty RDD. This may happen when recovering from a
// driver failure without any write ahead log to recover pre-failure data.//如果沒有輸入數據會產生一系列空的RDD
new BlockRDD[T](ssc.sc, Array.empty)
} else { // Otherwise, ask the tracker for all the blocks that have been allocated to this stream
// for this batch// receiverTracker會跟蹤數據
val receiverTracker = ssc.scheduler.receiverTracker// blockInfos
val blockInfos = receiverTracker.getBlocksOfBatch(validTime).getOrElse(id, Seq.empty) // Register the input blocks information into InputInfoTracker
val inputInfo = StreamInputInfo(id, blockInfos.flatMap(_.numRecords).sum)
ssc.scheduler.inputInfoTracker.reportInfo(validTime, inputInfo)// validTime是
// Create the BlockRDD
createBlockRDD(validTime, blockInfos)
}
}
Some(blockRDD)
}3. createBlockRDD源碼如下：
private[streaming] def createBlockRDD(time: Time, blockInfos: Seq[ReceivedBlockInfo]): RDD[T] = {
if (blockInfos.nonEmpty) {
val blockIds = blockInfos.map { _.blockId.asInstanceOf[BlockId] }.toArray
// Are WAL record handles present with all the blocks
val areWALRecordHandlesPresent = blockInfos.forall { _.walRecordHandleOption.nonEmpty }
if (areWALRecordHandlesPresent) {
// If all the blocks have WAL record handle, then create a WALBackedBlockRDD
val isBlockIdValid = blockInfos.map { _.isBlockIdValid() }.toArray
val walRecordHandles = blockInfos.map { _.walRecordHandleOption.get }.toArray
new WriteAheadLogBackedBlockRDD[T](
ssc.sparkContext, blockIds, walRecordHandles, isBlockIdValid)
} else {
// Else, create a BlockRDD. However, if there are some blocks with WAL info but not
// others then that is unexpected and log a warning accordingly.
if (blockInfos.find(_.walRecordHandleOption.nonEmpty).nonEmpty) {
if (WriteAheadLogUtils.enableReceiverLog(ssc.conf)) {
logError("Some blocks do not have Write Ahead Log information; " + "this is unexpected and data may not be recoverable after driver failures")
} else {
logWarning("Some blocks have Write Ahead Log information; this is unexpected")
}
}
//校驗數據是否還存在，不存在就過濾掉，此時的master是BlockManager
val validBlockIds = blockIds.filter { id =>
ssc.sparkContext.env.blockManager.master.contains(id)
}
if (validBlockIds.size != blockIds.size) {
logWarning("Some blocks could not be recovered as they were not found in memory. " + "To prevent such data loss, enabled Write Ahead Log (see programming guide " + "for more details.")
}
new BlockRDD[T](ssc.sc, validBlockIds)
}
} else {
// If no block is ready now, creating WriteAheadLogBackedBlockRDD or BlockRDD
// according to the configuration
if (WriteAheadLogUtils.enableReceiverLog(ssc.conf)) {
new WriteAheadLogBackedBlockRDD[T](
ssc.sparkContext, Array.empty, Array.empty, Array.empty)
} else {
new BlockRDD[T](ssc.sc, Array.empty)
}
}
}4. map算子操作，產生MappedDStream。
/** Return a new DStream by applying a function to all elements of this DStream. */def map[U: ClassTag](mapFunc: T => U): DStream[U] = ssc.withScope { new MappedDStream(this, context.sparkContext.clean(mapFunc))
}5. MappedDStream源碼如下：除了第一個DStream產生RDD之外，其他的DStream都是從前面DStream產生的RDD開始計算，然後返回RDD，因此，對DStream的transformations操作就是對RDD進行transformations操作。
private[streaming]class MappedDStream[T: ClassTag, U: ClassTag] (
parent: DStream[T],
mapFunc: T => U
) extends DStream[U](parent.ssc) {
override def dependencies: List[DStream[_]] = List(parent)
override def slideDuration: Duration = parent.slideDuration
//parent就是父DStream
override def compute(validTime: Time): Option[RDD[U]] = {
// getOrCompute是對RDD進行操作，後面的map就是對RDD進行操作
//DStream裏面的計算其實是對RDD進行計算，而mapFunc就是我們要操作的具體業務邏輯。
parent.getOrCompute(validTime).map(_.map[U](mapFunc))
}
}6. forEachDStream的源碼如下：
/**
* An internal DStream used to represent output operations like DStream.foreachRDD.
* @param parent Parent DStream
* @param foreachFunc Function to apply on each RDD generated by the parent DStream
* @param displayInnerRDDOps Whether the detailed callsites and scopes of the RDDs generated
* by `foreachFunc` will be displayed in the UI; only the scope and
* callsite of `DStream.foreachRDD` will be displayed.
*/
private[streaming]class ForEachDStream[T: ClassTag] (
parent: DStream[T],
foreachFunc: (RDD[T], Time) => Unit,
displayInnerRDDOps: Boolean
) extends DStream[Unit](parent.ssc) {
override def dependencies: List[DStream[_]] = List(parent)
override def slideDuration: Duration = parent.slideDuration
override def compute(validTime: Time): Option[RDD[Unit]] = None
override def generateJob(time: Time): Option[Job] = {
parent.getOrCompute(time) match {
case Some(rdd) =>
val jobFunc = () => createRDDWithLocalProperties(time, displayInnerRDDOps) {
foreachFunc(rdd, time)
}
//此時考慮jobFunc中一定有action操作
//因此jobFunc被調用的時候就會觸發action操作
Some(new Job(time, jobFunc))
case None => None
}
}
}7. 在上述案例中print函數源碼如下，foreachFunc函數中直接對RDD進行操作。
/**
* Print the first num elements of each RDD generated in this DStream. This is an output
* operator, so this DStream will be registered as an output stream and there materialized.
*/def print(num: Int): Unit = ssc.withScope { def foreachFunc: (RDD[T], Time) => Unit = {
(rdd: RDD[T], time: Time) => {
//action操作
val firstNum = rdd.take(num + 1)
// scalastyle:off println
println("-------------------------------------------")
println("Time: " + time)
println("-------------------------------------------")
firstNum.take(num).foreach(println) if (firstNum.length > num) println("...")
println()
// scalastyle:on println
}
}
foreachRDD(context.sparkContext.clean(foreachFunc), displayInnerRDDOps = false)
}上述都是從邏輯方面把RDD的生成流程走了一遍，下面我們就看正在開始是在哪裏觸發的。
在JobGenerator中generateJobs源碼如下：
/** Generate jobs and perform checkpoint for the given `time`. */private def generateJobs(time: Time) {
// Set the SparkEnv in this thread, so that job generation code can access the environment
// Example: BlockRDDs are created in this thread, and it needs to access BlockManager
// Update: This is probably redundant after threadlocal stuff in SparkEnv has been removed.
SparkEnv.set(ssc.env)
Try {
jobScheduler.receiverTracker.allocateBlocksToBatch(time) // allocate received blocks to batch
//生成Job
graph.generateJobs(time) // generate jobs using allocated block
} match { case Success(jobs) =>
val streamIdToInputInfos = jobScheduler.inputInfoTracker.getInfo(time)
jobScheduler.submitJobSet(JobSet(time, jobs, streamIdToInputInfos)) case Failure(e) =>
jobScheduler.reportError("Error generating jobs for time " + time, e)
}
eventLoop.post(DoCheckpoint(time, clearCheckpointDataLater = false))
}2. 在DStreamGraph中我們前面分析的RDD的產生的動作正在被觸發了。
def generateJobs(time: Time): Seq[Job] = {
logDebug("Generating jobs for time " + time)
val jobs = this.synchronized {
//此時的outputStream就是forEachDStream
outputStreams.flatMap { outputStream =>
val jobOption = outputStream.generateJob(time)
jobOption.foreach(_.setCallSite(outputStream.creationSite))
jobOption
}
}
logDebug("Generated " + jobs.length + " jobs for time " + time)
jobs
}RDD的創建和執行流程如下：
備註：
1、DT大數據夢工廠微信公衆號DT_Spark
2、IMF晚8點大數據實戰YY直播頻道號：68917580
3、新浪微博: http://www.weibo.com/ilovepains
本文轉自http://blog.csdn.net/snail_gesture/article/details/51448168