Spark streaming消費kafka程序報No output streams registered, so nothing to execute異常

在做kafka割接測試消費程序只消費不做數據處理時編寫代碼遇到了No output streams registered, so nothing to execute異常的問題：

代碼如下：

public void execute() throws MintakaException {
        // Create a Spark Configuration
        final SparkConf sparkConf = new SparkConf();
        // backpressure
        sparkConf.set("spark.streaming.backpressure.enabled", "true");
        sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
        Class[] classes = new Class[14];
        classes[0] = Long.class;
        classes[1] = double[].class;
        classes[2] = Set.class;
        classes[3] = scala.Tuple2[].class;
        classes[4] = long[].class;
        classes[5] = HashSet.class;
        classes[6] = CustomerflowXdrCommunicationRecord.class;
        classes[7] = CustomerCellRecord.class;
        classes[8] = CustomerCellRecordDb.class;
        classes[9] = String.class;
        classes[10] = SparkStreamingMapReduceJob.class;
        classes[11] = SparkStreamingMapReduceCollection.class;
        classes[12] = CustomerflowXdrTransformationMapReduce.class;
        classes[13] = HigwayCell.class;
        sparkConf.registerKryoClasses(classes);
        // Create a Spark Streaming Context

        JavaStreamingContext javaStreamingContext = new JavaStreamingContext(sparkConf, Durations.seconds(slideInSeconds));        javaStreamingContext.sparkContext().hadoopConfiguration().set("parquet.enable.summary-metadata", "false");        javaStreamingContext.sparkContext().hadoopConfiguration().set("spark.sql.parquet.output.committer.class", "org.apache.spark.sql.parquet.DirectParquetOutputCommitter");
        javaStreamingContext.checkpoint(checkPointPath);
        Map<String, Object> kafkaParams = new HashMap();
        kafkaParams.put("bootstrap.servers", bootstrapServers);
        kafkaParams.put("value.deserializer",
                "org.apache.kafka.common.serialization.StringDeserializer");

        kafkaParams.put("key.deserializer",
                "org.apache.kafka.common.serialization.StringDeserializer");

        kafkaParams.put("group.id", kafkaConsumerGroup);
        kafkaParams.put("security.protocol", "SASL_PLAINTEXT");
        kafkaParams.put("sasl.kerberos.service.name", "kafka");
        kafkaParams.put("kerberos.domain.name", "hadoop.hadoop.com");
        kafkaParams.put("enable.auto.commit", "true");
        kafkaParams.put("auto.offset.reset", "latest");

        LocationStrategy locationStrategy = LocationStrategies.PreferConsistent();
        System.out.println("topics are : " + topicss);
        Set<String> topicSet = new HashSet<>(Arrays.asList(topicss.split(",")));
        ConsumerStrategy consumerStrategy = ConsumerStrategies.Subscribe(topicSet, kafkaParams);

        JavaInputDStream<ConsumerRecord<String, String>> messages = KafkaUtils.createDirectStream(javaStreamingContext,
                locationStrategy, consumerStrategy);

        JavaPairDStream<String, String> xdrStreams = messages.mapToPair(new PairFunction<ConsumerRecord<String, String>, String, String>() {
            @Override
            public Tuple2<String, String> call(ConsumerRecord<String, String> record)  {
                return new Tuple2<>(record.key(), record.value());
            }
        });

        JavaPairDStream<String, String>  xdrs = xdrStreams.persist(StorageLevel.MEMORY_AND_DISK_SER());

        //mapReduce.mapReduce(xdrs);
        try {
            // Spark Job Start
            javaStreamingContext.start();
            javaStreamingContext.awaitTermination();
            // TODO: check whether need to close!
            javaStreamingContext.close();
            javaStreamingContext = null;
        } catch (Exception e) {
            javaStreamingContext.close();
        }
    }

執行程序時會報如下異常（如下代碼爲出現異常部分）：

JavaPairDStream<String, String> xdrStreams = messages.mapToPair(new PairFunction<ConsumerRecord<String, String>, String, String>() {
            @Override
            public Tuple2<String, String> call(ConsumerRecord<String, String> record)  {
                return new Tuple2<>(record.key(), record.value());
            }
        });

        JavaPairDStream<String, String>  xdrs = xdrStreams.persist(StorageLevel.MEMORY_AND_DISK_SER());

 

問題原因：當發生輸出操作的時候，會觸發流計算操作，當在 DStream上沒有輸出操作或者計算時，通常需要執行某些操作。代碼中把該mapReduce.mapReduce(xdrs)方法註釋了，導致沒有輸出操作。

解決方法：爲了能夠正常運行程序，達到測試效果，只要使用以下方法之一觸發即可：

1.print()
2.foreachRDD()
3.saveAsObjectFiles()
4.saveAsTextFiles()
5.saveAsHadoopFiles()

我在代碼中加了xdrStreams.print(1)即可正常運行