scala implicit - implicit parameters spark 應用

 

語法

class PreferredPrompt(val preference: String)

object Greeter {

   def greet(name:String)(implicit prompt: PreferredPrompt) {

     println("Welcome, " + name + ". Thee systeme is ready.")  
     println(prompt.preference)
   }
}

implicit val prompt = new PreferredPrompt("Yes, master> ")

Greeter.greet("Joe")

 

應用

RequestContext 傳遞

java編程中有使用ThreadLocal來保存當前請求，避免每個方法都帶上context；

scala通過implict，減少boilerplate code 

abstract class RequestHandler {
      def process(context:RequstContext)
}

//main handler class
def dispatch(request:Request) {
   implicit val context = makeContext(request)

   handler1.process
   handler2.process
   handler3.process

}

 

SparkSession

spark session 對象貫穿整個job/application，通過implicit 減少每個方法調用時顯示傳入

 

  def sql2df(sql:String, view:Option[String] = None, isCache:Boolean = true)(implicit spark:SparkSession):DataFrame = {
    println(sql)
    val df = spark.sql(sql)

    if(isCache)
      df.cache()

    if(view.isDefined)
      df.createOrReplaceTempView(view.get)

    df
  }