SparkSession介绍
- 官网链接
- Spark 2.x中所有功能的入口点是SparkSession类。要创建一个基本的SparkSession,只需使用SparkSession.builder():
import org.apache.spark.sql.SparkSession
val spark = SparkSession
.builder()
.appName("Spark SQL basic example")
.config("spark.some.config.option", "some-value")
.getOrCreate()
// For implicit conversions like converting RDDs to DataFrames
import spark.implicits._
实践SparkSession
import org.apache.spark.sql.{DataFrame, SparkSession}
/*
认识SparkSession
*/
object SparkSessionApp {
def main(args: Array[String]): Unit = {
// DF/DS编程的入口点
val spark : SparkSession = SparkSession.builder().master("local").getOrCreate()
val df : DataFrame = spark.read.text("file:///C:\\Users\\gosur\\IdeaProjects\\wordCount\\data\\input.txt")
//TODO... 业务逻辑处理,通过DF/DS提供的API来完成业务
df.show() // 展示出来 只有一个字段,String类型的Value
spark.stop()
}
}
+-------------+
| value|
+-------------+
| pk,pk,pk|
|jepson,jepson|
| xingxing|
+-------------+