Spark 2.1.1 with CarbonData 1.1.0

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark 2.1.1 with CarbonData 1.1.0

Bill Speirs
I'm trying to follow the directions for using Spark 2.1.1 with
CarbonData 1.1.0 found here:
http://carbondata.apache.org/quick-start-guide.html

I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
-Dspark.version=2.1.1 clean package

I ran Spark with: ./bin/spark-shell --jars
../carbondata-parent-1.1.0/assembly/target/scala-2.11/carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar

I imported the two classes (not sure if the second is a typo with the
underline):

scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession

scala> import org.apache.spark.sql.CarbonSession._
import org.apache.spark.sql.CarbonSession._

Then tried to create the carbon variable:

scala> val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
<console>:28: error: value getOrCreateCarbonContext is not a member of
org.apache.spark.sql.SparkSession.Builder
       val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")

Thoughts on what I'm doing wrong here?

Bill-
Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.1.1 with CarbonData 1.1.0

Pallavi Singh
Hi Bill,

CarbonContext is only for Spark 1.6, for Spark 2.1 you should use CarbonSession. Please refer to CarbonSessionExample  for more details. 

On Thu, May 25, 2017 at 12:01 AM, Bill Speirs <[hidden email]> wrote:
I'm trying to follow the directions for using Spark 2.1.1 with
CarbonData 1.1.0 found here:
http://carbondata.apache.org/quick-start-guide.html

I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
-Dspark.version=2.1.1 clean package

I ran Spark with: ./bin/spark-shell --jars
../carbondata-parent-1.1.0/assembly/target/scala-2.11/carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar

I imported the two classes (not sure if the second is a typo with the
underline):

scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession

scala> import org.apache.spark.sql.CarbonSession._
import org.apache.spark.sql.CarbonSession._

Then tried to create the carbon variable:

scala> val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
<console>:28: error: value getOrCreateCarbonContext is not a member of
org.apache.spark.sql.SparkSession.Builder
       val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")

Thoughts on what I'm doing wrong here?

Bill-



--
Regards | Pallavi Singh 
Software Consultant
Knoldus Software LLP
+91-9911235949
Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.1.1 with CarbonData 1.1.0

Ravindra Pesala
Hi Bill,

Actually CarbonData is not supported with spark 2.1.1, it is only supported with spark 2.1.0.
And about your error it says that method is not present so better give absolute path of CarbonData jar to spark-shell and also check the imports are done properly or not. Please refer to the example CarbonSessionExample inside examples folder for creating carbon session

Regards,
Ravindra
On Thu, 25 May 2017 at 10:30 AM, Pallavi Singh <[hidden email]> wrote:
Hi Bill,

CarbonContext is only for Spark 1.6, for Spark 2.1 you should use CarbonSession. Please refer to CarbonSessionExample  for more details. 

On Thu, May 25, 2017 at 12:01 AM, Bill Speirs <[hidden email]> wrote:
I'm trying to follow the directions for using Spark 2.1.1 with
CarbonData 1.1.0 found here:
http://carbondata.apache.org/quick-start-guide.html

I compiled CarbonData using: mvn -DskipTests -Pspark-2.1
-Dspark.version=2.1.1 clean package

I ran Spark with: ./bin/spark-shell --jars
../carbondata-parent-1.1.0/assembly/target/scala-2.11/carbondata_2.11-1.1.0-shade-hadoop2.2.0.jar

I imported the two classes (not sure if the second is a typo with the
underline):

scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession

scala> import org.apache.spark.sql.CarbonSession._
import org.apache.spark.sql.CarbonSession._

Then tried to create the carbon variable:

scala> val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")
<console>:28: error: value getOrCreateCarbonContext is not a member of
org.apache.spark.sql.SparkSession.Builder
       val carbon =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonContext("hdfs://localhost:9000/carbondata")

Thoughts on what I'm doing wrong here?

Bill-



--
Regards | Pallavi Singh 
Software Consultant
Knoldus Software LLP
+91-9911235949