Rajath Chandregowda
08/07/2023, 10:17 AM--packages org.apache.hadoop:hadoop-aws:3.3.1,io.delta:delta-core_2.12:2.3.0,io.delta:delta-iceberg_2.12:2.3.0,org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:0.13.1 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.hive=org.apache.iceberg.spark.SparkSessionCatalog
and I'm getting below exception,
scala> spark.sql("CONVERT TO DELTA iceberg.`<s3a://dataset/Iceberg/table1/>`")
org.apache.spark.sql.catalyst.parser.ParseException:
Syntax error at or near 'CONVERT'(line 1, pos 0)
== SQL ==
CONVERT TO DELTA iceberg.`<s3a://dataset/Iceberg/table1/>`
^^^
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:52)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:89)
at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:103)
at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
Is there anything I'm missing here ?
PS - even tried without using standalone iceberg lib and conf too,
Got a different exception,
java.lang.NoClassDefFoundError: org/apache/iceberg/Table
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2699)
at java.lang.Class.getConstructor0(Class.java:3103)
at java.lang.Class.getConstructor(Class.java:1853)
Tom van Bussel
08/07/2023, 2:01 PMRajath Chandregowda
08/08/2023, 5:55 AMjava.lang.NoClassDefFoundError: org/apache/iceberg/Table
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2699)
at java.lang.Class.getConstructor0(Class.java:3103)
at java.lang.Class.getConstructor(Class.java:1853)