https://delta.io logo
r

Rajath Chandregowda

08/07/2023, 10:17 AM
Hello, I'm trying to convert an existing iceberg table to delta table, by referring to below document https://docs.delta.io/latest/delta-utility.html#convert-an-iceberg-table-to-a-delta-table Below is the configs which I'm using delta lib for both iceberg and delta. along with this I'm using iceberg standalone library.
Copy code
--packages org.apache.hadoop:hadoop-aws:3.3.1,io.delta:delta-core_2.12:2.3.0,io.delta:delta-iceberg_2.12:2.3.0,org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:0.13.1 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.hive=org.apache.iceberg.spark.SparkSessionCatalog
and I'm getting below exception,
Copy code
scala> spark.sql("CONVERT TO DELTA iceberg.`<s3a://dataset/Iceberg/table1/>`")
org.apache.spark.sql.catalyst.parser.ParseException:
Syntax error at or near 'CONVERT'(line 1, pos 0)

== SQL ==
CONVERT TO DELTA iceberg.`<s3a://dataset/Iceberg/table1/>`
^^^

  at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:52)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:89)
  at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:103)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
Is there anything I'm missing here ? PS - even tried without using standalone iceberg lib and conf too, Got a different exception,
Copy code
java.lang.NoClassDefFoundError: org/apache/iceberg/Table
  at java.lang.Class.getDeclaredConstructors0(Native Method)
  at java.lang.Class.privateGetDeclaredConstructors(Class.java:2699)
  at java.lang.Class.getConstructor0(Class.java:3103)
  at java.lang.Class.getConstructor(Class.java:1853)
t

Tom van Bussel

08/07/2023, 2:01 PM
Are you able to use Delta when not loading Iceberg?
r

Rajath Chandregowda

08/08/2023, 5:55 AM
when only delta is used , I get the below exceptions
Copy code
java.lang.NoClassDefFoundError: org/apache/iceberg/Table
  at java.lang.Class.getDeclaredConstructors0(Native Method)
  at java.lang.Class.privateGetDeclaredConstructors(Class.java:2699)
  at java.lang.Class.getConstructor0(Class.java:3103)
  at java.lang.Class.getConstructor(Class.java:1853)