https://delta.io logo
n

Nitya Thacker

04/14/2023, 3:02 PM
Hi, might a silly question, but have been stuck for some time on this. My delta table detail shows the name as null. It was created by saving a dataframe as delta. Is there a way to update the name?
s

Scott Sandre (Delta Lake)

04/14/2023, 4:27 PM
Which version of Delta Lake are you on? This was a bug that was fixed: https://github.com/delta-io/delta/issues/1549
n

Nitya Thacker

04/14/2023, 5:28 PM
i am using delta 2.1.0 (OSS), spark 3.3.0
s

Scott Sandre (Delta Lake)

04/14/2023, 5:30 PM
Yup, this was fixed in 2.3.0!
n

Nitya Thacker

04/14/2023, 7:13 PM
@Scott Sandre (Delta Lake) is there a way to update it now that it is null? I want to run an alter table - alter column command to update a constraint on the table for which it needs the table name
s

Scott Sandre (Delta Lake)

04/14/2023, 8:58 PM
Seems like the commit fixes finding the tableName .... i.e. the information is there, just not returned in delta lake 2.1.0 in the describe detail command
can you try using delta lake 2.3.0?
n

Nitya Thacker

04/14/2023, 8:59 PM
i will give it a try. Thanks!
with 2.3.0 also it shows null for name and desc
is there any other way to find the name, other than detail?
r

Ryan Zhu

04/15/2023, 12:27 AM
What’s the code to show the details? If you use the table name to query, you will see it. But if you use the table path, name will be null
n

Nitya Thacker

04/15/2023, 12:29 AM
I am using deltatable.detail.toDF.show to view the details. I did use path to create the table. if name is null, how do i run queries like 'alter table...alter column' which expects a tablename?
r

Ryan Zhu

04/15/2023, 12:39 AM
`alter table delta.`your path``
You can use this pattern in any place that expects a table name in SQL
n

Nitya Thacker

04/15/2023, 12:48 AM
i tried
Copy code
spark.sql("ALTER TABLE delta.'/home/some/path' ALTER COLUMN col1 SET NOT NULL") 

and get error - 
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:52)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:89)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
  ... 49 elided
r

Ryan Zhu

04/15/2023, 12:49 AM
You need to use backticks.
By the way, SET NOT NULL is not supported right now. See https://github.com/delta-io/delta/issues/831
n

Nitya Thacker

04/15/2023, 12:57 AM
thanks Ryan, backtick worked..but as you mentioned, not null did not work
20 Views