https://delta.io logo
Powered by
c

Chetan Joshi

03/30/2023, 1:10 AM
Hi everyone! We are using Deltalake 2.0.1 and trying to get value of 
logRetentionDuration
for our delta table. Could someone advise on fetching this property using Python or Java? Thanks! We tried the following but “detail” method is not available in 2.0.1 it seem:
Copy code
DeltaTable deltaTable = DeltaTable.forPath(spark, pathToTable);
DataFrame detailDF = deltaTable.detail();
j

Jim Hibbard

03/30/2023, 6:40 AM
Hey, I think this should work for you:
Copy code
spark.sql("DESCRIBE DETAIL delta.`path\to\delta\table\path`").show(truncate=False)
The property should be under the 
properties
column. Hope that helps!
c

Chetan Joshi

04/03/2023, 9:21 PM
Hi @Jim Hibbard, thanks for your response. I tried the above in Java, but properties were empty. Also, I couldn’t figure out how to set 
logRetentionDuration
value too. Any advice for this?
j

Jim Hibbard

04/04/2023, 2:26 AM
Hmm, I'll take a look at this tomorrow 🤔
c

Chetan Joshi

04/06/2023, 9:45 PM
Thanks @Jim Hibbard for taking out time for this. Please let me know if you have some ideas for me try.
s

Sakthivel B

04/12/2023, 11:22 PM
Jim, any additional suggestions?
Would like update the following two properties on an existing delta table:
Copy code
delta.logRetentionDuration = "interval <interval>"
delta.deletedFileRetentionDuration = "interval <interval>"
What is sql command or API to use, can someone point to the documentation. though I am on Delta 2.0.1, I tried testing with 2.0.1 - I am able to see the detail()
Copy code
-RECORD 0-----------------------------------------------------------
 format           | delta                                           
 id               | 20ac2aac-b288-43b5-b98d-15442be9748f            
 name             | null                                            
 description      | null                                            
 createdAt        | 2023-03-19 08:38:51.615                         
 lastModified     | 2023-03-19 14:50:59.891                         
 numFiles         | 112                                             
 sizeInBytes      | 1144080                                         
 properties       | {}                                              
 minReaderVersion | 1                                               
 minWriterVersion | 2
What is the method on DeltaTable object to update 
properties
t

Tristen

04/13/2023, 11:48 AM
In SQL you can use ALTER TABLE to set them similar to this: https://docs.delta.io/latest/delta-batch.html#table-properties Right now I think this is the only way to do it afterwards but there has been discussion around adding it to the Python api, see e.g. https://github.com/delta-io/delta/issues/1656
s

Sakthivel B

04/13/2023, 1:51 PM
thanks for the confirmation, I initially had issue getting ALTER TABLE to work with “oci://…” looks like that was a HDFS connector issue, its working now.
5 Views
Powered by