Hi all, I have a question about partition values. One of my partition columns is a file path which has forward-slashes. It seems to work fine, but I want to make sure there aren’t any gotchas with this. As an example, the partition values for a particular Parquet file are

{"received_date": null, "context_page_path": "/bar/"}

and the resulting path in S3 after the write is

<s3://bucket/table/received_date=__HIVE_DEFAULT_PARTITION__/context_page_path=/bar/part-00000-da65176e-fb16-418b-be85-b3a28d3edc11-c000.snappy.parquet>

. It seems like if

context_page_path

was ever

/bar

it would collide with this partition, even though it’s technically a different value.

I have a question regarding the maximum file size created when using the Z-Order optimization command. I have a partitioned Delta Lake table where I structure streaming data and use the merge command in the foreach batch operation. I run the Z-Order optimization on every 100th batch, and I've noticed that each time it runs, it creates a very large file. The current large file is 6.3 GB, and it keeps increasing. I didn't set the 'spark.databricks.delta.optimize.maxFileSize' property, so it should be the default 1 GB, right? Do you have any insights into why it's creating such large files? I'm using Spark 3.4.1 and Delta Lake 2.4.0, and I'm not observing this behavior in non-partitioned tables where the maximum file size remains at 1 GB."

Does Deltalake supports COW and MOR tables ?

Hi all, I have seen the recently Databricks added the type

Run Job

which allows to trigger a workflow from within a workflow. There is a mandatory Parameters tab that has to have some values in it which is the parameters passed to job. Why is this a mandatory section? In the desired workflow i trigger i already have on each of the tasks the parameters required for that tasks’ run so i don’t need any parameters to be passed to the job itself? Is it possible that it’s a bug ? (forcing me to add parameters to the job)? If not, i can’t seem to use it because i have the same key parameter used for various tasks in my workflow with different values. Thanks.

Hello all, Does anyone starting from which release the DLT table would have these three expectations column? DROP_EXPECTATIONS_COL, MEETS_DROP_EXPECTATIONS, ALLOW_EXPECTATIONS_COL?

Hy all , i am new to delta lake and i have pipeline which saves data locally in delta tables. I want to know can i use delta share for local delta tables so that analysts can access the data or is there any other tool which can read locally save delta tables.

Hi all I am facing a situation where I am using

df.writeStream.foreachBatch(...)

to merge against a delta table. the merge is done on a uid column and an hour partition column. But when my data growths in size the query is failing due to

spark.driver.maxResultSize

. _I_f I understand the spark plan correctly (see down below) there is a broadcast join going. It clearly makes sense to have have merges without shuffles but is there a way bypassing the driver since this would not scale well? Thanks

== Physical Plan ==
AdaptiveSparkPlan (30)
+- == Current Plan ==
   HashAggregate (19)
   +- Exchange (18)
      +- HashAggregate (17)
         +- Filter (16)
            +- HashAggregate (15)
               +- Exchange (14)
                  +- HashAggregate (13)
                     +- Project (12)
                        +- BroadcastHashJoin Inner BuildLeft (11)
                           :- BroadcastQueryStage (6)
                           :  +- BroadcastExchange (5)
                           :     +- * Project (4)
                           :        +- * Filter (3)
                           :           +- * Filter (2)
                           :              +- * Scan ExistingRDD mergeMaterializedSource (1)
                           +- Filter (10)
                              +- Project (9)
                                 +- Project (8)
                                    +- Scan parquet  (7)

Hi all, I am relatively new to Delta and recently came across the VACUUM and Time Travel features.. The idea is to be able to remove files older than 15 days but also be able to time travel 15 days before.. I looked into the documentation and some examples and to achieve the same features i set the following properties using ALTER TABLE SET TBLPROPERTIES : • delta.deletedFileRetentionDuration = interval 15 days • delta.logRetentionDuration = interval 15 days Problem : when I check the DESCRIBE HISTORY i can easily find the Older Versions but when i wish to travel back to one of these Versions i have :

AnalysisException: The provided timestamp (2023-09-08 02:30:00.0) is before the earliest version available to this table (2023-09-08 23:12:30.0). Please use a timestamp after 2023-09-08 23:12:30

I am totally confused 😅 Am i missing something here ? Am i setting

deleteFileRetentionDuration

or

logRetentionDuration

wrong ? Also my vacuum command has a retain 360 HOURS Inputs are appreciated.. Thanks 🙂

Folks when is the delta lake 3.0 GA released planned?

Folks, good day A question from newbie ) If I already have a delta lake but without stats unfortunately how i can add them w/o converting to delta once again through the latest delta api. Would it be ok if I add them by

StatisticsCollection#recompute

command? If not - what would be your advice. ( I'm currently on

2.1.0

)

HI All, I'm wondering if there is any good reason to use primary key in delta tables if we don't care about the unique constraint ? Any performance related benefits ?

Hi, I’m trying to clone an existing delta table locally using delta-rs. I’m using `DeltaTable.get_schema().unwrap().clone()`; to get the schema and I create a new table with

let mut config = DeltaOps(table).create().with_columns(schema.get_fields().clone());

I use datafusion a dataframe by `let df = self.ctx.sql(sql).await.unwrap()`; . When I try to write the reecordbatch from this dataframe by `record_batch_writer.write(batch).await?`; I see that the recordbatchs schema has a change. I see that the timestamp is data_type:

Timestamp(Nanosecond, None)

but the table schema has data_type:

Timestamp(Microsecond, None)

. Is there a config to fix this behaviour. Any help is appreciated.

Hi All, I'm try to explore delta share docker image https://hub.docker.com/r/deltaio/delta-sharing-server on Docker Desktop and AKS (kubernetes), I tried 0.6.3 & 0.6.7 docker tags but I'm seeing file not found exception(no such file or directory) - I'm trying to mount delta-sharing-server-config.yaml and core-site.xml with the docker image from docker run - same issue. From AKS - created configmap and mounted to pod/deployment - same file not found exception. Is these docker image solution working? Anyone tried this? Did i missed anything? Thanks in advance....

Hello, I am looking to read data from a partitioned delta table using Apache Flink running in Batch Mode; I am following the documentation of the connector below to do so. When reading the source, I am looking to filter on the partitions that I read with Flink, but there does not seem to be a way to do this; can someone advise? https://github.com/delta-io/delta/tree/master/connectors/flink

Hello Team, we are facing an issue on delta tables where we are running vaccum and optimize scripts but for some reason the code is throwing a filenotfound exception recently

No such file or directory: <s3a://bucket/delta-table/part-00000-57b118c1-a1a6-4ea1-aaa6-e2069601e1b6.c000.snappy.parquet>

It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running ‘REFRESH TABLE tableName’ command in SQL or by recreating the Dataset/DataFrame involved. to note we store our delta tables on MinIO

Is there anyone here who have built some realtime database on top of delta tables? I don’t want to use apache druid due to it being a bit too heavy and it does not have native delta support.

I'm testing out using delta table with DynamoDB logStore. From what I can tell, if I delete the table in S3, the dynamoDB entry remains dangling, and will stop me from recreating the table later (because the dynamodb logstore errors when dynamodb knows about a table but S3 does not have a file for it, with "Old entries for table %s still exist in the external store" message). The only solution I see for this is to manually go into dynamodb and delete the entry if I wipe the table from S3.

Hi All, I am using the Delta Standalone library to update delta table for parquet files written by a SQS worker fleet. I am observing increasing latency in checkpointing as the table size has increased to contain 10's of millions of files. The latency continues to rise with increasing number of files. I have identified few options to experiment with : 1. Increasing checkpoint interval configuration 2. Retaining the logs for shorter period of time 3. Compacting the data before write I am yet to run the above experiments to see how much scaling horizon it gives us and understand what are the tradeoff's being made. I saw a related feature launch in the 3.0.0rc1 announcement

Support for disabling Delta checkpointing during commits - For very large tables with millions of files, performing Delta checkpoints can become an expensive overhead during writes. Users can now disable this checkpointing by setting the hadoop configuration property

io.delta.standalone.checkpointing.enabled

to

false

. This is only safe and suggested to do if another job will periodically perform the checkpointing.

I looked into the codebase but the standalone API's do not expose checkpointing capability. So here are my two questions 1. How can I configure another job (preferably using Standalone) to perform checkpointing ? 2. Are there any other recommendations/learnings for handling large delta tables ?

Hello team, I have a question for delta lake table. I use SPARK SQL run MERGE query, query have insert and update only, not contain delete statement. Operation metric is in image. After check log between before delta version and after delta version by COUNT() query. I see that some records had been removed [4314 records removed], it is match with “numSourceRows” metrics. Why it is happen ?

Hello team, I have a question for Delta Lake Multi-writer. I followed the Delta 2.2 Multi-writer setup steps to enable the Delta Lake Multi-writer for S3. However, after following these steps, I still see the

ConcurrentAppendException

when I write to the same table (use Delta

merge

) from 2 EMR clusters (EMR version 6.10.0, Spark 3.3.1, Delta 2.2.0), and there was no entries in the DynamoDB after writing. It looks like I did not enable Delta Lake Multi-writer successfully. My questions are: 1. How can I debug this? I checked online, but I did not find the useful solutions/debug steps 2. I use the

SparkSession.builder().appName().config(…)

to config the spark session. Is this correct? Besides the spark configuration listed in the Setup steps, is there any required Spark configurations I need to setup? Thank you!

Hi folks I am having concurrent write errors on a delta table. The table is partitioned by day and I am using replaceWhere to filter out partitions to write. • I have a job that writes past hour only • And another job that writes past data [D-30, D-1} This way the hourly job and the daily one will never write to same partition at same time... but for some reason I am still getting the

ConcurrentAppendException

error:

Files were added to partition [event_date=2023-09-18] by a concurrent update

This log is from the daily job that have a

replaceWhere = event_date between '2023-08-18' and '2023-09-17'

and I am sure that there is no 2023-09-18 data on that dataframe. So my idea now is use upsert on hourly run and continue with replaceWhere on daily one... but I don't know if daily will continue to fail since it will be an insert and an update operation at the same time.... Any thoughts? Thank you

Hi team, I have some questions about the Delta Lake multi-cluster: 1. What does it mean that Delta Lake supports multi-cluster writes? I found an answer in the Databricks community, can someone confirm if this is true? 2. Can Delta Lake support writing to the same partition at the same time if I enable the Delta Lake multi-cluster writes. 3. The Delta Lake multi-writer uses the DynamoDB as the logstore, am I supposed to see new entries after each table writes? Thank you!

How does deltalake ensure durability from ACID? In hdfs, by default, if you write something, it 1) writes it to 1 node, 2) reports success to the client, 3) replicates data in the background What if we got to write to that 1 node from step 2) and then this node died and didn’t get to replicate its piece of data. In this case, will deltalake report to the client that it successfully wrote its data to HDFS, when in reality it lost it? Or how does it work?

I just stumbled over the very same issue as described here https://kb.databricks.com/data/null-empty-strings now I am wondering if this is a Delta issue or a Spark issue and/or if there is any Spark setting to solve this?!

Hi Folks I have a spark streaming application running in k8s i have some questions regarding Delta Lake optimize 1)How is your delta optimize z-order optimize setup in production, Are you doing it as part of the streaming job itself? like every 100th batch you run optimize command or is it running as a separate job? 2)When I run z-order in my spark application on every 100 batches I notice its ends up scanning the entire data from s3 and sort, Which makes an OOM error and Ephmerial storage full error in my cluster, Is there a way I can z-order with less resource, Say my table is 100 GB , and can I run the z-order with 50 GB of the total cluster, I don't want to add the partition to my table since I observed it creating very large files compared to the non-paritioned table, 3)I am thinking of running z-order optimize as a separate job which runs once to twice every day but there will be a potential conflict with my streaming job, Is there a way to handle this conflict?

Hi all -- Can anyone share how they estimated needed resources to perform a merge with delta lake? I have a dataset with ~45B rows, and need to upsert tens or hundreds of millions but I'm consistently hitting various spark failures around OOM issues -- dead executors, fetchfailed, lost RDDs and tasks etc. I'm trying different configurations for executor and driver memory, partitions, shuffle partitions etc but the issues are persistent. How can I estimate how much resources would be required to perform the merge successfully without all the trial and error?

Hi all

the GUI in this demo

is part of Delta lake ?