Hello Everyone, I am working on a project where we are trying to save table using delta lake on a ADLS gen 2 using a SAS token. Is this even possible because I am not able to find any documentation for it. It only talks about oAuth for now for ADLS gen2.

Hello Everyone, Please help me with the below scenario? Scenario- RawDf written to delta table with schema - Id, Name, Age, Salary Incremental data comes in with an additional column - Id, Name, Age, Salary, Designation I'm using merge as below: targetTable.alias("tgt").merge(df.alias("src"), condition = "tgt.id = src.mergeKey and tgt.iscurrent = true").whenMatchedUodate(set = {"enddate ": currenttimestamp(), "isCurrent ": "false"}).whenNotMatchedInsertAll().execute() Spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", True) --> this is also enabled But new column I..e Designation not inserted. Am I doing anything wrong here...please help. Thanks!

Can Schema Enforcement feature be Disabled ? Written the parquet files to delta table using Delta Standalone.

I wrote a blog about Write-Audit-Publish including with Delta Lake, and would love your feedback. I've tried to be as accurate and impartial as possible - corrections, clarifications, and comments are very welcome 🙂 https://lakefs.io/blog/how-to-implement-write-audit-publish/#delta-lake

I really hope that someone can clarify the issue we're currently facing with the

delta merge

operation. What does it happen if during a merge operation the underlying data is changing? Can this lead to data inconsistencies? This would explain some issues that we're facing..

Is there a way to set

maxBytesPerTrigger

dynamically? I would like for my Spark streaming job to process large batches (i.e. 50GB) when there is a lot of data to backfill, however when it’s caught up it should process smaller batches.

how do I check in a databricks notebook which version of the delta lake I'm using? It seems a trivial request but I cannot find a way to do it in the docs..

Hi! If someone is using a shared table partitioned by a column is there a way to read certain partitions with pandas without bringing the whole table first? I see the load_as_pandas function doesn't have an option for it. Does it have to be done in the share mandatory?

Hello everyone! I am hoping to find a solution. I am Running into delta.exceptions.ConcurrentAppendException even after setting up S3 Multi-Cluster Writes environment via S3 Dynamo DB LogStore My use-case is to process a dataset worth 100s of partitions in concurrency. The data is partitioned, and they are disjointed. I was facing ConcurrentAppendException due to S3 not supporting the “put-if-absent” consistency guarantee. From Delta Lake 1.2, with the help of S3DynamoDBLogStore API, all writers across multiple clusters and/or Spark drivers can concurrently write to Delta Lake S3 while ensuring that only one writer succeeds with each transaction. My Delta Lake version is 2.1. I created a Dynamo DB table with auto-scaling enabled for number of reads/writes and passed the configuration to the delta job. Please find the configuration below (omitted some spark related config). spark = SparkSession \ .builder \ .appName("Delta Operations") \ .config("spark.driver.memory", args["spark_driver_memory"]) \ .config("spark.executor.memory", args["spark_executor_memory"]) \ .config("spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName", args["log_table_name"]) \ .config("spark.io.delta.storage.S3DynamoDBLogStore.ddb.region", args["log_region"]) \ .getOrCreate() spark.sparkContext.setLogLevel('WARN') Please find the actual logic below: delta_table.alias("old").merge( input_df.alias("new"), f"old.{primary_key} = new.{primary_key}") \ .whenMatchedDelete(condition=col(f"old.{primary_key}").isin(deletes_df)) \ .whenMatchedUpdateAll() \ .whenNotMatchedInsertAll() \ .execute() delta_table is the destination table in delta lake. input_df is a combined data frame of all the inserts, and updates. deletes_df is the dataframe that has just the deletes. I am still running into delta.exceptions.ConcurrentAppendException irrespective of these settings. Am I doing something wrong?

@Scott Sandre (Delta Lake) @Krzysztof Chmielewski We are running into some performance and load issues with streaming Delta table data from Flink. Our data volume is about 4G per second but we are not able to consume anywhere close to this volume. Do you guys have any performance and load testing results that you can share?

Hello.. i am reading aws kinesis data into delta table. Here one column named "data" is nested json which would have the actual information of my interest. i was explicitly applying schema to this column to get desired output columns. But my question is - can we get/generate the schema dynamically so that any new columns added to "data" column are also added to delta table ?

Hello all, I wanted to know if there is a plan to add mod function to use as a partition generated column. I want to mimic the concept of partition by bucket on a column that iceberg has, it can be easily done by defining a generated column = id%bucket_number and use it as partition. Is it planned in the open source?

Which type of data model is preferred with delta: Kimball star, Inmon snowflake, data vault, other?

Hi guys. Wanted to ask if someone has a good way to know if a delta table is z-ordered by some fields ? I saw i can use the history and filter on

OPTIMIZE

operation with

operationParameters.zOrderBy

. The above method worked for me locally after i z-ordered a table, but i was trying to use it on a table i know for sure was z-ordered in the past (half a year ago) and it returned empty results. Is it possible for a table to just lose it’s z-ordered when the only operations made on it are add and update ones ? Thanks

I am exploring delta lake connector. I created a delta lake connector in trino. And created a table but when i am firing query on that new table. Query is going into blocked stage , any idea why it might be happening ?

has anyone used delta lake connector ? I am not able to run queries on delta based table ?

Can we use the delta table parquet files directly for loading the data as a parquet file in other system ? Is that compatible ?

hey guys, we are facing challenges with delta tables and upserts for delta tables with many columns (approx. 4000) in spark@k8s. essentially, after the executors spin up we have period of 15-20mins where no job/task gets scheduled/executed. debug log statements say following DEBUG BaseSessionStateBuilder$$anon$1: Resolving 'col1 to col1#14457. so it seems like delta maps internally the source and target table columns. I described the problem and settings in more details also here https://github.com/delta-io/delta/issues/1806. We use Spark 3.3.0 and delta 2.3.0 (Seems like it is related to the fix from delta 1.2.0 https://github.com/delta-io/delta/commit/9a86ca79). any ideas or suggestions what to do?

Hello Everyone, Please help me with the below scenario? Scenario- RawDf written to delta table with schema - Id, Name, Age, Salary Incremental data comes in with an additional column - Id, Name, Age, Salary, Designation I'm using merge as below: targetTable.alias("tgt").merge(df.alias("src"), condition = "tgt.id = src.mergeKey and tgt.iscurrent = true").whenMatchedUodate(set = {"enddate ": currenttimestamp(), "isCurrent ": "false"}).whenNotMatchedInsertAll().execute() Spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", True) --> this is also enabled But new column I..e Designation not inserted. Am I doing anything wrong here...please help. Thanks!

Need help with delta lake connector. All queries are getting stuck in Planning stage for unlimited time. But same table i am able to query from aws athena query engine 3. And i am able to check table schema , run explain queries via trino cli but any select query is getting stuck in planning stage.

Hi team, What is the best way to manage delta table schemas specially if these schemas might change down the road?

Hi Folks, I am new to delta lake. In my organization, we are planning to build lake house. Could you please someone recommend how to proceed?

Hey folks, I am pulling data using the delta-sharing protocol and python SDK. I have a question regarding how the Change Data Feed is handled. Consider the following scenario: 1. I load the whole table to the current version, say version 5, with the

delta_sharing.load_as_spark(table_url)

function. 2. An update to record x occurs in version 6 3. Another update to record x occurs in version 7 4. I load the data from version 5 to 7 to get the most recent updates with

delta_sharing.load_table_changes_as_spark(table_url, starting_version=5, ending_version=7)

Will step 4 give me one combined update for record x (with the

update_preimage

from version 5), or will it have both updates (two

update_preimage

, one from version 5 and one from version 6)? And then say in step 3 record x was deleted instead of updated, would only the delete come? Appreciate any clarification on this, thanks!

Hey folks, are there guidelines on when we should be using managed vs external delta tables? my understanding is the the difference is when i drop a managed table i also purge the data (after 30 days by default). this is not true for external tables I also understand that

CREATE TABLE USING DELTA LOCATION …

creates external tables. Is there a way to create a managed delta table with an existing dataset?

Hi everyone, I am facing an issue with two jobs that are writing to a delta location. Here are the details: • Job 1: This job runs hourly and uses the overwrite mode to write data. After writing, it executes the vacuum operation. The purpose of this job is to update recently dated data. • Job 2: This is a backfill job that processes some past dates. It also uses the overwrite mode, but it does not execute the vacuum operation since the hourly job already takes care of it. The problem occurs when both jobs run simultaneously. It leads to data conflicts, and as a result, the data overwritten by Job 2 becomes inaccessible. Error message:

"File does not exist: /delta_path/part-00068-23256f70-a297-416f-8c5d-3650343929b3.c000.zstd.parquet"

Please let me know if you have any suggestions or solutions to resolve this issue (execute 2 job at the same time without conflict).

Hello Team, i am using Delta Sharing server . i have lot’s of table in which have around minimum 10 gb of data can anyone tell me where to deploy delta sharing server that can balance the load of data

Hi everyone, we are getting the "error" as we ran streaming API: • "org.apache.spark.sql.execution.streaming.state.StateSchemaNotCompatible: Provided schema doesn't match to the schema for existing state! Please note that Spark allow difference of field name: check count of fields and data type of each field." What are the consequences of setting the

set spark.sql.streaming.stateStore.stateSchemaCheck to false

? Main goal would be to merge schemas while streaming (adding new columns, ...). When running this code in Notebook (12.2.x-scala2.12) it works as opposed to Job (10.4.x-scala2.12). Was it some change done? Thanks and regards, GapyNi

#deltalake-questions #kafka-delta-ingest Hi everyone, My Kafka topics source for multiple delta table, which is filtered with data frame before applying to delta. Now we are using loop with in a micro batch to process all tables, i tried to google but of no luck Any insights on how to process a stream of data into multiple tables with out hardcoding

Hi Team, I store my delta tables schema definition in a central repo in json format, to enforce and validate schema of the incoming data, but when I tried to update a column data type from Timestamp to string using this commands: struct = StructType().fromJson(new_schema_string) df2 = spark.createDataFrame(data=[],schema=struct) df2.write.mode("append").format("delta").option("overwriteSchema", "true").saveAsTable("table_name") it throws this error: Failed to merge fields 'createdt' and 'createdt'. Failed to merge incompatible data types TimestampType and StringType what should I do?

Someone once told me about a graph schema built on top of delta lake but now I can't find anything about it. Anyone know of a graph schema/db built on top of delta lake?