Is there a way to do merge-on-read in delta oss with spark batch job.

We see a phenomena where queries that search only for MAX(TIMESTAMP) in delta tables are yielding a scan and not a metadata lookup. Has anyone encountered this?

Hi all, I configured a table to use named column mapping mode and bumped the min reader and writer version as follows:

ALTER TABLE <table> SET TBLPROPERTIES (

'delta.minReaderVersion' = '2',

'delta.minWriterVersion' = '5',

'delta.columnMapping.mode' = 'name'

)

This resulted in new files written to the storage account to be placed into directories named with random two letter names e.g. 'pi', 'Wo', '32', etc. instead of the configured partition directories as before the change. My question is, why does this happen and is this something I need to care about?

Hello everyone, I'm working on enabling

S3 Multi-Cluster Writes

in my Databricks Workspace. I have confirmed that I have

read/write

permissions to the

DynamoDB

table, and have successfully tested the permissions using

boto3

in a Databricks notebook. I have also set the Spark configs as per step 4 of the documentation. The runtime version on my cluster is

11.3 LTS

, and I have installed the following dependencies:

io.delta:delta-storage-s3-dynamodb:2.1.0

and

com.amazonaws:aws-java-sdk:1.12.427

. I ran a script that reads a delta table and saves it to a different location, but there are no records in the

DynamoDB

table. I'm wondering if I'm missing something or if there's a step I haven't completed. I would appreciate any insights or advice on how to successfully enable

S3 Multi-Cluster Writes

. Here is the link to the documentation that I'm following: https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/

Hi All, I have existing delta table name as XYZ (having 1M records) and having two columns namely ‘Name’ ( in string type) and ‘Comments’ (in string dtype). But now I want to change dtype as Array of MapType for ‘Comments’ column for the same table. Is there any way to handle this situation? Can anyone plz handle this using pyspark? Thanks!

Hi All, I’m trying to setup data table clones to run data backups for security / disaster recovery purposes. Following this guide here requires me to run

deep clone

. Looking at the open source docs, I don’t see any mentions of

cloning

but looking at github, I do see tests running

clone

in CloneTableSqlSuite.scala. It appears to have been added pretty recently (3 months ago). Is there an estimated timeline of when

clone

functionality will be released?

Hi all, Need guidance in Delta lake. I am constantly getting data from DMS to S3 in parquet. Some times, i see 5 files coming with in a minute due to changes in Source RDBMS table. How can i capture changes in Target Delta lake efficiently. Can i make it as streaming architecture. My application is completely in AWS.

Hi all, Could you please clarify below scenario. I have been going through different videos and documents. Answers for below questions will help a lot for me. If i create partitioned Delta table, and run Merge data into Delta table and where clause is having Partition & Primary column, will look up go through all the files in the Partition? Will data skipping stats help here? Will running "Optimize deltatable ZORDER by (X,Y)" speed up this lookup? Do i need to run this command after every Merge?

Does any one create Glue Custom classifier for log relation and also automatics values to run Vaccum?

Hi All, maybe a newbie question... I've setup a minio storage on localhost:9000; no problem accessing it with minio client and s3fs (everything connects via http rest api, works like a charm). Then I have setup a delta-sharing server (deltaio/delta-sharing-server docker image) with a yaml config file as in delta-io/delta-sharing/server/src/universal/conf/delta-sharing-server.yaml.template on github. But how to set the "tables: location:" attribute? There are only examples on how to connect to open online data sources which require no credentials (access_key, secret_key ), and how to configure credentials when connecting to aws and gcp through environment variables and azure through xml-config-file. How do I configure my delta-sharing-server to connect to my minio server at localhost:9000 with access_key=abc123 and secret_key=abc123 - an example would be much appreciated? Note: I can access the delta-sharing-server using the python lib delta-sharing.SharingClient("mydeltacfgfile") and see the share, schema and table I have configured, but to access the actual file I need a way to connect to the path on the minio server.

Hi all, My team and I are using PySpark to manage delta tables in a delta lake folder on AW S3 We have to manage big tables ( table folder size of .parquet files is in TB ), and we have to manage merge query on them. We are getting bad performance by doing this, and so our plan is doing partitioning on those big table to improve the merge performences on our big tables I want to ask you if this is the way to go to improve our merge in our tables ? IS there another ways of improving the merge with Spark ?

Hi All, I created this issue last month => https://github.com/delta-io/delta/issues/1620 Can somebody check if it makes sense? I am willing to contribute if it is accepted

cc: @Matthew Powers @Denny Lee

ConcurrentModificationException - will this happen when we have more than one process writing the same table? Do you think we could have the same issue if two processes writing to the same table but different partitions?

Hi All, I have a scenario where I need to add key-value pair (maptype) to an existing maptype data using pyspark. Example: I/P: Day_field | id_field | maptype_field Day1 | 1 | {k1->v1} Day2 | 1 | {k2->v2} Day3 | 1 | {k1->v1_updt} Day4 | 1 | {k3:->v3} O/P should be : Day4 | 1 | {k3->v3, k2->v2,k1->v1_updt} Can anyone plz help me on this?

Has someone worked parsing complex XML column data into multiple columns in Databricks' pyspark ?

Hey folks. Can Delta Sharing handle cross cloud/region?

Alternative data sharing typically requires us replicating a database into a managed target cloud/region.

Yes it can

Hey folks, is anyone running kafka-delta-ingest in production to sync data to delta table with schema evolution. Wanted to understand the challenges faced in terms of infra resources required for setting it up.

I have a question about the metadata field in the data lake table and how to get it with python. Intuitively, this call seems like it should work:

# Load the Delta Lake table
delta_table = DeltaTable.forPath(spark, "/path/to/delta_table")

# Get the metadata of the table
metadata = delta_table.metadata

# Print the metadata
print(metadata)

However, I find that this doesn't work, even through the metadata JSON is right there, so I'm not writing code like this to find a simple table schema on a metadata table.

delta_dir = join(table_path, "_delta_log")
for file in os.listdir(delta_dir):
    if file.endswith(".json"):

        with open(join(delta_dir, file)) as json_file:

            for json_line in json_file:
                json_obj = json.loads(json_line)
                if "metaData" in json_obj:
                    self.catalog_metadata[table_path] = json_obj["metaData"]
                    schema_string = metadata["schemaString"]
            result = json.JSONDecoder().decode(schema_string)

It seems like such a simple basic question to ask of the delta table, what is your schema. Why isn't it baked into the API? Or if it, how do I get at it without resorting to writing code? Thanks, Tim

Hey! what/where can i find the differences between the open-source vacuum on delta tables and the databricks vacuum?

Hi all, there are some best practice about how long to store data in delta lake ? Given a table you have updates, insertions, deletes every day in it. There are some guidance about to handle it ?

Hello, Team I think there is a bug in delta-rs while getting schema scenario:

schema": {
    "date": {
      "data_type": "string",
      "partition_column": true,
      "nullable": false
    },
    "shift": {
      "data_type": "string",
      "partition_column": false,
      "nullable": false
   }
}

*the above is the schema of the table that we have created but when using deltaTable.sc*hema() it gives the below schema

schema": {
    "date": {
      "data_type": "string",
      "partition_column": true,
      "nullable": true
    },
    "shift": {
      "data_type": "string",
      "partition_column": false,
      "nullable": true
   }
}

nullable is returned true when we have set it to false we are using spark to create the delta table

Hi team, anyone have any idea as to why does spark always set the nullable field as true even when i have mentioned it to be false in the schema? I am trying to create a delta table with empty dataframe and a schema

Hi team, Anyone found some workarounds to mitigate the following error? I know the feature request is already on github, but wonder if someone has a workaround

Failed to merge decimal types with incompatible scale 4 to 8

Hello! I have a question, does delta sharing support sharing tables that are stored on Hadoop or in Minio? Thanks in advance

Hi all, ideally it is a best practice to partition a table, or there is some case which is not recommended?

Hi all, I wrote some 60 million records to a S3 and created Delta table on top of it. Initially, i could see all files are created of 128mb size When i run Merge on day 2 which has updates and inserts, all files merged to single file of size 786mb. If i run OPTIMIZE tableName ZORDER BY columnName, i could see files are created back with 128MB, These are some of Spark configs i use ("spark.databricks.delta.retentionDurationCheck.enabled", "false"), ("spark.databricks.delta.autoOptimize.optimizeWrite", "true"), ("spark.databricks.delta.autoOptimize.autoCompact", "auto"), ("spark.databricks.delta.autoCompact.maxFileSize", 134217728), ("spark.databricks.delta.optimize.maxFileSize", 134217728), ("spark.databricks.delta.tuneFileSizesForRewrites", "false"), I am unable to find what's kicking off this file merge to single file. Can you please give me an idea.

Is there any document having different spark config's that can be used for Delta files/tables