Does anyone have experience with ML modeling on top of delta table with deep learning framework such as pyTorch?

Is SQL endpoint faster than Interactive Spark Cluster?

Hello guys ! I have a delta lake detabase folder on a AWS S3 bucket, on it i have delta table on which i am continuously doing updates, insert and deletes Some weeks ago its size was 6 TB, now it is 17 TB ! So my questions is : how can we reduce the size of tables were updates, inserts and deletions are made continously ? I saw there is with delta table what we call 'Compact files' , I guess I should go toward this solutions but i wanted to ask you to be sure if this is THE way to go when facing this problem or if there is another way to do it Thanks in advance :)

Hi Delta community, may i know what will happen if a schema has evolved while a spark job is reading the table

Hi, what is the recommended way to 'read' a subset of data from a Delta Table and either display the results in a notebook or use them in a DataFrame for further computation? Ideally, the chosen approach would utilize metadata statistics & partitioning (where available) to achieve optimal performance. a) spark.read.table("MyTable").filter("created_date > '2020-07-01'").display() b) spark.sql("select * from MyTable where created_date > '2020-07-01'").display() c) DeltaTable.forName(spark, "MyTable").toDF().filter("created_date > '2020-07-01'").display() Or does it not matter, and behind the scenes, they all result in the same processing logic being executed by Spark? Thank you

Hi delta community, is there anyway to preserve records from vacuuming. E.g. a user record that has been inserted in the beginning but not updated for a duration of time.

Hi, I'm using delta lake with AWS Glue. when i try to run the command --> deltaTable.optimize().executeCompaction() i get the following error --> AttributeError: 'DeltaTable' object has no attribute 'optimize' I'm using the jar --> delta-core_2.12-1.0.0.jar Pls help.

Hi, I'm trying to implement multi-cluster setup so that I can have multiple EMR clusters writing to the same Delta table (in S3) at the same time. I followed all the instructions on that page and ran a couple parallel test jobs in EMR. I confirmed that locks were being added to the table (for entries in the transaction log). However, one of the jobs failed raising a

ConcurrentAppendException

error. Is this expected? Do I need to implement retries in my write logic? I figured that after enabling multi-cluster setup that a cluster would wait for the lock to be released and then write a new transaction, I didn't think I would have to worry about retries. Do I have something misconfigured?

Hi all, I have a delta table with a timestamp column

enqueuedTime

that I want to rename to

enqueued_time

. The column

enqueuedTime

is used to define three generated columns:

year, month, day

The table itself is then partitioned on the

year, month, day

generated columns. When I do:

ALTER TABLE <table> RENAME COLUMN enqueuedTime TO enqueued_time;

I get the following error: "Cannot rename column enqueuedTime because this column is referenced by the following generated columns:" • year • month • day How can I perform this renaming?

The delta protocol contains the following sentence, but check constraints should allow

unknown

(e.g.

1 > null

) in addition to

true

from SQL standard perspective in my understanding. Can we update the protocol?

Rows in a table must satisfy CHECK constraints. In other words, evaluating the SQL expressions of CHECK constraints must return

true

for each row in a table.

cc: @Slawomir Pajak

Hello Everyone! I have a question.We are using Glue and Athena along with Delta Lake .We are able to query the Delta Lake tables in Athena ,but not able to do Vacuum on the delta tables in Athena.So checking if there are way to do Vacuum on the Delta tables using Athena. Thanks in advance for the response.

Hello delta community, is it possible to incremental consume from multiple tables into single delta tables. We are building a wide table of 1000 columns where it will be aggregated from 10 delta tables. The rough idea at the moment is to run sequential Spark streaming jobs (trigger=availableNow) with a while loop to sync this in real time.

Hello Delta community, what is best and recommended way to ingest data from multiple sources like many s3 into Delta using spark structured streaming. Spark.readStream......load()

Hello Delta Community,

DEEP CLONE & CLONE

syntax/feature isn’t yet supported as of oss

delta-lake:2.3.0

right?? it’s only available in

DBR

i suppose. Can someone please confirm if this is indeed the case?

spark.read.format("xml").option("rowTag","Documents").load("/data/pat/66086B.XML").write.format("delta").mode("append").save("<s3a://delta-lake/demo1>")

org.apache.spark.sql.AnalysisException: Failed to merge fields 'PatentDocument' and 'PatentDocument'. Failed to merge fields 'BibliographicData' and 'BibliographicData'. Failed to merge fields 'Parties' and 'Parties'. Failed to merge fields 'Agents' and 'Agents'. Failed to merge fields 'Agent' and 'Agent'. Failed to merge incompatible data types StructType(StructField(Name,StructType(StructField(_VALUE,StringType,true),StructField(_lang,StringType,true)),true),StructField(_format,StringType,true),StructField(_sequence,LongType,true)) and ArrayType(StructType(StructField(Name,StructType(StructField(_VALUE,StringType,true),StructField(_lang,StringType,true)),true),StructField(_format,StringType,true),StructField(_sequence,LongType,true)),true)

Why do the above errors occur？

Hi Delta community , how to write concurrently using delta-rs library in python , do we have some existing support for using some logstore like dynamodb ?

Hey

AttributeError: 'DeltaTable' object has no attribute 'optimize'

Any one face follow error in Glue

hii Team , i have set raw log and file retention duration default but why i am getting below error and what would be the best solution. if i change checkpointing loc then job will start duming data again in refine , please give me optimal solution so i can resume my application where it goes failed

hi friends, working with functions on apache 3.3 which you do not need pre define a col to work with functions. However if you chain with .

transform

, it returns that "it not possible to groupBy with .transform method" Here is my code:

def qtd_faturada(col1,col2):

return (sum(when(col1=='S',col2))

.otherwise(-col2))

def with_qtd_faturada(df,col1,col2):

return df.withColumn("qtd_faturada",with_qtd_faturada(col(col1),col(col2)))

df=(df

.select(A,B,C)

.groupBy(C)

.transform(with_qtd_faturada,col1="A",col2="B"))

Am I missing something ? Some advice would be really appreciated, 🙂

I'm doing some reading about the pattern, and wondered if Delta Lake has any functionality that supports it? Apache Iceberg has built-in support, and Apache Hudi has Pre-Commit Validators which aren't quite the same thing but in the same kind of ballpark at least. Thanks.

Hi All, Is there any way to rename the delta table name ? xyz_delta_table ___deltalog_ some partitions … i wish to rename the delta table name

xyz_delta_table

to something else

Hii Team ,is autoOptimize works in OSS Deltalake ,and if it is work then deltalake could compact also the file ?

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Delta Lake Auto-Optimization") \
    .config("spark.databricks.delta.optimizeWrite.enabled", "true") \
    .config("spark.databricks.delta.autoCompact.enabled", "true") \
    .config("spark.databricks.delta.autoCompact.maxFileSize", "512") \
    .getOrCreate()

# Your code to read and write Delta Lake files here

spark.stop()

Hi Team, is spark....format("cloudFiles") available in Open Source Spark or it is specific to DB Data Loader.

Hi Team. Primary key and reference key constraints for deltalake are in public preview on databricks. Are there any plans to bring this feature to OSS Deltalake ?

Hi Team, Can we sync users and groups between AWS IAM and Databricks unity catalog as an effort to unify control access in one place

Hi friends, As a best practice when some table is updated period to period, is it a better option to upsert it, or overwrites is also an option depending the situation ? Currently we are working like this :

result.write \

.format("delta") \

.mode("overwrite") \

.option("overwriteSchema", "true") \

.saveAsTable('schema.table')

Does the OSS 2.3 version of delta lake actually support whenNotMatchedBySource for Spark 3.3.2? https://github.com/delta-io/delta/issues/1724 - I am running into class not found issues

Is there any solution for metadata management in Delta+S3?