Hey team, We are using delta standalone library to write data to ADLS, we are using

timestamp-micros

to sync timestamp columns, but looks like the library is unable to read the data. I am getting the below exception

scala.MatchError: LongValue(1677595007976000) (of class shadedelta.com.github.mjakubowski84.parquet4s.LongValue)

But i am able to read the data fine using Azure spark pool and also read the parquet files fine. Is there a way to read

timestamp-micros

using the library?

Hi 👋 We are using Delta Lake 1.2 (on EMR) and I'm trying to understand why our largest tables all end up with exactly 1,000 files in S3 and this does not change as we merge in more data. They are of varying sizes (e.g. 1.1 TB / 7B records, 330GB / 1.5B records, 53 GB / 180M records), but all have 1,000 files even though we have not explicitly repartitioned (nor set shuffle partitions). I know delta now does repartition before merge by default, could that have an impact? It looks as if there is a max setting at play somewhere?

Hi guys, it's great to be here with you all! Yesterday a raised a question to @Denny Lee on Linkedin and he suggested putting it here to provoke you all. Days ago, I had a discussion with my tech lead about the framework Lakehouse using Delta Lake. We were debating ( health discussion ) whether the lakehouse approach applies Clean Architecture from Uncle Bob. He has a background in Software Engineering, anyways. He said the architecture, bronze, silver, and gold are not compared for example from "Entities" and "Use Case", using TDD, and so on. As I know, the lakehouse is built to be an SSOT and among other AMAZING things. Do you have any thoughts to share or even similarities that I can show to him? We use dbx, databricks, scala and pyspark to develop our jobs. I appreciate it in advance, many thanksss.

Hi Team 👋🏽 While doing cost analysis I started thinking about below

Change Data Feed (CDF) includes the row data along with metadata indicating whether the specified row was inserted, deleted, or updated.

This effectively doubles storage for tables with CDF enabled. is that correct?

Is it adviceable to write databricks notebook's pyspark code in Object Oriented way (using python classes and methods) ? Embedding all pyspark code in python class & methods. Will it have performace issues same as python udfs ? Any input is appreciated. Thanks.

Hi everyone! I'm playing with Python Delta-RS library to build a mechanism to iterate over N partitions and perform some cleanup tasks (Ex. Deal with deduplication and drop columns fulfilled with null values) but I'm stuck with the way that write_deltalake() method handles the overwrite mode. Instead of overwriting only specific partition, is overwriting the entire table. Does anyone know any workaround or a "config" to deal with this problem? Thanks in advance!

Hey folks, we are exploring migrating from parquet/iceberg/hudi to delta. https://docs.databricks.com/ingestion/clone-parquet.html is this like “zero copy cloning”? does the CLONE command need to be only issued once?

I am wondering if there would be a large penalty to including a column with nothing but empty strings in a delta table or if compression would make it negligible? I'm considering the merit of including standard meta columns although they might sometimes be unused vs adding columns case by case.

hey, I’m trying to drop a column from a delta table. I followed the alter table mentioned here but I keep getting the message:

DROP COLUMN is only supported with v2 tables.

but I can’t found anything about that… can someone please explain to me what is going on and how to handle it? Thanks!

Hi guys, I am working on the medallion architecture of delta but there is limited resources to read. Databrick says the gold layer requires fewer join and de-normallized while silver layer has the 3rd normal form. So 1. The silver is the dim of star-schema, and we can join all the dim to the universal set (as the gold table) 2. The silver A . If Join: silver can only be generated from join bronzes on primary key B. For other operation: mainly support clean, filter, augment, create new column 3. After the gold table is generated, which kind of data should it hold: A .The fully joined table of star-schema B. summarized data after groupby and agg (groupby zone, agg count*) Can anyone have experience help me to clarify these concern?

Hi everyone, I have create EMR cluster following this guide: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Deltausing-cluster.html And now I'm trying to add a new step using airflow, however there is something wrong:

from delta.tables import *
ModuleNotFoundError: No module named 'delta'

Here is how I added a new step:

Hi, I have used delta lake (v1.0.1) and would like to migrate to v2.2.0. Do you have any advice or recommendations about the process of migration? How does it better to conduct?

Hi Everyone, Is there any dummy's guide that describes SCD-Type-2 implementation with Deltalake/Spark? Specifically how to "_*optimally join"*_ with an SCD-Type-2 dimension table while aggregating facts for reporting. I have working solution with a query. When I run my query in databricks, it gives me a little warning at the bottom: "_Use range join optimization: This query has a join condition that can benefit from range join optimization. To improve performance, consider adding a range join hint_." and points to this link. Question: Is there a more optimal way to query when I'm joining using a

between

condition (instead of

=

condition)? Or use something more optimal than

between

?

Hi everyone, I have a question regarding the use of spark to write data in deltalake. Are there any alternatives? (apart from delta-rs)

Hi I'm looking to integrate Delta sharing in our Data collab platform. But I would have some questions. Would someone could be available for a short chat?

Hi guys, I'm having some issues on persisting a table in delta, my cluster cannot process the data to do it, there is some best practices to do it ?

(df.write.format("delta")

.mode("overwrite")

.partitionBy("year_partition")

.option("overwriteSchema","true")

.saveAsTable(path))

I'm looking to convert a parquet into delta before persist, I do not know if it is a viable option

Hi - I'm trying to create a delta table with

write_deltalake

It raises `deltalake.PyDeltaTableError: Schema error: Invalid data type for Delta Lake: Null`error if there is a column with all none values. Is there a way to solve this? E.g. define the schema for table or define column type

import pandas as pd
from deltalake.writer import write_deltalake

data = [['tom', None], ['nick', None], ['juli', None]]
data = pd.DataFrame(data, columns=['Name', 'City'])
write_deltalake("s3a://.....", data, mode='overwrite')

Is it possible to create DeltaTable given a metastore table name. Specifically, I want to delete a partition, and the documentation say I should use

import io.delta.tables._

val deltaTable = DeltaTable.forPath(spark, "/tmp/delta/people-10m")

// Declare the predicate by using a SQL-formatted string.
deltaTable.delete("birthDate < '1955-01-01'")

But since Hive Metastore already has a location for a table, I don’t want to repeat myself, and would rather have

val deltaTable = DeltaTable.fromMetastore("mart.my_table")

is it possible?

I'm experimenting with Databricks and Delta. Noticed that spark.sql() operations can return a dataframe with affected rows. Doing a saveAsTable instead does not seem to return a similar result dataframe. Does anyone know of some tricks to get the same information from delta api or spark itself with low overhead?

Hi. Has anyone seen this error while creating a view over a complex struct type datasets:

Cannot have map type columns in DataFrame which calls set operations

To reproduce:

Select * from
(
SELECT struct('Spark', map(1,2)) as a
Union
SELECT struct('Spark', map(1,1)) as a
)

Please do share best practices on handling these cases

Any suggestions on this error

java.lang.ClassCastException: class org.apache.spark.sql.catalyst.plans.logical.DeleteFromTable cannot be cast to class org.apache.spark.sql.delta.commands.DeleteCommand (org.apache.spark.sql.catalyst.plans.logical.DeleteFromTable and org.apache.spark.sql.delta.commands.DeleteCommand are in unnamed module of loader 'app')

The code that cause this exception is something along the lines of

df.write.format('delta').mode('overwrite').option("mergeSchema", false).option("overwriteSchema", false)\
                    .partitionBy("date").option("replaceWhere", "date BETWEEN '2023-01-01' AND '2023-01-02' AND vertical like '%xyz%' ").save("s3://")

running on Spark 3.3.1 Hadoop 3.4.4 and ``delta-core_2.12-2.1.1.jar` and ``delta-storage-2.1.1.jar`

Hi everyone. Does zordering by specific columns a data into only one or two files, each around 1GB, speed up the query? Delta selects the files using the metadata from zorder, but I don't know if the performance improves if there's only one file.

has anyone faced this issue in OPTIMIZE :

Predicate references non-partition column 'customerid'. Only the partition columns may be referenced: []

OPTIMIZE works for partitioned columns only.??....i am using unpartioned table

Hi I am using delta lake version 1.0.0. I want to revert my table to previous version. Please help how can I do that. I am unable to run the following query

spark.sql("SELECT * from governor.governor_transactions VERSION AS OF 516")

Getting following error

mismatched input 'AS' expecting {<EOF>, ';'}(line 1, pos 53)

Using the following property

spark.sql.extensions='io.delta.sql.DeltaSparkSessionExtension'
spark.sql.catalog.spark_catalog='org.apache.spark.sql.delta.catalog.DeltaCatalog'

Please help

Hi team, we have our

datalake

of parquet files which we want to convert to

deltalake

format. Existing folder structure in our datalake is like this.

user_data
     user_hash=1
          date=2022-01-01
          date=2022-01-02
     user_hash=2
          date=2022-01-01
          date=2022-01-02
     user_hash=3
          date=2022-01-01
          date=2022-01-02
     user_hash=4
          date=2022-01-01
          date=2022-01-02

Can someone suggest how can I do this conversion faster ? I have already tried following code snippet

data = spark.read.format("parquet").load("/data-pipeline")
data.write.format("delta").save("/tmp/delta/data-pipeline/")

but it gets rid of the folder structure and create the deltalake format in flat structure

Is it possible to incrementally deep clone/replicate a delta table ?

Hello everyone. Recently, we have received a data stream from Kafka, which contains several fields in the form of text with very large lengths. I was wondering if anyone has any solutions to optimize storage for this data.

Hi folks, I was wondering if there was a way to collect column statistics from delta tables using spark? I am specifically interested in the distinct values. My current presto cluster is dependant on those stats for optimizations

Would want to do this during the etl phase

Read/Write Delta using SAS Enterprise Guide / SAS Viya Hi everyone, Does anybody know if there is a way to read/write from/to delta lake using the analytics tools SAS Enterprise Guide and/or SAS Viya?