This message was deleted.

Hey there, does delta (via databricks in this case) write out multiple parquet row groups in parquet files?

This message was deleted.

Hey everyone! Just joined and I have a few questions: • What version of delta lake is used in Synapse? • Is there a query I can run to answer my own question?

With partitioning, does a query have to match the partition exactly? If I have a query using between date1 and date2 but they are full timestamps and my date1 and date2 uses times like 000000.000000 will it still use a partition like ingeat_date=2023-01-01 125620.246314?

is there a way to write a small dataset to delta without spark or a spark cluster? if yes, what language support that? i know pandas can insert records directly into a delta table, but it still needed to connect to spark cluster.

Autoloader question: Can autoloader detect, when a new feed is missing columns? From what I see, it automatically fills the missing columns with nulls.

👋 not really a question but mainly feedback: on the delta lake documentation for table optimization it wasn't clear to me that z-ordering will also perform compaction as part of the operation. after reading that page i was thinking that i'd have to run compaction and z-ordering as two separate commands, but after looking at the code i realized that they both perform compaction and use mostly the same logic under the hood (with the exception of multidimensional clustering). this might be worth clarifying 🙂

Anyone has successfully read/write iceberg table in databricks environment using glue as catalog? I was able to successfull read iceberg tables but when I try to write Databricks is failing "NoSuchCatalogException: Catalog 'my_catalog' not found" my catalog is virtual catalog for iceberg

qq: so currently we can use the wrapper pypi package to write json/df to GCS: https://delta-io.github.io/delta-rs/python/usage.html#writing-delta-tables without spinning up a spark cluster? or if without a spark cluster have to use the rust crate

delta-rs

directly?

Design decision: batched inserts vs. lots of appends + regular compaction? We'd like to store process data of a Workflow Management System (WFMS) in a Delta table. We have an Azure Function (JAVA) that is triggered ~300k times per day and fetches the process data from the WFMS in near real-time. The triggers are not evenly distributed over the day, but rather with two peaks in the late morning and in the afternoon, with 10k triggers per minute. Currently, we consider two options to ingest the data to a Delta Table: 1. Having the Azure Function(s) write the data to JSON files (300k files per day) in an Azure Data Lake Gen2 and then read newly arrived data in batches every 15 minutes to a Spark Dataframe and append it to the Delta Table. The Azure Function would create a new folder every 15 minutes for the JSONs to prevent excessive file listing when reading with Spark. 2. Having the Azure Function(s) to directly write to the Delta Table using Delta Standalone library. Since every append will create a new parquet file and new Delta version in the Delta table, we'd need regular "compaction" (

OPTIMIZE

) in order to keep the Delta table healthy. We are currently leaning towards (1). What are your thoughts?

Delta Lake

VACUUM

command is taking 15 mns every day for a very small table. I even enabled

delta vacuum parallel delete option

. Still no improvement. Anyone any suggestion to improve this?

We have a large timeseries database in Druid - say 300M records; Let's say we move this to Delta ; What are our options for querying this data using various rich filters, groupby conditions and perform slice and dice queries on this data from non spark JVM ? It seems Delta standalone is insufficient. Does Presto/Trino etc. help in this case?

I am trying to execute zorder on a delta table and running into an error. Any clues?

Error - org.apache.spark.SparkUnsupportedOperationException: Cannot evaluate expression: rangepartitionid(input[14, string, true], 1000)
Versions - delta 2.1.0, spark version '3.3.0+amzn.1.dev0'
errors at - mytable.optimize().where("parititionfield=somevalue").executeZOrderBy("col1", "col2")

Considering Spark 3.4 is almost released, there is any plans to release a new Delta version compatible to 3.4?

Hi Folks , I am getting XML data from source converting it into delta format. Afterwards I need to create a azure SQL Tableau on top of that delta table. How do I keep the azure SQL Table updated with the incremental changes in delta table ?

Hi, Based on the databricks doc for GCP https://docs.gcp.databricks.com/sql/language-manual/delta-copy-into.html It seems Copy into a delta table from Google Cloud storage is not supported, am I correct ? or am i missing something ? Thanks

Has anyone access External Location or Storage Credentials in UC enabled workspace in private endpoint.

Hi, might a silly question, but have been stuck for some time on this. My delta table detail shows the name as null. It was created by saving a dataframe as delta. Is there a way to update the name?

Hi Everyone!! I have a huge table which has some key data in three columns.. data type, column name and value.. I have data as is in raw/Bronze layer.. now I have DLT pipeline where I need to Apply Pivot to get all this information at column level.. DLT does not simply support Pivoting for streaming data I need to apply transformation using UDF and Foreachbatch… which seems complicated and unnecessary.. just wanted to see should I just apply pivoting at raw layer or using above foreachbatch.. any views or anyone has done same in past.. appreciate help

Hi All - I'm facing problem with Pandas 2.0.0 release wherein when I call toPandas() on a dataframe it's failing here is a link 🔗 that produces the issue - https://stackoverflow.com/questions/75804781/how-to-create-pyspark-dataframes-from-pandas-dataframes-with-pandas-2-0-0 . Now , I have moved to Pyspark 3.4.0 , now while doing so again it's failing as Latest Pyspark is not compatible with Delta 2.3.0 . Is there a workaround? I just saw databricks runtime 13.0 where they are using Pyspark 3.4.0 with delta 2.30 - Just wondering how can we achieve the same in local? https://docs.databricks.com/release-notes/runtime/13.0.html

Hi all, I saved a Delta Lake table to GCS without a name. How can I rename it or set a name for it? I’m using Delta 2.0.0 in Spark 3.2.2. Appreciate any help.

Hi Team, I'm using OSS delta 2.3.0 and spark 3.3.2. I'm using the mentioned configurations to set the max file size (optimize.maxFileSize=500*1024*1024) to 500mb I was playing around with configuration and noticed that. If i set the file size to 500mb offcourse the part file are adhered to the configs but when I change the max file size (optimize.maxFileSize=128*1024*1024) to 128mb, the compaction is not working. Is there any extra configurations I'm supposed to add or doing a repartition or coalesce is the only option as of now ? PS - The other way round of compaction works i.e. 128mb to 500mb

Hi Everyone, I'm using delta 1.0.0 version and spark 3.1.2 I'm trying to extract data using spark jdbc with partition column, no of partitions, lowerbound & upperbound values but somehow even though partition column has lot of distinct values. In spark sql UI, It's showing numpartitions as 1. The job basically runs forever and fails with connection denied error. Can anyone shed some light into this if I'm missing any key argument. Sample code:-

df = spark.read.jdbc("jdbc:db2//host.com:400/server", "db.table", properties=conn_prop :  {'user': 'user', 'driver': 'com.ibm.db2.jcc.DB2Driver', 'customSchema': 'a smallint, b string, c int, d int', 'DELIMIDENT': 'Y', 'partition_column': 'd', 'num_partitions': '20', 'lower_bound': '2022-02-26', 'upper_bound': '2023-04-04', 'securityMechanism': '18', 'sslConnection': 'true', 'sslTrustStoreLocation': 'path', 'sslKeyStoreType': 'PKCS12', 'sslKeyStorePassword': 'xxxx', 'sslKeyStoreLocation': 'path_to_key', 'sslTrustStoreType': 'jks'})

Is this is correct to use a bronze layer as a delta file(map file with target schema), instead of ingesting the Raw (as-is) file in Medallion architecture? I know there is a cons, but would like to understand if anyone has implemented this kind of architecture? Later on, are they find any issues with this? Appreciate your reply to this

Hi everyone, I'm currently working on a code snippet to demonstrate the basics of Delta Sharing. While working on this, I came across the following implementations: • https://github.com/delta-io/delta-sharing/blob/7f0126073db3bf26c29648687c996db20e1b64b5/python/delta_sharing/__init__.py#L23 • https://github.com/delta-io/delta-sharing/blob/7f0126073db3bf26c29648687c996db20e1b64b5/python/delta_sharing/delta_sharing.py#L54 • https://github.com/delta-io/delta-sharing/blob/7f0126073db3bf26c29648687c996db20e1b64b5/python/delta_sharing/rest_client.py#L271 After reviewing these implementations, it seems that the Python client library does not consider the JSON predicates public. I would like to demonstrate this functionality, but should I implement my own API by adding

predicateHints

parameter to load_as_pandas for now? Thanks.

I need to import a parquet table to make it accessible via delta tables. it's about 55GB, but there's an extra string field in all the records that I'd like to remove, since it can be reconstructed from other fields. So, i don't want to ingest the data into a delta table yet, as it would create metadata for each of the records. ideally, i'd like to ingest it into a new delta table entirely (clone from

mp

into

lake

) I'm looking at the partition discovery section of the Spark SQL guide (and a few other sources), but I can't quite figure out how my data will be treated when it's pulled in. the directory structure looks like this, except I've used a regex to change those directory names to

participant_id=$id

. inside each directory, there are parquet files whether the name is $id.parquet, but they're not in a directory named

sequence_id=$id

.

.
├── lake
└── mp
    ├── sign_to_prediction_index_map.json
    ├── train.csv
    └── train_landmark_files
        ├── 16069
        ├── 18796
        ├──  ...
        ├── 61333
        └── 62590

is the directory required? or can i just rename the files to

sequence_id=$id.parquet

? I'm getting ready to try messing with a DLT using just one participant's data to see what happens. I haven't extracted any additional data, but

I can't seem to get COPY INTO to work. Here's what I'm doing. I'm not sure if it's because of the parquet partition path format, but the Spark SQL seems to be rejecting the syntax. https://github.com/dcunited001/zettelkasten/blob/master/slips/20230418022448-data_lake_etl_for_parquet_files_with_delta_lake.org

Hello the team! I would like to have a clarification when storing the metadata on s3. Let's say that I'm consuming my input data either from s3 either from kafka and I write in the Delta Lake that will be store in S3 (metadata included). May I face a consistent issue if I parallelize the writes in the same Delta Lake table inside the same component ? It will be the only component that will writes in the Delta Lake tables. Thanks 🙂

Hi guys, Is there a way to use subquery in pyspark ?

(select sum(case faturado.shkzg

when 'S' then faturado.menge

else - faturado.menge

end)

from sapsr3.ekbe faturado where faturado.ebeln = b.ebeln and b.ebelp = faturado.ebelp and faturado.vgabe = '1' -- entradas

) as qtd_faturada

Did not find any useful resource If someone have some tip i will appreciate, thanks