Hello-

Does anyone know why my delta log json files might be missing minvalue and maxvalue properties and values?

Example: {"commitInfo":{"timestamp":1684837794183,"operation":"WRITE","operationParameters":{"mode":"Overwrite","partitionBy":"[]"},"readVersion":4,"isBlindAppend":false,"operationMetrics":{"numFiles":"4","numOutputBytes":"47035915","numOutputRows":"517737"}}} {"add":{"path":"part-00000-0e4aaf3e-3471-42e9-9977-7f248c5f9931-c000.snappy.parquet","partitionValues":{},"size":12591124,"modificationTime":1684837792436,"dataChange":true}} {"add":{"path":"part-00001-2ec1bbf3-57a4-4159-b780-119c66fb62bf-c000.snappy.parquet","partitionValues":{},"size":12587807,"modificationTime":1684837793642,"dataChange":true}} {"add":{"path":"part-00002-d332fb7f-e6ed-4d95-a004-2dfa6de480e0-c000.snappy.parquet","partitionValues":{},"size":12385161,"modificationTime":1684837792518,"dataChange":true}} {"add":{"path":"part-00003-9eb30dbe-e3a4-45e5-a39d-8aaefe1f6a68-c000.snappy.parquet","partitionValues":{},"size":9471823,"modificationTime":1684837793003,"dataChange":true}} {"remove":{"path":"part-00000-db74addd-8ca9-475f-b9c3-361f289b87f7-c000.snappy.parquet","deletionTimestamp":1684837794183,"dataChange":true,"extendedFileMetadata":true,"partitionValues":{},"size":2342}}

I have a common question, but didn't find a simple solution online. How do I remove duplicated rows from delta table? I'm using azure databricks and reading an external delta table stored on azure storage account

Dear all, we are loading data into bronze layer with "Append" option by using Autoloader. We are than reading them in the silver layer with

readStream.option("startingVersion", lastMaxVersionFromAppend)

, where we do

foreEachBatch

in the next step to merge them into silver layer. Now the question is as i looked into the metrics it seems that the process read all the

numAddedFiles

but the

numOutputRows

(from bronze layer) does not exactly match with the

numSourceRows

from silver layer, when performing

forEachBatch

merge commando. Do you know what could be the reason? Thanks and regards, GapyNi

Hey all, I am using isDeltaTable method (Delta version 2.0.0) in my Jupyter notebook and have weird behavior with this code:

DeltaTable.isDeltaTable(spark, "/path/to/table")

For given root path:

"/path/to/table"

if the sub path

"/path/to"

has deltalog directory, the function return True. The Documentation the provided is: "classmethod *`isDeltaTable`*(sparkSession: pyspark.sql.session.SparkSession, identifier: str) → bool Check if the provided identifier string, in this case a file path, is the root of a Delta table using the given SparkSession." _(_https://docs.delta.io/latest/api/python/index.html) So the question is : if the root the I provided path don't contains Delta table , but the sub-root path has delta table and 'isDeltaTable' returns True - is this not a bug? or am I missing something?

Can you provide a lock state for a delta table via dynamodb if you run spark clusters in k8s?

Hi everyone, is there a library or something that i can use so that i can partition my spatial dataset following the hilbert curve on the databricks runtime? Thank you in advance

Hello Everyone, I am currently facing a challenge with Delta tables regarding how to design the write flow. I am working on integration between an Open Source ML monitoring project developed by my company and Delta lakehouses. The solution currently support Influxdb for data storage, loose schema requirement in InfluxDB due to the way it stores the data makes it satisfying solution to handle such cases. We want to also leverage Delta tables for data storage. I am having a hard time wrapping my head around the best way to leverage delta tables for time-series where the schema/metadata for every ingested timeseries dataframe/chunk changes depending on the ML model version/meta-data that might from a model version to another. The schema enforcement in Delta is a disadvantage in this case, how to work with that feature? is it a good idea to create a new table when schema/metadate changes? Any other course of action? What are your suggestion to deal with Delta specificities? I am pretty new to the delta eco-system, and the learning curve is pretty steep. I am always happy to answer any concerns and give clarity on what I am trying to achieve, Thank you for your help!

Hello, has anyone saved a geopandas table (including the

geometry

column) as a Delta table in Databricks?

Hello Everyone, How to Read parquet data files using delta connector when the data files committed to delta table via standalone library ? Am using Apache Avro and Apache Parquet libraries directly to convert data to the Parquet format.

System.out.println("Delta Read");
spark.read().format("delta")
        .load("<s3a://delta-laketest/spark_table>")

Also when am running this(above code) it Does not read the parquet files. As the logs and checkpoint files points to the parquet files registered. How do i read it using delta connector ? 2nd way: Reading Parquet data (single-JVM)

DeltaLog log = DeltaLog.forTable(conf, "<s3a://delta-laketest/my_table>");
CloseableIterator<RowRecord> rowItr = log.snapshot().open(); //update().open();

Code above was able to Read parquet data files after committed to delta table via standalone library. Is this correct way to Read parquet data files using delta connector when the data files committed to delta table via standalone library ? I would like to understand whats the difference in both ways? I am pretty new to the delta, please guide me through this. Thank you for your help! :)

Hello All , Just started exploring Delta Lake features . using Delta 2.3.0 spark 3.3.x python 3.6.8 When installing :pip3 install delta-spark==2.3.0 Error: WARNING: Running pip install with root privileges is generally not a good idea. Try

pip3 install --user

instead. Collecting delta-spark==2.3.0 Cache entry deserialization failed, entry ignored Using cached https://files.pythonhosted.org/packages/34/9e/c06f3b701de4746defc240fe7a2cc973f7bbfaa8fa17d57e045868c16925/delta_spark-2.3.0-py3-none-any.whl Collecting pyspark<3.4.0,>=3.3.0 (from delta-spark==2.3.0) Cache entry deserialization failed, entry ignored Could not find a version that satisfies the requirement pyspark<3.4.0,>=3.3.0 (from delta-spark==2.3.0) (from versions: 2.1.2, 2.1.3, 2.2.0.post0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.1.1, 3.1.2, 3.1.3, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 3.2.4) No matching distribution found for pyspark<3.4.0,>=3.3.0 (from delta-spark==2.3.0) According to the docs .

Hi, Is there any way to manage delta tables schemas externally instead writing a dataframe with the new table schema and merge it?

Is there a docker image of delta lake so I can run everything locally (with examples). I see open github issues. https://github.com/delta-io/delta/issues/919

Hi, I recently read that INSERT INTO support is available for Generated Columns in Spark3.4 release. I tested that by running the below scenario and it still seem to fail. Could you kindly help if this support is available in Spark3.4 %sql CREATE TABLE default.delta_table_test_v13_4 ( id INT, name STRING, created_date TIMESTAMP, modified_date TIMESTAMP, total_sales DOUBLE, total_profit DOUBLE GENERATED ALWAYS AS (total_sales * 0.2) ) USING DELTA; # Created a table delta_test with the columns id, name, created_date, modified_date, total_sales %sql INSERT INTO default.delta_table_test_v13_4 SELECT id, name, created_date, modified_date, total_sales from default.delta_test; Error in SQL statement: DeltaAnalysisException: Cannot write to 'spark_catalog.default.delta_table_test_v13_4', not enough data columns; target table has 6 column(s) but the inserted data has 5 column(s)

Hello !, Does anyone have a working setup where some code running inside a docker image, tries to create a delta table? If yes, what image do you use and can you share your

Dockerfile

? Our usecase is we're running some e2e tests (written using

pytest

) in a Jenkins pipeline, that create some delta tables and perform merges etc. The Jenkins agent is docker image with

alpine

as base (full Dockerfile attached). Problem is that snappy doesn't like the environment and we get following error (more complete stack trace attached):

py4j.protocol.Py4JJavaError: An error occurred while calling o162.saveAsTable.
: org.apache.spark.SparkException: Job aborted.
   at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:638)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9.0 (TID 13) (k-ci-fgao-2ffxcdata-10-2franking-report-by-session-type-1-wx95c executor driver): org.apache.spark.SparkException: Task failed while writing rows.
   at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:642)
Caused by: java.lang.IllegalArgumentException
   at java.nio.Buffer.limit(Buffer.java:275)
   at org.xerial.snappy.Snappy.compress(Snappy.java:156)
   at org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:78)
   ... 19 more

I've tried many workarounds (to make snappy use some other writable-tmp-folder), but none work. So I'm hoping to create a docker image where I can write a simple python code that creates a delta table and writes to it. Thank you!

Hello Everyone, When a parquet file is committed to delta table using Delta Standalone API, does the delta log created follow "Schema Enforcement" and Schema Evolution" rules ?

Hello, I am using the delta standalone library to read my data. I see that the latest delta protocol is not supported. I keep getting an error

No delta log found for the Delta table at

Is there some way for me to specify the minreader and minwriter versions?

@Nick Karpov Any chance we can release delta-spark 2.4.0 on Conda as well?

Hi there, I have an int64 column and would like to know if it is possible to force delta encoding when Delta Lake/Spark writes the parquet file. And second, if it is possible to introspect the parquet files to check which encoding is used.

hey folks, I am in the attached image, you are seeing the schema for my table. Problem statement: one my column have json object, and in this json object one of the key have space in it’s name (physical input read time) and due to this i am not able to write this as delta table as this is a part of my column schema. I need to replace spaces with “_”. I have tried several ways but I am not able to do it. PLEASE HELP ME IN THIS

Good morning. I started working with Delta Lake and I am new to this whole process. We have installed a delta lake, spark, EMR, Pulsar with the aws cloud and created a database in aws glue in the Data Catalog and there several tables with their location in the S3 and Classification type parquet. I want to know if when creating these tables here are delta type or where they should be created to be delta type. Thank you very much for your collaboration.

hey folks, I have some dataframes with inferSchema = True and some columns have a nullType specified automatically. When I try to write using Delta, I get this message:

Delta doesn't accept NullTypes in the schema for streaming writes.

Is there a way to write delta using null type? I did some searches but I didn't find anything until the moment

Hey folks. I have a delta table, generated in Databricks with proper schema. However, Athena & Glue only recognize the schema as a single column of type

array<string>

. Has anyone had this problem before? Running `SHOW COLUMNS IN table_name`in Athena shows a list of column names as expected.

Hello I have a Delta table that is defined by a path and is partitioned by time. Currently, when I use the Spark reader to count all the records in the table, it scans the file system and retrieves files in order to perform the count operation. However, I believe that the necessary information to determine the count of records already exists in the checkpoint.parquet file. I would like to know if there is a way to leverage the existing checkpoints and retrieve the count more efficiently, without the need to scan the file system or retrieve additional metadata (just use computed snapshot).

Hello Everyone, Testing Schema Enforcement feature using Delta Standalone. Running multiple JVM’s updating the Schema of delta table. Findings : Detected incompatible schema change in 2nd JVM. Questions: 1. This detection is based on metadata update ? 2. Can this feature be disabled ?

HI, I am currently working on a project where I have stored my tables as individual Parquet files in HDFS. To leverage the benefits of Delta Lake, such as data versioning and easier data transformations, I am looking to convert these Parquet files to Delta Lake format. I have a few questions regarding the conversion process and would greatly appreciate your guidance. *1-*Firstly, I would like to know if it is necessary to convert each Parquet file individually to a Delta table. Since I only need a subset of tables, I don't want to convert all of them. Alternatively, can I convert the directory containing the Parquet files into a single large Delta table? My concern here is whether I will be able to access and manipulate each table individually, such as performing joins and selecting specific columns, after consolidating them into one Delta table. 2- my second question, I am ingesting data from a PostgreSQL database into HDFS using Apache NiFi. My intention is to configure the ingestion process to continuously capture new data from the database and store it in the same Parquet files in the data lake. In this scenario, would it be necessary to reconvert the Parquet files to Delta Lake every time new data appears, or is there an alternative mechanism that ensures the seamless integration of new data into the Delta Lake format?. I genuinely appreciate any assistance or recommendations.

Hello Everyone, I am working on a project where we are trying to save table using delta lake on a ADLS gen 2 using a SAS token. Is this even possible because I am not able to find any documentation for it. It only talks about oAuth for now for ADLS gen2.

Hello Everyone, Please help me with the below scenario? Scenario- RawDf written to delta table with schema - Id, Name, Age, Salary Incremental data comes in with an additional column - Id, Name, Age, Salary, Designation I'm using merge as below: targetTable.alias("tgt").merge(df.alias("src"), condition = "tgt.id = src.mergeKey and tgt.iscurrent = true").whenMatchedUodate(set = {"enddate ": currenttimestamp(), "isCurrent ": "false"}).whenNotMatchedInsertAll().execute() Spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", True) --> this is also enabled But new column I..e Designation not inserted. Am I doing anything wrong here...please help. Thanks!

Can Schema Enforcement feature be Disabled ? Written the parquet files to delta table using Delta Standalone.