this is not deltaTable table it simple df ..

how can convert this df to deltalake

from delta.tables import *
from pyspark.sql.functions import *

In [11]:

deltaTable = DeltaTable.forPath(spark, "tmp/delta-table")

In [12]:

# Update every even value by adding 100 to it
deltaTable.update(condition=expr("id % 2 == 0"), set={"id": expr("id + 100")})

i want to run this code locally with pyspark

but this is not working deltaTable.update .. as deltaTable just db

can any share delta lake example which can run without databrick supports

This may be more of a Spark/Notebooks question… but hopefully some bright folk here can help me anyway 😉 (I've been told flattery always works) I'm using a single Jupyter notebook to test out a series of steps for loading a table, whilst mucking about with the underlying files in the background. If I do the following pseudo-code, I get an error (which I expect and understand why) 1. In a Jupyter notebook, Create a SparkSession 2. Read CSV file, write it to Delta table on S3 3. Read the delta table - works fine 4. Append more data to the delta table 5. Read the delta table - works fine 6. Reset the underlying files to their state after step #2 7. Read the delta table - as expected, this throws an error:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 14.0 failed 1 times, most recent failure: Lost task 2.0 in stage 14.0 (TID 66) (1e1dc78cf259 executor driver): java.io.FileNotFoundException: 
No such file or directory: <s3a://example/data_load/raw/soil/_delta_log/00000000000000000001.json>

It is possible the underlying files have been updated. You can explicitly invalidate
the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
recreating the Dataset/DataFrame involved.

❓ So, on to my question: I've found that if I restart the Jupyter kernel I can re-query the delta table successfully. However, I also thought I'd be able to re-create the spark session:

spark.stop()
    spark = SparkSession.builder.appName […]

to the same effect - but doing this I still get the

FileNotFound

exception, suggesting that something's cached somewhere. I've tried the suggestion in the error message, but this also throws the FileNotFound exception:

refresh table <delta.s3a://example/data_load/my/table>

Is there a way to programatically flush whatever's been cached, without the "turn it off and turn it back on again" approach of restarting the kernel?

Hello, how do you use Spark Streaming with Delta Lake, the pattern “forEachBatch”, when the table does not exists yet?

Hello, I started exploring the capabilities of the DynamoDb log store for the transaction log, and I have a tiny PR where I add some instructions based on the steps I took to ran the test. Please take a look, it's my first attempt to contribute to delta: https://github.com/delta-io/delta/pull/1760

Hello team, I have 1 question. I am using Apache Spark Streaming to run Merge Query to Delta Lake table. My table turn on optimize performance. Meaning after run Merge Query complete, it is automatic run Optimize query. I want to turn of that to improve Streaming batch. How can I turn off this feature ?

Hello team, I have a quick question around delta lake support for aws services;

Do you need to use AWS Glue Crawler (with Native Delta Lake option) to be able to access delta table from AWS Redshift & Athena or are there other marketplace connector options ?

Q2: Is Delta change data feed available from AWS Redshift & Athena ? If so, can you point me to documentation links ?

Q3: Any suggestions/best practices to incrementally load a delta table in S3 to AWS Redshift ?

I seem to remember that delta lake only tracks metadata for the first 30 columns (and thus column ordering matters for improving query performance) but I can't seem to find this in the docs, can someone point me in the right direction?

Hello Delta community, I want to check the progress of open source all Delta to Delta OSS (announcement). I really like Delta but the open source version is missing some critical features like auto compaction. I am also curious about why there is no merged pull request for OSS version in last month

Anyone working with the Rust

delta

crate and could shed some light on how to hook it up to the storage providers it says it supports?

#deltalake-questions I am pretty new to delta tables, tried Googling on differences with vacuum and optimise Could find any good info, can any one share me good documentation that explains technically please

Hello... can someone point me to scala examples to read data from s3 into delta table please ?

What will happen if I merge data in the existing delta table and enable randomizeFilePrefixes? I mean, If I already have data in DeltaTable but disable randomizeFilePrefixes. In one moment, I allowed it and started merging data. Will it break my table?

Hello, I have a question related to Apache Delta vs Apache Hudi for upsert operations. Which one is better? In my company, they made a benchmark six months ago, and Apache Hudi was faster for upserts operations. (hudi 0.12.1 / delta 2.0.1 / iceberg 1.0.0). Is that still true? Because if that is the case, they will use Hudi instead of Delta.

Hello everyone ! I'm coming to you if you know, on the Delta lake, wich size of .parquet file is ideal for a partitionned table to benifit the most in terms of performances ? (ex : 50 GB size of the Table )

Hi, I have a PySpark DataFrame in Databricks. I included the following code multiple times during processing:

df_final.where(col('dh_state_name')=='Wisconsin').where(col('dh_place_name')=='Madison').where(col('property_type')=='Single-family').display()

These should be showing the exact same results, but sometimes it randomly says "Query returned no results". There is no consistency either; the exact same code will show different results across runs. All of the code is deterministic, so I'm confused about why this is happening. Any thoughts?

Hi all, we want to move streaming delta tables from one directory into another. We have used Deep Clone, however it seems that Deep Clone creates last checkpoint in the SAME folder as Delta table and we have defined checkpoints in some different folder. The error we are getting is "Delta table ... does not exist, Please delete your streaming query checkpoint and restart".... Should we rather not use Deep Clone if we have Checkpoints defined in some different folder or do we have to delete the existing Checkpoints (in the separate folder)? Thanks and regards, GapyNi

Is anyone here using Delta Lake 2.3 on Spark 3.4? I have a non-production (yet) project and I'd love to start using 3.4 features. Any breaking issues I may run into? I'm willing to accept some oddities if 3.4 support is around the corner.

What's the release date for delta lake 2.4?

I am using spark 3.3.0 but got a problem when count dataframe with pyspark (python) scan 900Mb too slow with spark (scalar) scan 145Mb 5 times faster why it have different between pyspark and spark statement: df = spark.read.format("delta").load("path").count() help!!! It'll save my day

image.png,image.png

Hello everyone, 😅 I have an intermittent issue when trying to create a Delta table the first time in Databricks: all the data gets converted into parquet at the specified location but the

__delta__log

is not created or, if created, it's left empty, thus the resulted data folder isn't considered to be a Delta table. The command it's a basic one:

sourceDf.write
    .format("delta")
    .mode(SaveMode.ErrorIfExists)
    .partitionBy("date")
    .option("mergeSchema", "true")
    .save(deltaLocation)

If anyone has an idea on this, please share... Thank you in advance, Ovi

Hi all , I have real time fact table update requirements. A fact table will be created by joining multiple azure SQL tables . These SQL table are transactional table. I need to keep the fact table updated with the changes happening in the transactional table. Should I create the fact table as a streaming table in spark ? If yes , has anyone done this already ? If there is any other way please suggest

Has anyone here worked with apache rangers or similar to enable row level based access on delta tables via let’s say trino or similar?