Can postgres feed a FDW (foreign data wrapper) to an external table hosted on S3?

Hello! Where can I find, documentation about the config parameters available in Delta? I mean, config params available like:

spark.databricks.delta.schema.autoMerge.enabled=true
spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite

Hi All, Can anyone plz help me for the below questions. 1.Is there any way to list all directory recursively in sorting order or based on last modified date(which tells the latest file or directory) from mount checkpoint using pyspark? 2. How to check whether the file/directory is empty or not in the mount point? Please note that my mount point liknked with ADLS location. Thanks!

Hi All, Is it a good practice to have side effects in the delta live table pipeline? I would like to write data into Postgres DB as a last step.

I am doing some optimization and I am wondering if anybody has found a way to reduce or improve the number of ListBucket operations from delta/spark ?

Hello 👋 I have a question regarding exporting delta tables from one databricks workspace to another. I have a heavy volume data that I processed in lower environment, lets say dev. Now, can I have an option to just export the ingested data into prod and then run incremental jobs on prod to reduce the time/cost/effort?

i cant seem to create a cluster in my databricks workspace, and when I inspect ec2 dashboard there are all manner of instances running and shutting down, even when I have no clusters defined. whats going on here?

the clusterid tag on the ec2 instances also doesn’t match anything in my workspace

and despite going through the cluster creation process multiple times, and always selecting i3.xlarge - the ec2 instances that are running amuck in my AWS account are 2x and 4x… 😕

only way forward appears to be nuking the workspace stack in cloudformation and starting again

Hi all, Any plans on supporting Apache superset for Delta sharing?

Hello, I am using batch processing to load previous day data to write to Delta table which is partitioned by Folder Date and using Overwrite mode. Now my requirement is to run this Job every hour, what would be the best way to do it ? Note: I should have re-run option for a particular date

Hi Team, when i start the delta sharing server using the below command, its not starting properly. Please help. 1877 [armeria-boss-http-*:8080] INFO com.linecorp.armeria.server.Server - Serving HTTP at /[0000000:0%0]:8080 - http://127.0.0.1:8080/ root@Ubuntu:/delta/delta-sharing-server-0.5.4# curl http://localhost:8080/ 404 Not Found

Is it possible to add new parquet files to a delta table without going through spark?

Hi guys Question here, I saved the following table: test.write .format("delta") .mode("overwrite") .option("overwriteSchema","true") .saveAsTable(path), when i try to upsert this table i got the error: "path" is not a Delta table Any advices to solve it ?

Hi everyone, im experimenting with delta + minio for a customer and recognized that i am in need for dynamodb to cover all szenarios in the wild (when i would switch to real aws s3 and over multiple clusters) like -> https://docs.delta.io/latest/delta-storage.html#requirements-s3-multi-cluster I found scylladb with alternator as an replacement for dynamodb -> https://www.scylladb.com/alternator/ but i dont find a way to specify endpoints in delta configs for this LogStore. I think we would need an additional parameter or the Clientbuilder here -> https://github.com/delta-io/delta/blob/master/storage-s3-dynamodb/src/main/java/io/delta/storage/S3DynamoDBLogStore.java#L306 What do you think about this idea?

in databricks, can you set an instance profile at the notebook level?

Autoloader question: Autoloader can ingest JSON, CSV, PARQUET, AVRO, ORC, TEXT, and BINARYFILE file formats. Does it treat wsdl as an text file? wsdl is essentially xml file. thank you.

Hi guys, what's the difference between "optimize compaction bin-packing" and "repartition" in terms of combing small files into fewer big files? Do we always suppose to choose "optimize operation" over manually re-partition?

Hello all , cross posting a question from #deltalake-on-aws , https://delta-users.slack.com/archives/C018KR1D43T/p1676330095877139

Hi all, I’m running issues when trying to consume from Kafka using structured streaming and writing data on delta lake. The delta libraries are loaded in a docker image I created to run Spark on K8s using the spark operator. The problem appears when we put the following dependency in the pom:

groupId = org.apache.spark
artifactId = spark-sql-kafka-0-10_2.12
version = 3.3.1

The application runs in local without problems, but when distributed over a K8s cluster, it breaks with the exception

Caused by: java.lang.ClassNotFoundException: delta.DefaultSource

Anybody experienced something like this? thx Logs here:

2023-02-13 16:40:37.990Z  INFO  org.apache.spark.storage.BlockManagerMaster:61 - BlockManagerMaster stopped
2023-02-13 16:40:37.996Z  INFO  org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:61 - OutputCommitCoordinator stopped!
Exception in thread "main" java.lang.ClassNotFoundException:
Failed to find data source: delta. Please find packages at
<https://spark.apache.org/third-party-projects.html>

	at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToFindDataSourceError(QueryExecutionErrors.scala:587)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:675)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:725)
	at org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:864)
	at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:256)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
	at ....
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at com.izicap.delta.loader.DeltaLoaderTableLauncher$.run(DeltaLoaderTableLauncher.scala:155)
	at com.izicap.delta.loader.DeltaLoaderTableLauncher$.main(DeltaLoaderTableLauncher.scala:178)
	at com.izicap.delta.loader.DeltaLoaderTableLauncher.main(DeltaLoaderTableLauncher.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at <http://org.apache.spark.deploy.SparkSubmit.org|org.apache.spark.deploy.SparkSubmit.org>$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: delta.DefaultSource
	at java.base/java.net.URLClassLoader.findClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:661)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:661)
	at scala.util.Failure.orElse(Try.scala:224)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:661)
	... 32 more

Hi All, I am writing a Java Spark application that writes to a Delta table without any partition. Another application updates and/or deletes rows using a query on one of the fields in the data. I'm using the following Delta API:

DeltaTable.forPath(sparkSession, hdfsPath)
                    .alias("oldData")
                    .merge(data.alias("newData"), deltaMergeQuery)
                    .whenMatched()
                    .updateAll()
                    .whenNotMatched()
                    .insertAll()
                    .execute();

The deletion will be in a similar way. Now to the question: Since there are no partitions on the fields on the query, does it mean that Spark will have to open and read all files? Thanks in advance!

Hello everyone - We are using Delta Sharing to enable our customers to access their data in Power BI, but our customers are complaining that the imports are taking way too long (hours) because they have no way to simply ask for e.g. the last 30 days of data or some filter like that. Are we missing something obvious? because if the user has to reimport their 100 million + rows of data every time, then this will be unusable for them. Any tips/tricks or features, capabilities we are unaware of? Thank you in advance for your insight.

Hi guys there is some resource about unit test in databricks? I was using databricks connect, in official documentation it is recommended to use dbx

This feels like a relatively simple question, but when I ALTER a table, what is actually happening at the protocol level?

Can I use autoloader in Databricks community?

Write to delta table is failing due to parallel writes. I cannot use the partitions here. Is there any workaround to avoid such issues.

I have an interesting/werid issue that I can not explain what is happening: I have several delta tables where

change-data-feed

is enabled. And it has been working fine. Today though, for one of the tables we get a weird error message when we try to use

table_changes("the_table", <commit>)

, namely:

No file found in the directory: <gs://bucket/raw/.../_delta_log>

1. The

_delta_log

is at the location 2. I can query the table itself

select * from the_table

3. I can load previous versions

select * from the_table@v100

4. But this fails:

select * from table_changes("the_table", 100)

, with the above mentioned error Nothing in our logic has changed for the specific table where this is happening for. Other tables that run on the same logic of ingestion etc. don't have this specific issue. Anyone an idea what could be going on? We use Databricks on GCP

Hi guys, I see that the vacuum operation is being logged in the transaction history for delta 2.0.2. But the same functionality has not been included to the newer versions i.e for delta 2.1, 2.2 Any particular reasons why its not yet available in the newer versions, when can we expect it to be released ?

as

VACUUM

deletes orphand and outdated files, has anyone ever had a case where • a long UPDATE/MERGE/DELETE operation was running that creates data files (parquet) •

VACUUM

is run at about the same time and cleans up those temporary orphaned files. Orphaned in a way that the log file has not yet been written as the concurrent UPDATE/MERGE/DELETE operation was still running?