Does anyone by any chance know of a slack chat where spark p

Question

Does anyone by any chance know of a slack chat where spark pyspark jdbc developers hang around I m playing around with a small framework and something seems to have changed from databricks LTS 10 4 to LTS 11 3 that causes reading with pyspark jdbc sqlserver to generate SQL ending in a invalid WHERE 1 when there should not be any where clause at all as far as I can tell It seems to be caused by joining comparing incoming dataframe from sqlserver to a empty delta table before writing Once the table has some data the exact same query runs fine

JosephK (exDatabricks) · Answer

There is the apache mailing list but otherwise doesn t exist

Lennart Skogmo · Answer

Ok thanks slightly smiling face too bad I suppose it could be related to some of the Spark 3 3 0 pushdown or jdbc changes

Lennart Skogmo · Answer

I ve been testing some more and it does indeed seem to be a bug I will have a look at the mailing list and see if I can get it reported slightly smiling face

JosephK (exDatabricks) · Answer

<https issues apache org jira projects SPARK issues> should work

Lennart Skogmo · Answer

Problem seems to still be around on the new DBR 12 2 Beta Spark 3 3 2 No response on the mailing list so far Seems like the issue tracker needs another separate signup stuck out tongue I guess I should just report to databricks instead