Does anyone by any chance know of a slack/chat where spark/pyspark/jdbc developers hang around? I'm playing around with a small framework and something seems to have changed from databricks LTS 10.4 to LTS 11.3 that causes reading with pyspark-jdbc-sqlserver to generate SQL ending in a invalid 'WHERE (1)' when there should not be any where clause at all as far as I can tell. It seems to be caused by joining/comparing incoming dataframe from sqlserver to a empty delta table before writing. Once the table has some data the exact same query runs fine.
02/25/2023, 6:43 PM
There is the apache mailing list, but otherwise doesn’t exist.
02/25/2023, 6:44 PM
Ok thanks 🙂 too bad. I suppose it could be related to some of the Spark 3.3.0 pushdown or jdbc changes.
I've been testing some more and it does indeed seem to be a bug. I will have a look at the mailing list and see if I can get it reported 🙂
Problem seems to still be around on the new DBR 12.2 Beta / Spark 3.3.2
No response on the mailing list so far.
Seems like the issue tracker needs another separate signup 😛 I guess I should just report to databricks instead.