Hi Everyone, I'm using delta 1.0.0 version and spark 3.1.2 I'm trying to extract data using spark jdbc with partition column, no of partitions, lowerbound & upperbound values but somehow even though partition column has lot of distinct values. In spark sql UI, It's showing numpartitions as 1. The job basically runs forever and fails with connection denied error. Can anyone shed some light into this if I'm missing any key argument. Sample code:-

df = spark.read.jdbc("jdbc:db2//host.com:400/server", "db.table", properties=conn_prop :  {'user': 'user', 'driver': 'com.ibm.db2.jcc.DB2Driver', 'customSchema': 'a smallint, b string, c int, d int', 'DELIMIDENT': 'Y', 'partition_column': 'd', 'num_partitions': '20', 'lower_bound': '2022-02-26', 'upper_bound': '2023-04-04', 'securityMechanism': '18', 'sslConnection': 'true', 'sslTrustStoreLocation': 'path', 'sslKeyStoreType': 'PKCS12', 'sslKeyStorePassword': 'xxxx', 'sslKeyStoreLocation': 'path_to_key', 'sslTrustStoreType': 'jks'})

You have a lot of config parameters I have never used. Maybe try to a '(SELECT CURRENT_TIME()) AS q' as table so you can verify your connection first and foremost.

@Lennart Skogmo Yes, connection works fine because It was working fine for some tables (smaller one's) but not for biggest one's.