jonathan rowe

08/04/2023, 11:49 AM
hi all, I'm trying to unit test code running locally, but the delta sql extensions/V2 datasource don't seem to be loaded: got spark configuration with: delta_core is version 2.2.0 spark is 3.3.2 sparkSession.sql(""" CREATE TABLE IF NOT EXISTS test (timestamp TIMESTAMP, year INT GENERATED ALWAYS AS (year(timestamp))) USING DELTA PARTITIONED BY (year);""" Syntax error at or near 'GENERATED' year INT GENERATED ALWAYS AS year(timestamp)) -----------^^^ (a) works fine on databricks (b) If I remove GENERATED clause works fine and create delta table in local test Seems the SQL extension is not loaded? anything else I need to do? thanks
Looks like not supported and need to use delta API OPTIMIZE works, so extension registered