https://delta.io logo
v

Vigneshraja Palaniraj

05/25/2023, 8:56 PM
Hi, I recently read that INSERT INTO support is available for Generated Columns in Spark3.4 release. I tested that by running the below scenario and it still seem to fail. Could you kindly help if this support is available in Spark3.4 %sql CREATE TABLE default.delta_table_test_v13_4 ( id INT, name STRING, created_date TIMESTAMP, modified_date TIMESTAMP, total_sales DOUBLE, total_profit DOUBLE GENERATED ALWAYS AS (total_sales * 0.2) ) USING DELTA; # Created a table delta_test with the columns id, name, created_date, modified_date, total_sales %sql INSERT INTO default.delta_table_test_v13_4 SELECT id, name, created_date, modified_date, total_sales from default.delta_test; Error in SQL statement: DeltaAnalysisException: Cannot write to 'spark_catalog.default.delta_table_test_v13_4', not enough data columns; target table has 6 column(s) but the inserted data has 5 column(s)
a

Allison Portis

05/25/2023, 10:30 PM
What Delta version did you use here?
v

Vigneshraja Palaniraj

05/26/2023, 7:58 AM
@Allison Portis I used the DBR latest Runtime 13.0 that has Spark version 3.4 in the Community edition. I believe it also has the latest delta version
a

Allison Portis

05/26/2023, 7:17 PM
You need to insert into by name for this to work. i.e.
INSERT INTO tbl(id, name, ...)
👍 1
v

Vigneshraja Palaniraj

05/28/2023, 10:54 AM
Thanks. Let me try that