https://delta.io logo
s

Simon Thelin

08/10/2023, 7:48 PM
How can I run:
Copy code
ALTER TABLE <table> SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
On a delta table outside of a databricks runtime? via a spark shell? I don’t use databricks.
t

Tom van Bussel

08/10/2023, 7:50 PM
Yes, you can just run
spark.sql("ALTER TABLE …")
in Spark Shell.
s

Simon Thelin

08/10/2023, 7:52 PM
but how do I point to the actual table?
because register it as a view is not right here
should I add location?
t

Tom van Bussel

08/10/2023, 7:52 PM
Copy code
ALTER TABLE delta.`<scheme://path/to/table>` SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
s

Simon Thelin

08/10/2023, 7:53 PM
okay I tried
Copy code
ALTER TABLE 's3a://....'
or wait I did not use
Copy code
`
t

Tom van Bussel

08/10/2023, 7:53 PM
Spark uses
Copy code
format.`path`
👀 1
s

Simon Thelin

08/10/2023, 7:54 PM
@Tom van Bussel thanks so much! I could not find this in the delta docs for CDF!
will try this now…
Okay It does not work but I might be doing something incorrectly. I tried:
Copy code
spark.sql("ALTER VIEW `scheme:<s3a://bucket/path/table_name>` SET TBLPROPERTIES (delta.enableChangeDataFeed = true)")
And
Copy code
spark.sql("ALTER VIEW `<s3a://bucket/path/table_name>` SET TBLPROPERTIES (delta.enableChangeDataFeed = true)")
but that did not work 🤔
omg nvm I am high
t

Tom van Bussel

08/10/2023, 7:56 PM
Yeah, it’s more of a Spark thing than it is a Delta thing, but we do use it in some examples in the docs
ALTER TABLE not ALTER VIEW
s

Simon Thelin

08/10/2023, 8:01 PM
Yeah I was too stressed…. Still does not seem to work 🤔
Copy code
spark.sql("ALTER TABLE `s3a://<bucket>/<path>/<table_name>` SET TBLPROPERTIES (delta.enableChangeDataFeed = true)")
Copy code
spark.sql("ALTER TABLE `scheme:s3a://<bucket>/<path>/<table_name>` SET TBLPROPERTIES (delta.enableChangeDataFeed = true)")
Copy code
+- 'UnresolvedTable [<path>/<table_name>], ALTER TABLE ... SET TBLPROPERTIES, Please use ALTER VIEW instead.
Strange…
I know I have permissions because I can read the table vanilla DeltaTable or vanilla spark.read style
the table has a dash in the name, that might be why
table-1234
. This table was originally created by trino
t

Tom van Bussel

08/10/2023, 8:03 PM
It needs to be
Copy code
spark.sql("ALTER TABLE delta.`s3a://<bucket>/<path>/<table_name>` SET TBLPROPERTIES (delta.enableChangeDataFeed = true)")
👀 1
1
s

Simon Thelin

08/10/2023, 8:04 PM
Success, wow this is awesome @Tom van Bussel! Thanks that worked!
t

Tom van Bussel

08/10/2023, 8:04 PM
👍
s

Simon Thelin

08/10/2023, 8:05 PM
We are doing some wild stuff with delta table created by trino, enabling CDF for a downstream CDF app 👀 …. trino adds a random id in S3 but when queried via metastore it looks like it does not have the random id, so was worried this would cause trouble but since you just point to the full path in s3 it does not matter.