https://delta.io logo
p

Pip Mk

05/01/2023, 5:39 PM
Hi all, I've changed the configs of my delta table (
minWriterVersion
,
minReaderVersion
,
columnMapping
) to support the
DROP COLUMN
command. It worked: when I query the table with Athena, the dropped column doesn't show up. But it still appears on Glue Catalog. If I try to run a Glue crawler on that table, it throws an error ("Internal Service Exception", no further information). I assume it's because Glue crawlers don't support the
DROP COLUMN
command and I need to revert the config changes. So my question is: is there any other non-manual way of updating the Glue catalog if I wanna use the `DROP COLUMN`command? Or perhaps some table property I can add on Glue catalog so that it works?
I removed the configs and redid a full load of the table. It's fine because I don't think I'm gonna drop columns very often, but it'd be nice to have that functionality working with glue crawlers