This feels like a relatively simple question but when I ALTE

Question

This feels like a relatively simple question but when I ALTER a table what is actually happening at the protocol level

Scott Sandre (Delta Lake) · Answer

a new `Metadata` action is committed to the table commit gt a new `N json` delta log commit file is appended to the ` delta log` this new metadata action will contain the new schema or latest table properties

rtyler · Answer

ah so the whole new schema gets committed as a new action and then the readers are expected to basically just use the latest metadata action in the log right

Scott Sandre (Delta Lake) · Answer

yup

rtyler · Answer

in the spark implementation is there any form of ALTERs which would cause parquet file modifications I know the schemas of the parquet file and the delta log needn t be in lock step

chris fish · Answer

nope

chris fish · Answer

only modifies `Metadata`

chris fish · Answer

i think the approach is the opposite it treats the parquet files as immutable and if you try to make a schema modification that doesn t work it fails

rtyler · Answer

without reading all the parquet files I don t know how it would possibly do that

chris fish · Answer

i believe because its actively tracking the super schema in the metadata

chris fish · Answer

so it doesn t need to constantly verify with the data files

chris fish · Answer

<https www databricks com blog 2019 09 24 diving into delta lake schema enforcement evolution html> describes the logic burak implemented