TIL about Column Invariants from which are 100 not bei

Question

TIL about Column Invariants from < TD> which are 100 not being looked at when the Rust writer does its writes rolling on the floor laughing

Will Jones · Answer

They are in some code paths I thought

Will Jones · Answer

Also they are deprecated because Spark doesn t always respect them either

Will Jones · Answer

See <https github com delta io delta issues 1239>

rtyler · Answer

where is there any comment about deprecation

Will Jones · Answer

We have a function to check these but I don t think it s used outside of the Python writer and the writer in the operations module <https github com delta io delta rs blob d275c6a2cbbaf96ba5994dd24ad3c7420a9fe34c rust src delta datafusion rs L735>

Will Jones · Answer

Well it s not explicitly deprecated but I ve asked it to be

Will Jones · Answer

It s not documented like I had to really dig into order to figure out how to make a table that has column invariants in Spark

rtyler · Answer

heh well I m on the fence on whether I want to explore using these more then

Will Jones · Answer

Use constraints instead

Will Jones · Answer

That s the thing that is favored over invariants

Will Jones · Answer

Though we don t support them yet

Will Jones · Answer

It s on my radar but whenever I ve asked users they always seem to prefer prioritizing more operations like UPDATE DELETE MERGE over supporting higher protocol versions

rtyler · Answer

that makes sense to me

Robert · Answer

we also use the InvariantChecker in a dedicated writer we use in the operations module I kept a separate writer implementation to be able to experiment more without impacting too much of the code base Eventually the current writer module and the one in the operations module should merge

Robert · Answer

correction the check does not happen in the writer itself but rather in the write command before passing record batches to the writer <https github com delta io delta rs blob 4f9dec2054cd8591d92b7a17d5c3ef1acc1c8068 rust src operations write rs L363 L367>