Alessandro Biagi

06/13/2023, 9:31 AM
Hi all, I'm performing a merge statement to implement an scd2 on my delta lake. The source here is a transactional database. I'm detecting that from time to time merge is not working properly. I have some flags called min_flag, max_flag that indicates if the row represents either the first or the last status. However, sometimes when we have updates in our transactional database the merge statement updates correctly the max_flag for the existing row (setting it to 0) but it misses to add the new row that represents the last status. This behaviour is not sistematic, it happens on some days and only for 1 or 2 rows per table, it's very strange. My delta lake version is 2.2.0. Did anyone detect anything similar to this? It doesn't make sense and it's affecting my trust on delta to be honest.
I thought that this fix would prevent a behaviour like this to happen: