https://delta.io logo
r

Rahul Sharma

02/22/2023, 5:17 AM
Hii Team , this is question not related to DeltaLake it’s related to CDC(Debezium) i am fetching the data from sql server via debezium connector but i have a scnerio where __source_ts_ms is max for insert but should be for update .bcz i have applied rank function on refine table partition by id order by __source_ts_ms desc can someone confirm what should the best way to find last insrted record for same id
Copy code
+----+--------+-----------------------+------+----+------------------------+
|rank|amount  |lastUpdateTime         |status|__op|__source_ts_ms          |
+----+--------+-----------------------+------+----+------------------------+
|1   |null    |2023-02-08 12:38:55.407|0     |r   |08-Feb-2023 10:53:52.000|
|2   |100.0000|2023-02-08 15:02:06.7  |2     |u   |08-Feb-2023 09:32:06.000|
|3   |100.0000|2023-02-08 15:01:36.95 |0     |u   |08-Feb-2023 09:31:36.000|
|4   |100.0000|2023-02-08 15:01:34.147|0     |u   |08-Feb-2023 09:31:34.000|
+----+--------+-----------------------+------+----+------------------------+