https://delta.io logo
r

rtyler

03/28/2023, 4:27 AM
@Robert I am working with the
OptimizeBuilder
tonight, and I'm curious why
with_filters
takes
PartitionFilter<&'a, &'a str>
where T =
&'a str
rather than an allocated `String? Lifetimes are not my strength but I'm not sure how it would be possible to create
PartitionFilter
with anything other than static strings for the partition values. I have tried a couple different options with no success, and I think the
T
used for
PartitionFIlter
is just too restrictive πŸ€”
r

Robert

03/28/2023, 4:29 AM
That’s a good point! The partition filters have been around since before the operations, and beyond avoiding allocations, I am not aware of any reason wha this choice has been made.
More generally speaking, how to express filters - partition or more general - has been a point for some time. We thought about pulling in datafusion_expr and use their representation for expressions and scalar values.
r

rtyler

03/28/2023, 4:30 AM
Ah, you're highlighted on
git-annotate(1)
because you moved the code, not because you wrote it πŸ™‚
πŸ™‚ 1
r

Robert

03/28/2023, 4:32 AM
The open PR for optimistic commits actually starts to use datafusions expressions more deeply. With the upcoming core activities though, I hope we get a lower cost abstraction (in terms of dependencies) ,,,
r

rtyler

03/28/2023, 4:33 AM
the definition of
PartitionFilter
isn't too awkward, the lifetime constraints are a little odd. I'll experiment with
OptimizeBuilder
and just a different
T
πŸ‘ 1
I feel like trying to write a lot of little Lambdas has me stretching a number of these APIs πŸ˜†
πŸ˜‚ 1
r

Robert

03/28/2023, 4:43 AM
I do believe that table maintenance / optimisation is something where delta-rs might actually be very well suited.
… after we do some optimizations πŸ˜„
one thing that comes to mind is that I wanted to look into actually leveraging the batch delete APIs in object-store. I.e. the API is there, but they don’t yet use the optimized APIs from the backing services.
βž• 1
w

Will Jones

03/28/2023, 2:43 PM
(I’ve been trying to nerd snipe Rust devs into doing that upstream object store work for months πŸ˜† )
🀣 1
r

rtyler

03/31/2023, 4:23 PM
@Robert FWIW I started looking at updating
PartitionFilter
with some changes and it got messy really quick, so I just hacked around the lifetime issue a bit: https://github.com/buoyant-data/lambda-delta-optimize/blob/main/src/main.rs#L37-L53 πŸ˜†
2 Views