I believe we should be strongly encouraging downstream consumers to use our re-exported

arrow

and

parquet

modules in our documentation. I am finding downstream users importing these themselves and not being able to take on new versions of delta-rs without some manual upgrade work. This is especially problematic for downstream consumers who are 2 orders removed from us. Do y'all have opinions on why somebody should be adding both

arrow

and

deltalake

to their

Cargo.toml

?

You have probably seen more cases than I have, but I suspect if they are using the arrow feature we aren’t the only crate they are using that has an

arrow

dependency

arrow and parquet as a transitive dependencies are kind of nightmarish IMHO. The APIs are so fluid from one release to another that I feel like the only way to provide stability is to encourage pulling our re-exports 😕

actually, how does using our re-exports solve this? when we upgrade, won’t they still have to change their code to adapt to breaking changes?

Most of the use that I have seen has been for benign things like schema definitions or writing RecordBatches, by encouraging our re-exports they'd be basically pinning to our version of these crates that are known to work, instead of getting hit with a bunch of ABI mismatches when they try to take an upgrade of either deltalake or arrow. If they've adopted some genuinely interesting APIs from either arrow/parquet then of course we cannot help them 😛

Fair enough. It seems fair to give a warning in our docs they should use the re-exports or else be sure to match the same version